 More cross-team collaboration, higher job satisfaction among your employees, better code, sounds good, right? Today we'll learn how Delta has delivered these outcomes for their teams through the adoption of two approaches to collaboration, inner-sourcing and mob programming. In this talk, Delta engineers, Ankur and Lauren will show how and why they adopted inner-source and mob programming to deliver shared solutions across their organization. You'll hear about where the inspiration for these approaches came from, how they grew these initiatives over time, lessons learned along the way, and even how some of this work led to a contribution to Gillette. I am Ankur Marfathya. I have been at Delta Airlines for 17 years. Over the years, I have enjoyed various roles at Delta. I'm currently managing the DevSecOps experience team. My team helps set up development and DevSecOps standards for Delta, in line with our short-term and long-term vision by collaborating with various stakeholders. I regularly organize and facilitate enterprise-wide developer discussions on various topics. I also develop, facilitate, and conduct, both in-person and video-based corporate training on the latest tools and technologies. I like learning new things, as well as participating in innovation challenges. Of work, you can find me at various virtual, as well as in-person, meet-ups around Atlanta. Hello, my name is Lauren Kirby, and I am a DevSecOps engineer at Delta Airlines. I've been at Delta for about two and a half years. I'm a delivery automation leader, with more than 37 years experience in software development. Recently, I've been working on creating Tecton pipelines to support migration to OpenShift on AWS. The Tecton pipelines have benefited from knowledge I gained, working on the Jenkins Templating Engine inner-source project. These pipelines provide full functionality, including testing, code coverage, sonar scans, security scans, packaging, and deployment. We support Maven, Gradle, and very soon Node.js build types. I've also been a DevOps coach in the Delta Airlines Speed to Market Dojo, and the Delta Agile Coaching Office. I'm an experienced full-stack developer on many platforms and using a wide variety of languages and tools. Delta Airlines is the leading US global airline leader in products, services, innovation, and reliability, reliability, and customer experience. Powered by its people around the world, Delta continues to invest in its people, improving the air travel experience, and generating industry-leading shareholder returns. With an overall score of 860 out of 1,000, Delta was recognized as number one in customer satisfaction among airlines in North America by JD Power. Delta Airlines is headquartered in Atlanta, Georgia. Our journey begins in pursuit of a good pipeline, so let's define it. A good pipeline should have various points of feedback that aid in continuous improvement. Feedback about simple things. Like, does the code compile correctly? Does the added code have any negative impact on existing code? Are the unit tests written for this purpose of good quality? What is the quality of the code itself in terms of industry standards? Are there any security vulnerabilities in the components? Are there any security vulnerabilities in the compiled code? In no particular order, of course. This feedback should be available at points where it is most important in terms of providing value. We have all heard about shifting left, security shifts left, quality shifts left, but how much left should it shift? When does it give us most value to execute each step and seek feedback? The answers differ from organization to organization, and sometimes from team to team within the same organization. There are a lot of tools listed here. To create a pipeline that encompasses all these is extremely difficult. Some may say the most difficult things about DevOps is the integration of tools into a pipeline. In fact, it is not uncommon in today's DevOps landscape to spend more time integrating new tools with existing tools than to learn its use. Too many questions and decisions before we build a pipeline. In August of 2019, my teammate Eranseri and I were invited to attend the DevOps world Jenkins world event organized by Cloudbees and the Jenkins open source community in San Francisco. We picked up an inner source idea based on an open source project by a company, Booz Allen Hamilton, known as the Jenkins templating engine. Developers would have to write a small configuration file and a pipeline would be automatically created for them. Not only that, this pipeline would be centrally controlled. So later, if a new stage needs to be introduced, it could be added with minimal developer action and executed on the next run. Imagine redoing hundreds of pipelines when you need to make a small change and you can understand why this idea was of great interest to us. We have a practice in DevOps COE after any conference that any one of us attends, we come back and discuss what are the new ideas that appeal to us and sort out ways in which we can benefit from it as a company. The JTE idea was no different. Inner source is the use of open source software development best practices and establishments of open source like culture within organizations. At any given time, at a large organization, there are various teams running into the same issues or similar issues. Various teams across the enterprise are working in silos to solve these issues individually and sometimes repeatedly on different projects. The goal of the inner source effort is to bring these individuals together around these issues so they can be solved at a common place and solutions can be collectively improved upon over time. The industry has realized various benefits of inner source. The openness of the project extends across many teams within the organization. Programmers share their work with a wide audience instead of just with a manager or team. Anyone in the team is free to view the code, comment on it, learn new skills by examining it, submit changes that they think will improve it or customize it to their needs. Each team can understand the code and the architecture of modules developed by other teams. New code repositories based on the project can be freely created so teams with unanticipated users of the code can adopt it. It is a great way to reuse code across the organization and make cross team collaboration relatively frictionless. These efforts are known to promote greater intellectual growth and job satisfaction. So how does an inner source project differ from a regular development project? First, continuous learning is a critical part of each development. It instills a culture of change to give employees the confidence to contribute code to other teams. The host organization allows others outside the team to play a major role in developing, testing, documenting and promoting the software, creating a level playing field for all contributors. Writing unit tests becomes a key programming task. Constant code reviews are part and parcel of life. The written comments exchanged among team members, although taking up some time, more than pay for themselves by helping new developers learn the system faster. The project involves the end users in relatively old-fashioned ways such as through alpha and beta releases. At Delta, we implemented inner source first by creating an enterprise inner source committee. The job of the committee was to seek out and approve projects appropriate for this program. Set and review standard operating procedures. Decide upon and award contribution initiatives. Make sure that the promotion system functions correctly in the inner source organization. Second, we created a group of trusted contributors for each onboarded project. Next, we identify and create a committee of maintainers who are responsible for driving the vision of each product. Could be the same group as the trusted contributors or a subgroup or completely different group. Their job is to promote the projects and encourage committers. Also maintaining a channel of communication with users of the product so that they can gather new requirements and track bugs. The idea of an automated way to create enterprise pipelines was appealing to all of us. We invited our entire DevOps team to our first mob programming session however only I and Ankur turned up. I had done some reading about JTE and Ankur had glanced at the documentation. It was fair to say we did not know much other than that we had a goal and a strong desire to achieve it. We spent about two hours in that session mobbing or rather pairing interrupted by my one-on-one with my manager. At the end of that session we were able to set up the Jenkins instance and we could build and unit test a sample application using Gradle. It may seem trivial but this was a significant first step. It was a success that left us confident in what we were doing and wanting to do more. What is mob programming you may ask? Mob programming is a variation on pair programming. In mob programming there is a driver and several navigators. The driver can be thought of as an intelligent input device as opposed to a keyboard or mouse which is a dumb input device. The role of the driver is to take what the navigators describe and enter it into the computer. The role of the navigators is to analyze a problem and give the driver instructions on what to input into the computer. At predetermined intervals the driver becomes a navigator and one of the navigators takes over as driver. We usually do this as a round robin. At first we used an ordinary kitchen timer to time the intervals. To quote a leading expert mob programming is all the great minds coming together in a room at one computer to solve a problem. For a thought to pass from one person's mind into the computer it must first pass through another person's hands. Mob programming is not one person coding while a bunch of other people watch. I first learned about mob programming at the dojo consortium which is an annual conference of companies that have established internal dojos. I was impressed by the idea and eager to try it. We had a situation where in one of our classes only about 10% of the learners were completing the lab. We tried mobbing on the lab in small groups of four to five participants and went to 90% of the participants completing. Since that went well we tried it on another class with the same results. Those of you who facilitate adult learning know that having 90% of the participants complete the work is an excellent result. Why would we want to work this way? Well there are a number of reasons. Knowledge sharing, continuous code and design reviews, many perspectives on the work, we work on the right things, rapid feedback, better solutions, the flow of the work, higher quality, it's more fun, less stressful, and more engaging just to name a few. How can five people be productive, all working together on the same thing? Well we can think of productivity as the output we get from certain inputs. We can think of efficiency as doing things right and we can think of effectiveness as doing the right thing. When we're mobbing we don't make we don't mistake productivity for accomplishment. A better question we might ask would be how can we be effective if we separate the people who should be working together? Some of the things that make us less effective are communication problems, decision-making problems, technical debt, politics, meetings, mob programming addresses all of these. Over the next few weeks we recruited a few more contributors and they brought along a few more themselves. Soon we had Iran, Anthony, Rakesh, Krishna, Macau, Michael, Donald, Jason, Shailesh, myself, and Ankur meeting regularly to try and build this system. We were truly mobbing. As we started mobbing we realized a few things. This was a very creative and active way to work but a few issues started to surface. We started with one dedicated device that every driver used. This led to some awkwardness. Some of us were used to using Macs, some Windows, some of us used Atom, others used Eclipse or IntelliJ. In addition to being told what to type we ended up coaching the driver on how to navigate what felt like a foreign computer. There had to be a better way. We decided that everyone should use their own machine when it was their turn to drive. This made the drivers happier. It also meant that every driver had to take extra time to check in their code and the next driver would have to pull before starting. The minutes started adding up and our efficiencies started to drop. There was a better solution. This is when we decided to give GitLab's Web IDE a try. We managed our backlog in issues. Issues became merge requests. Merge requests became branches and branches we would open in the Web IDE so we could make quick changes. For our simple pipeline design applications this worked well. It worked very well with our GitFlow process that we had established. Committing the code between transitioning drivers was quick. Then came the pandemic and we found ourselves working remotely. Fortunately many of the things we had learned mobbing together each on our own computers translated nicely into remote working. We found that screen sharing and using the Web IDE worked well for small changes. For large changes we were able to use Microsoft Visual Studio Code with the live sharing feature. When we were in person we used the kitchen timer but working remotely we found that the timer provided by Pomodoro gives more visibility although we still use the kitchen timer as well. We also used Microsoft Teams meetings and screen sharing. Most team members were able to adapt and be productive working remotely. Some didn't like the remote mobbing and found it difficult to stay focused and present. Some key takeaways we learned were to minimize distractions like cell phones and other devices. Be disciplined in the use of the timer and changing drivers. Don't require everyone to drive. If there's one person who's significantly more knowledgeable about a particular topic ask them to drive last or not to drive and be sure to take regular breaks. We also learned the importance of setting a single clear objective for each mobbing session staying focused and having a brief retrospective or review at the end of each session. Every inner source effort requires a website one that can communicate well what the project is about attract new contributors as well as users. The original open source project used a framework known as PINX to convert Markdown to HTML. One of the advantages of using GitLab we have found is you can use Git pages to host a static website. There are various frameworks supported which will help you convert Markdown documentation to HTML. Unfortunately, PINX isn't one of them. So we decided to build one ourselves. As you can see we have started with a basic image installed a few system and Python packages moved out generated HTML code to public folder and published the website. To do this we studied how PINX would work offline and converted it into a pipeline process and implemented it with GitLab CI.yaml. And yes, we mobbed over this sub project as well and it was amazing how quickly it was built out. We now have a merge request pending with GitLab and hopefully it will get approved soon. Ever since that first project we have tried mobbing in a few different inner source efforts and currently we are actively using it for our daily sprint stories. We've added a few more tools to our repertoire based on the projects that we're working on. For simple changes Web IDE is still our go-to but for a bit more complicated development Visual Studio Code along with Live Share and Live Share Pomodoro are soon becoming favorites. We will be available after this session if you have any questions about our journey or if you want to know how you can start your own. Also, we have shared some helpful links on this slide. Thank you for your time and we hope that you have learned something new today. Don't be afraid to reach out to us. Thank you.