 I'm Matt Davis, Red Hat Solution Architect, and with me is Bubba Perrier, an IT fellow at Syngenta and Jay Huddleston, a senior Red Hat consultant. So Syngenta is one of those DevOps unicorns that you read about. When they were implementing DevOps, they started by building out their DevOps culture, then they laid down their processes, and after all of that was done, they selected the right tools for the job. They chose OpenShift Dedicated as the platform for running their containerized workloads. The journey from legacy was challenging. They were trapped on proprietary platforms in a managed data center. They had very little access to the platform, the configuration, the logs, very little knowledge as to the application behaviors. But his team was able to containerize those workloads and lift and shift onto OpenShift. And then they re-architected these applications, breaking them down into microservices. So with DevOps tooling and containerization, they were able to realize the benefits that we commonly talk about, such as shorter application delivery times, better value back to the business. But today we're going to focus on some of the lesser discussed operational benefits, such as making it easier to manage and debug applications. Specifically, he's going to talk about and tell the story of a nasty N plus one query issue in production and how they were able to discover, debug, fix, test, patch a live critical production application all inside of three days with no application down time. So thank you, Matt. So as Matt said, my name is Bubba, and I work for Singenta, and it's my privilege to bring you the story of this team that you see here. So we're a small team in a large company, and we've gotten this ethos over the last few years espoused by our fearless leader who you see out front there, Chia Tana. She's our magical unicorn lady. She's always telling us to be a unicorn in a field of horses. And so this is that story. So first off, I work for Singenta, which you may not have heard of unless you're a farmer. So we sell the tools to farmers so that they can bring plant potential to life. So big farmers, small farmers, organic, conventional, it doesn't matter. We sell them these tools to seeds and chemistry. So to maximize the benefit of their plants to the farmer, and also to bring that potential to you and I. So I don't know about you, but I am looking forward to dinner in this foodie town, and I'm going to thank a farmer when I get home for providing it. So they're our customers. And kind of why Singenta's mission really matters, really hit home to me when I watched Neil Holleran's The Fallen of World War II interactive video online. So it's a very somber video, and I highly committed to you. I do, and I'm a positive person, really. And it ends with this acknowledgment that we've really had seven decades of unprecedented global peace and prosperity. And what that means is that there are more people living longer, living productive lives, eating higher on the food chain than ever before in human history. And so as the population explosion has happened, we're losing farmland to expansion development that's needed. And so we have to get more out of our agricultural farmland. So in a sense, Singenta's mission is really to make sure that your kids and mine don't have to go to war over who gets to eat dinner, which is an awesome mission. So my job at Singenta, I'm primarily a technology evangelist these days. I'm supposed to be an architect. And for the last four years, I've been an architect on an internal transformation program that is delivering a set of tools to implement like ancestry.com for plants, since our seeds business, those are our products, and an integrated inventory tool and migration engines and supporting services. So it's got this whole ecosystem and a lot of business change to do. It's a lot of work. One of these internal multi-million dollar, multi-year transformation programs. About two years ago, we got tapped and said, hey, like everybody else in Singenta, you need to pick your load up out of the data center and get it to cloud. And by the way, you've not given any additional time for your regular work and you're not going to get any additional funding to do it. So a fearless leader looked at me and said, hey, you're the architect on the program, you need to figure this out. So get us to cloud. And because we're a unicorn team, it can't be boring and it's got to pay for itself. And two, two and a half years ago, to be honest, I didn't know Jack Didley about cloud. So I did what any good architect would do and I got on YouTube and did a lot of blogs and, yeah, I see some nods. So thank you, Dave Hahn and Burst Sutter and Kelsey Hightower and Matt Stein and all the others. And I kind of put together this theory and I kind of conned our graphics artists into putting this propaganda where together to say, hey, this is what winners are doing on cloud. So I went looking for how do I do cloud and I kept running into microservices, DevOps and containers. And yes, I did steal Joe's quote there from the Forbes article and didn't ask him for the likeness, I hope you like it. And I took this to DevOps stage Raleigh, which was the first one in 2016. And I asked everybody I could, is this true? I've drank the Kool-Aid pretty hard, but I've got zero experience with this. And I kept running into all these people from this team on this product called OpenShift and they said, yeah, we believe this. And not only that, we have a platform for you. And of course that platform is OpenShift. And so I go back to work the next day and I go into the boss's office and say, hey, Chai, I found this thing called OpenShift. I think we should do it. It should really leapfrog us on our innovation journey. And I started going into the whole sales spiel about how dedicated, Red Hat will take care of all the hardware and the software stack. We just have to put our load on it. It was great. And she kind of stopped me cold and said, cut it out. And she said, go get it. So I was like, excellent, I like this. So like I said, we're for a big company, so buying things isn't always easy. So it took about three months to get everything done between knowing how to spell OpenShift and having an agreement in place at Red Hat to purchase our first OpenShift dedicated cluster and sign up our first professional service's engagement to do a proof of concept. Cuz we still had a full play to work that we're gonna do. And we still had crops adopting our tools and all kinds of business change to do and that lift and shift up to the cloud that we're doing anyway. So all that's remained on task and in flight and in parallel we brought in additional technology and help from Red Hat to kind of boot strapless on this journey and it went really well. And so we decided we wanted to do real infrastructure. So it's one thing to demonstrate that you can take your internal line of business application that's got internal authentication authorization and push it out on the internet and make sure it works with your data that secured internally. It's another thing to do that in a repeatable, scalable way. So we got another engagement with professional services with Red Hat. And we also purchased a second cluster to be our production cluster. So we have two clusters. We've got the lower environments all in one and production separate, which is a pretty common occurrence. I think you've heard several times today. And then in September of last year, we picked up all of our infrastructure, deployed it on OpenShift dedicated and unleashed it on our users. And one of the interesting things about being in an agriculture company. So harvest season is go time for us. When the combines are out in the field, that's when all the data's coming in and material and everyone's busy. So kind of our Black Friday in our company is when the Northern Hemisphere, which is where most of landmasses on the earth exists, is in harvest. So between the middle of August and the middle of October, it's not a good time to be messing with things. That's go time. And that's right when we did this. Anyone up without a hitch? It was great. A month later, we had our incident that I'm gonna tell you about. And so that was about three days. We had to deal with this production issue. And since then, we've really been doing a lot of innovation on top of OpenShift. So kind of jailbreaking out of our internal lockdown, VPC, kind of the environments that you've heard so many other stories talk about having, busting out and becoming online. We're able to do all kinds of things, so like mobile application development. And just today, we enabled bring your own device of our internal line of business tool that's designed for Internet Explorer on the Sinjenta Windows laptop. Now our users can use on iPads and whatever, which is really cool. And useful, it's not just cool, it's useful. So I mean, it'd be clear. So we spent quite a bit of money last year on OpenShift and professional services engagement, but the speed and the resolution that we're able to get has really paid for itself. So this is, we're really, really excited about technology platform. And when the culture changed. So now I'm gonna tell you about this incident we had. So the deal was we had a service that took a list of things. And then for each thing in the list, it would spawn a thread, each thread to go grab a database connection and do its thing, which is bad. So, but it's not so bad if nobody ever passes a lot of things to the list. We didn't know this was the problem, of course. All we knew is October 10th of 2017, we came into work. And the help desk board was lit up like a Christmas tree with people saying, hey, that system you foisted on us, it's down, it's stuck, it's frozen, I can't log in, a bunch of new users, why did you do this to us? It's our busy season. And so OpenShift really helped us in four different ways managed through this issue. The first way was just with pods. So it didn't take us long to realize that not everybody was complaining about the system. And so that made us wonder why some people were having success in the system and other people weren't, was it a regional thing, what was going on? And within a few hours, we figured out that it was basically one pod was stuck. And so spinning up more pods reduces the blast radius of the issue. So we did. And my counterpart who kind of runs production support, so if you think of me as like the guy over all the devs, my equivalent on the op side said, hey, look, that pod is dead and users can't do anything, isn't this platform supposed to fix things? So what if we just killed that pod? And so we said, okay, well, we've never done that before, but fine, so we killed the pod live and OpenShift did what's supposed to do and spun up a new pod. And thanks to the magic of session persistence, all the web sessions from the spun down pod got attached to the new one. And for most users, the experience went something along the lines of the system was spinning and waiting or whatever. And it came up with some goofy error message. I don't know about your users, but 90% of mine in that case, when it comes back, they just push the button again. And when they did, it worked because it was a healthy pod. So I mean, it doesn't fix the problem, but it kept us in business and kept us running. So the next thing that really helped was all the integrated logging that OpenShift has. And this actually was a stumbling block for us at first and actually kind of how Jay came to us. He was helping us out on a number of things. One of which was with distributed pods, you got distributed logging and distributed logging is painful. So our logs initially just logged for Jay logging straight to the standard out or whatever. We're going over each line as an individual entry into Kibana, so it was not very useful for searching. So Jay helped us fix our stuff up so that individual log entries are going over a whole, even if it had a full stack trace in it. So that allowed us to very quickly, that's actually what helped us discover which pods that we had a problem with a pod and which pod it was. Because the database guys are basically saying, hey, this connection is starved. And then we were able to look in the Kibana logs and say in that connections associated with this pod, there you go. That's the pod that we need to do something about. And that also of course gave our teams all the context clues necessary to get started trying to debug and find the issue. Which is tricky because the symptom was a bunch of people saying the system doesn't work, so it wasn't a lot to go on. So the next way OpenShift really helped was having sandboxes for testing fixes. So the team obviously were trying to find things that might be wrong, looking at different cases. So standing up sandbox environments and we eventually figured out, hey, yeah, duh, who wrote that code, we fixed it. And then of course we commit the change and then we test it in our QA environment. So it's awesome being able to have sandboxes and QA environments that are exact replicas of your production environment. And having infrastructure as code on top of a container platform, that's not hard. And there really is no excuse for not having it once you're there. So that was really awesome. And then really cool on the Thursday. So again, Tuesday morning, it was a circus. We had to set up the circus, but we were all upset that we had this issue. Had to pull the War Room team together. By Thursday, we've got the fix. We've got the image, all right, we're ready to push it out. We just did a hot deploy. And the live system and the bug was gone, which is cool. We'd never done, before OpenShift, we'd never done rolling deploys and hot deploys. So we'd always have to broker an agreement with our users. One, can we get an outage window? Because remember, it's Black Friday season. So we've got people running 24 by seven ops and whatever. So we can't just take the system down for maintenance, whatever we feel like it. So not being able to keep the system up and not taking it down was a huge benefit. So I mean, honestly, between the technology platform and the partnership with Red Hat, I mean, we came a long ways in less than a year. And now Jason's going to tell us a little bit about the benefits of working with the partnership with consulting. So Bubba's story really shows how the partnership with Red Hat Consulting can shorten your time to market by utilizing our consultants experience with various products. We can also reduce the learning curve through our training programs or just by knowledge transfer by working with our consultants. I was assigned to Syngenta to help with a few things that are on the screen. The cross cluster promotion. We were promoting code between their dev and their product cluster. Also the logging issue that Bubba talked about earlier. A few registry and user issues and some AWS integrations. While Syngenta's employees could have worked through these issues on their own, they were able to save significant time by letting me help out a little bit. And this allowed Bubba's team to focus on future efforts. Yeah, so that is very, very true. I mean, so hats off to the OpenShift team. I mean, you guys have put an awful lot in the box. So it's a cool product. But it's also an awful lot to really, as Matt says, to go from kind of 101 to senior level work. And I can't commend any stronger. Get help. It's tough to tackle a large learning curve on your own with your own team. So I mean, since then, we've really kicked up the innovation level, honestly. So you think about this. We had an internal line of business application that had never seen the light of the internet, right? As internal users only. So you assume it's a hardened perimeter. So your team is not rigorously focused on security issues. Not that we're sloppy, but it's just not ever done in anger, right, because it's internal. We took that and lifted out and stuck it out on the internet. I've got a diagram I'll show you up here, the basic connectivity. So the databases are still in Syngenta's private virtual cloud. So we've got firewalls and all that in place. But this has really allowed us to take advantage of other online internet-facing services. So maybe a picture is worth 1,000 words. So we've got two OpenShift clusters. And of course, they're sitting out on the big, bad internet. And this has allowed us to do things like mobile application development, which mobile apps are easy to create these days, but it's a lot harder to tie that to your organization's authentication mechanism. So it'll owe off to you from, say, Cognito, from AWS is what we use. And now my mobile apps can consume services that are hosting private services and private data from internal Syngenta sources and do it do so securely. And have InfoSec not like throwing the flag on you when you're trying to innovate, which is really great. We've also got some serverless architecture things that we do so that the typical stacks of API Gateway, Lambda, and DynamoDB. We have some microservices out there that are easy for us to consume and integrate with as well. It'd be a lot harder to do just in our virtual private cloud with no internet ingress. So our databases and all of our build tooling and dev support tools are all internal. And our application code is hosted externally and that's kind of how we set it and how we do it. So like to say a huge thank you to Red Hat for sponsoring DevOps days, because without that we wouldn't have found, those unlikely would have found what you guys were up to. Thank you to Red Hat's professional services and consulting for helping them bootstrap us. I mean, honestly, our team has come a long ways in a very short period of time and hats off, of course, the OpenShift team for such a fantastic product. And to this team up here, so I'm largely a purveyor of power points and diagrams. So these are the folks that actually made it real. And of course, to our leadership for having inspiration. I mean, I can't stress that enough. If you're gonna make a transformation journey, you've gotta have leadership's commitment to say, hey, innovate, give you freedom to operate and to go after things. So appreciate it. And of course, to my management for, you know, authorize me to come out here and speak to you today. I appreciate that as well. So thank you, Patrice.