 Hi, my name is Ivan and I'm director of engineering in ring central. So I've worked in different roles in an essential I work as software developers or for architect team lead Product manager like every everything can manageable and right now I'm working in ring central and advocating can implement in github's processes across the whole company So it will be my pleasure to share with you our experience how we approach github's in the large enterprise company thousands upon thousands of users Huge work laws and what challenges we faced Excellent, and my name is Toma. Oh Nakahara. I run the developer experience team at we've works the company that created flux and flagger And coined the term github's so we're really excited to share our stories here for you to learn from So please take a picture of this QR code It leads to a site where we have all the flux things for this week whether it's our flux booth and all these talks And Activities that we have and for those of you watching the recording We'll still have the site up so that when the recordings are up. Hopefully you can see it It also has ways for you to follow up with questions. So please do chat with us I'll show this at the end as well if you miss it, but thanks for taking the picture So ring central will talk about what they do and what their drivers are and they're kind of in four key buckets Security and compliance scale velocity and reliability and we'll share how flux and flagger help with all of these in really important ways So first of all security and compliance so Ring central is a company that absolutely needs security and compliance and get ups helps with that First of all, if you don't know they have multiple products. There's Telephone and messaging and video I'm sure many of you have worked at companies where you actually have those physical phones As well as other capabilities. So imagine it's a company that has a chief compliance officer It has auditors both inside the company and outside and so auditability and observability are absolutely required Needs that they have they both out. They also have legacy and modern systems which brings with it dependencies and Requirements and configs and secrets management stuff which brings certain levels of complexity that they need to take care of and Similarly because of that they have multiple solutions and products that bring those same challenges so that means they absolutely need to have a reliable get-ups system and Tools like flux help with that and to be able to give them the security compliance Auditability and policy management that they need that will cover in this talk Especially when you have human beings involved It's just not gonna everything that you see on the left here The things that they need to do is not going to happen if they're chasing after human beings are like who made what change Under what context when you know all that is too manual and that would not be able to be possible So flux helps with that So quickly if has anybody not used flux yet Right, so great if you're new to flux will kind of segue and share little bits about it So flux is a project in the CNCF. We're incubating and actually right after cube con will be finishing the final steps We hope to get to graduation. So it's been a wonderful journey here in the CNCF and flux Like I said, it comes from we works and the company that coined get-ups so it's been providing get-ups both for apps and infrastructure and One thing that it does is it listens to the repo and if there's a change in that repo as a single source of truth It talks with kubernetes and implements that change So it kind of created that that idea of having that single source of truth in git And because you already have a version controlling system that means you already have an audit trail That's part of that. So there we go about the need for audit ability And also provides added security because you're not exposing your API to the outside world And you're also not Putting your secrets in CI and if people have been following the news, you know More more hacking is starting to move in the CI space because they're seeing vulnerabilities there and being able to access a lot of data So flux is already, you know steps ahead of this and always making sure that we're evolving so that we meet those security needs That includes supporting OCI the open container initiative and cosine, which means that you have added capabilities for verification Okay Yeah, and like Tomas said we will start firstly with security in mind because if you're working in a large enterprise and if you're a large enterprise when you're trying to implement some delivery tools or development tools which tightly coupled with your production system I think the first question everyone answers is What you will do with compliance and security and I would not talk only about just the compliance I will talk about actual security because ring central is a As you can see from this map every country market in orange We actively operating in you can buy all services and most likely every one of you in your life encountered or dial it to the phone line or contact center line which Operated or hosted by the ring central So we like biggest telephony company in the United States at least on unified communication as a market space So when we Yeah, so when we're trying to tackle such a diverse Geography and tackle different types of compliance. So we are creating government and health care So it's a hippo compliance different sock compliance. We are public company. So you can imagine it's Like audit upon audit and we have it constantly during our years. We also Always try to target it by the hackers we have different types of attempted breaks in our system and we won't make sure all our clients are safe and Every data is protected. So the first first and most important question is a security Not the scale not any other ones. But if you fail at security, you basically do not need to use this to or If you cannot answer this question so When we talk before we talk about the security itself and how flagrant flags approach it I want just to segue into one particular Important part we use and we use it. I think it's very tied to our success of using githubs in the company And we use it everywhere. It's a dramatic build So using if anyone views familiar with build systems such as Basel or Nick sauce or just a term of hermetic build It means regardless of the environment regardless of the computer you run your build on or build your container on Daytime Different libraries you get beat to beat exact copy of the container if you build from the same source So what does it mean the same source source is a source of truth? It's our source code. So if we have everything which needs to be so there's no dynamic variables dynamic dependencies Everything can spin down and if we have it everything pinned down and inside the repository We can build a dramatic container And we can argue it's Basically, we can build on ten different machines It should be beat to beat exact replica if something goes wrong and the replica is not exact It means the machine could be compromised as a process is compromised and As you can and it's a development practice yet It's not have nothing to do with githubs yet But you can see we how we go from here to githubs because in our development life Git repository is a source of truth as this is source of our source code in all the dependencies So we wanted to do the same in our infrastructure So how we can approach it in our infrastructure and here is DevOps or githubs Helps us and answered question. It's a little bit complicated image. I think we straight slides will share I will not go detail it through it but We definitely divide our domain into runtime and think configuration domain and configuration means everything which sits in gith so we Apply it everywhere we use githubs and it's very important. We use gith centric approach or configuration as a code infrastructure as a code as I would say it in the keynote's presentation we're trying to See what everything we do can be outlined as a YAML file or terraform configuration file Or any other file inside the git repository which can we act upon and act in a mutable manner same as for example with our Distralis and our grammatic images So going from here we can definitely see From the security point of view. We also while we're using flux We implement in different security practices such as static scan for the CV vulnerabilities as CV lists So if we have every dependency pinned down, we can easily do the reverse scan which helps the security a lot So what does it mean? It means if you have zero days as vulnerability in your in your system You can and you know every desired in your desired state of our cluster and the desired state diverges It's an incident. So we manage and with basically Always monitor the version of the state But if we have the desired state in the repository we can Go and see on the release of zero-day vulnerability We know exactly whether this vulnerability is existing thousands of clusters So we do not need to time to scan because if we scan every system for vulnerability After we receive the CV vulnerability it can take weeks in a large company such as ours So it definitely helps us to with detecting the CV vulnerabilities zero-day vulnerabilities We also employ flux integration with Mozilla sops So if you don't know it's really great tool to keep your secrets inside the git repository sounds counterintuitive But it's very nice thing very nice Solution to keep things atomic so it changes. I'm not distributed. I'm on different repository like a secret wall And for example your git repository. It's all in one place in atomic state and a git commit. So we do not wait with Slowly Adhere to the practice of storing everything everything can get it It's helped us and helped security compliance a security audit in our company It made it a lot easier all the trails all the changes in git Merge requests is it's change requests So if you for example enterprise, you know this change management system Etc so merge request is completely a maps one-to-one and we managed to convince all our Auditors such as KPMG price waters house coopers and different other auditors and our internal security teams what it really does Adhere to our existing security practices. It was a big win because we did not need to re-certificate So we do not need to do additional certification in terms of security and compliance here So yeah, this is how we approach the security Come on All right So hopefully company it's like yours that have regulations as such. That's something that's relatable I'm sure many of you have issues with scale So ring central's company as well as we talked about Their global company, but they so they have millions of users hundreds of thousands of businesses 200 Kubernetes clusters and growing And thousands of deployments and workloads. I think tens of thousands perhaps and as we showed on the map multiple regions around the world They've got hybrid clouds got their own data centers as well as GK and EKS and again As we mentioned multiple solutions as well as different development practices within house So there's a lot that needs to get managed and internal customers that need to be happy at scale So how does flux help with scale? so the flux project is actually a Microservices architecture and then a set it's a set of controllers that do things like creation deletion and updates to both your App deployments and to changes that you make for your infrastructure The source controller is one of the key controllers and one of the other ways that it helps with Scalability is that it has caching features that kind of unblock any issues that happen And it's important to mention here that our controllers support natively both customize and helm So that means like if you've seen the story in the CNCF the Department of Defense Absolutely wanted to use flux because they needed to use their entire helm ecosystem and flux Allows that to happen because it has native support And you have many get-offs options as you hopefully heard in the keynotes You know, there's the get-offs principles by the get-offs working group Flux completely complies with them and within those options you have the version that we talked about where you have Get repo that has a single source of truth But also with OCI now you have the availability to have flux consume artifacts OCI artifacts and to implement get-offs in that way So some of our companies enterprise companies who I can't name who Major major scale issues, you know, they're at the forefront and their major uses of flux So we've made sure that we've met those needs so they can not have Scale issues that come with git itself. So git is still there. It's just moved to the left so that we can meet their scaling needs So wherever you are in your journey just starting or wherever, you know that you can start with a project That's already built for scale So a little bit about your repo structure Mono repos are one methodology that we Kind of recommend and I know that Eva is very passionate about Mono repos Yeah, so the second need is Scale so if you figure it out your compliance it in the security need you may need to make sure you can run it at a large scale So meaning meaning yes, you can definitely try and many projects starts like this as a toy project inside the company But usually it does not make much sense to have many different processes Many different ways to deploy and manage your infrastructure in a large enterprise You want basically to make sure the things you build the things you're trying to adopt it can be adopted in the entire company and When we're talking about the large deployments and like Tomo said we have thousands of 10,000 of clusters different all around the world in Europe and Southeast Asia multiple clusters and multiple servers in the United States also we present on the cloud servers and a Google and an Amazon so it's a huge infrastructure and If we cannot answer answer how do we scale it infrastructure efficiently? We will most likely fail so it will not be adopted widely by the company It will be adopted by the some departments, but make me it will not work will not work So how do we approach it already on this slide? So we approach it on scaling on Mono repositories of what does it mean? So why Mono repository is so important so many of you may be already heard worked with Mono repositories They've just your software development for example Google has a really nice way of managing this Mono repository system Which we also use called Basel now for build process So we have quite extensive for history and working with Mono repositories and we also have history of Kubernetes So we started to use Kubernetes back in 2016 and quickly found out Two things first thing is ain't easy So it was quite complicated to manage multiple clusters and the second thing is we most likely it's really hard to manage large Kubernetes clusters and We started to think about how we can manage and create multiple smaller Kubernetes clusters and Dissect them on the networking level and trying to configure in a way Different types of workloads work on different types of Kubernetes clusters in the latest systems We also have huge lab environments, which have spun so pound like the 200 Workloads Kubernetes workloads already old number around head 500 Kubernetes cluster up and running right now and half of them in the lab and the staging environment And yeah, so you have multiple clusters So how do you manage this multiple clusters efficiently and here's a Mono Rapa? I think is a great answer to that because you could have in the single repository Information and configuration of all your clusters in the same place You can have multiple Mono repositories if you can distinct them by some common Think like for example in the central we have multiple product lines Which are more or less separate like video and phone a messaging Contact center and voice over IP in telephony solutions. So we have Mono repositories for issues of slides, but They still are huge product lines. So what we do we basically do very simple Thing and it's worked for us great So first of all one thing I want to mention try to keep it simple, especially in large enterprises So what we do we just using customized with flux and we have our main branch Acted as a as a basis So we keep our basis all obeys and I come on things in the main branch and we have multiple branches and Folders inside this repository for different environments which top builds on this top Main branch, but what it gives us it gives us ability to infer the state of all our infrastructure and Will it will come a lot more important later when we will talk about how we keep it reliable But Essentially as it's very simple distinction and flux Helps you to basically as long as you get repositories performant enough and you can pitch changes and you can fetch State of the git repository fast enough from the multiple clusters. You're good. We're using custom github deployment our self-hosted one enterprise solutions, so We have multiple caches and set up and making sure our git repository is always available and performant But yes, this is important part of it but if you again a large company you really want to have your git repository and your Workplace with every developer is relying on to be always available Performant and well behaved. So if you already have it, I think it's a great way to scale Flux and only any github the repository simply without any complicated solution to Basically multiple installation of multiple clusters Excellent. All right Third point third need velocity. I'm sure that's important for all of you So imagine at ring central, right? They've got various teams. They got to move quickly. They can't Have slowdowns from miscommunications or people not knowing what the desired state is and human error has come up a lot in our topics of Conversation so git ops with flux provides this ability to have a single pane of glass for the desired state And that really helps with these issues of miscommunication and that helps them really reach a state of Being able to implement dev ops in their company, which means a shared ownership and end-to-end ownership So their teams even talk about the own design creation maintenance testing to even support tickets and That means there's no separate delivery team. They're not detached from the production process So how does flux help with that flux has a lot of capabilities with observability and notifications on through Its own use and and integrations that our community has worked on I love the quote that even was thinking about that flux makes Kubernetes easier. So if you're You know having certain challenges will talk about how flux can help with that So one of the basic things that flux can do is it has events And so you can get notifications through the platforms that you use such as slack or discord or others that we support And we're also increasingly working across various integrations. So flux is moving into your IDE So if your app developers are doing deployments, they can do less context switching and they can just be within one place So we've started with visual studio code. How many people use visual studio code or people in there? Excellent. So you should definitely check out. It's our get ops extension Kingdoms here. Who's working on it? It's really fantastic. So you can literally Manage flux and do your deployments from within an IDE and not have to context switch If you like you eyes, there are options There's a free and open source. We've get ops UI that we have out there You can just check it out and I think it's really important to mention here that you can also use tools like arc Kubernetes EKS anywhere and D2 IQs platform and so Microsoft Amazon and D2 IQ have chosen flux and they trust flux as The tool that delivers get ops to their end customers So if you're already using those clouds, then you have those options as well So that the cure code will have links for you to check them out So I think that's it Okay, so I'll try to keep it very short because we have not much time and I want to leave it for then so Very important flight site against sigway into the velocity is a DevOps. So as it develops So I'm very heavy advocated DevOps and DevOps not I think it's misunderstood term in many terms I especially in many ways especially in enterprise because right now, but not right now But before we had so cold in so cold. We still have DevOps engineers. So the people who Try to build infrastructure and try to support the infrastructure for our development teams and we have around 3,000 developers since 3,000 engineers and about half of them was on the supportive role so it was not not not way were not software developers and Like big bunch of them was developed the ops engineers It outlines a big problem because if you need so many engineers specific engineers to maintain you infrastructure something is wrong and For me DevOps was always been a shared responsibility within the team and shared responsibility for the product So the DevOps team as a team who owns the product who makes the changes to this product for maintain it and deploy it So when we're talking about github and the flux and all other solutions We bringing this infrastructure closer to the development So we were able to simplify infrastructure management We were able to make it in in a way It's easy for our developers to work it because it's a source code it git repair monorepositories all the familiar way and It makes So basically I think it make the DevOps it like github's and flux the main DevOps What it's meant to be and then what's it evolved right now, which is like kind of some kind of abomination from my point of view so And it of course it increased sorry about it increased our velocity Sometimes ten falls when we use like other practices when you have this waterfall process or different agile processes and Pipelines so we do not have any pipelines in the company right now when we use github's and it made our delivery Like with 10 times or even 20 times faster. So it's it's a great tool and DevOps is a key Practice to utilize when you're working with github. So you always should think about Awesome. All right, and the fourth need reliability, which I'm sure is important for all of you as well So here again, we'll talk a little bit about flagger, which is a sub project of flux And this is really important at ring central because as we talked about millions of users hundreds of thousands of businesses These are business critical needs. I mean their telephones have to work, right? So these are things where they need reliability Built into the infrastructure They also need deployment processes that are verifiable testable and safe And we'll talk about how exciting it is that they can create policy-led environments for your developers to Be able to do the things that they need and know that they have the guard rolls up And so the human error won't come into play So flagger the sub project that provides progressive delivery Which I really love about this story. It helps users if all of you are kind of on your get-ups journey We're aware that there's a lot of changes that you have to make or your teams have to make even in the way that you're thinking Like things can feel new and unsafe So flagger really helps with that and it's really great to hear people out in the wild feeling that safety net for errors because Flagger provides so this term is fairly new progressive delivery. It's an umbrella term for things like canary deployments blue green AV testing etc And what it does is it decouples the app deployment process from the release process so basically if you have a service mesh or ingress controller flagger can use that to Decide to route traffic from your first source and then slowly dole it out to the new one based on metrics of success or failure and Roll back if it doesn't meet that threshold So basically flux will notice the change and start the process the get-ups process But before Kubernetes can take over flagger will come in and say okay the user set thresholds for me I'm going to take metrics for example from Prometheus or datadog or whatever and based on those metrics I'll keep checking, you know every 10 seconds, you know I'll route a little bit of traffic are there errors. Am I within the threshold? Okay, I'll route a little bit more and a little bit more until you've completely moved over or if it says no I haven't met the threshold it will roll back And so what's really great about this as Stefan has said is like you can deploy on a Friday and know that if anything goes wrong You know, it's not going to Have you be on your pagers on the Saturday? So you'll talk about how that helps with you. Yeah, so very important thing about the flux So if you're just using flux and just using get-ups. It's very important to understand It's a very powerful tool and if you approach it blindly It can make things worse because you can make a change we can make changes which bring your whole production down especially if you use monorepus, which is counterintuitive, but still so What you want to do is to build a Like set on the forcement of rules which you enforce upon the developers and in enforce upon the process in a way We cannot avoid it. So what I'm talking about. So I have I had not much time But I will bring one simple example of enforcement is a high availability and chaos engineering So when you build your system, you have this non-functional requirement of high availability Every like everyone needs to be high available cool But if you just write it a non-functional requirement and do not test it and do not run it an environment Which is inherently unstable you will miss it You'll most likely it will be just high availability on paper which will not work when Something keeps the fun. So what you do you implement chaos engineering. So we use the litmus for that purpose and You bring you you make sure you all your environments even production one. So you run house engineering from production Sounds crazy sometimes what it is You make it inherently unstable and developers have no way to avoid this non-functional requirements the same way it goes with everything else and when we build our systems and when we're trying to Like basically make it universal across a large enterprise We want to have this enforcement rule these best practices to be shared and to be enforced it upon everyone. So this is I Think it's the most important part of success if you do not do it. You most likely will either break Your management will stop it. Something goes wrong People will not understand that we'll have too many ways to do stuff and it will get a break So again, we as I said we have runtime and configuration domain for configuration domain We run GitLab State like it's not pipelines just a single verification with GitLab runner But we have custom because we have more than repository like I said before we can run static analysis on this money repository We can detect drifts in Configuration for example if you have different FQDNs with different services which aims each other You can statically check and analyze what the QDN is the correct one Or it leads to some another resource which will will not exist in the class You can run policy management and policy enforcement using cavernet for specific resources exist And we also have runtime so configuration is not enough because during runtime things can have go wrong people make mistakes Yes developers may especially makes a lot of mistakes So we use flagger for that purposeful but with question asked about red green blue We have canary deployment. We have progressive deployment different types of deployments by using flagger Which flagger support and we tied it with test cube So it's run test cube checks in all our environments in lab and stage and production is very important And it makes sure what when we deploy stuff and the developers deploy stuff We are not afraid so we build trust into the system and when you build trust into the system when we started basically Trusting what we can really deploy on a Friday We start to deploy stuff on a Friday because we are not afraid to break stuff and then and I last finger in central Have five nines SLA for availability. It's a six minute lesson six minutes of downtime per year So when I'm talking about the stuff, I mean like we're really really serious about it So, thank you so much. I think we are Yes, no problem. So just a summary. So security compliance scale velocity and reliability Again, if you didn't take this QR code, we've got plenty of talks throughout the week. So We also have the slack channel. So we'll be outside if anybody has follow-up questions Or if you want to follow up questions in slack, please do that We also have our email address And you can email us if you have questions So, thank you so much, and I hope you have success with flux and flagger Yeah, you can ask questions me directly later. I valuable I whole day. I will be holding Thank you