 Hey guys, please slow down. We're about to start Okay, next up John Frizzell and the topic is from Monolith to microservices in 10 steps There we go. Thank you very much Okay, good afternoon. Can't everyone hear me? Okay? Okay, I'm going to take silences. Yes. So my name is John Frizzell. I'm platform architect with Red Hat mobile joined Red Hat in 2014 as part of an acquisition of a company called feed Henry and Had been working with them since 2008 on a mobile application platform So I just want to get a bit of a feel for the type of audience we have So how many people are familiar with the term microservices and feel they know what it means by show hands Most people okay good How many people are actually implementing a microservices architecture or have played with microservices? Okay, it's about half and how many people are doing microservices proper microservices architectures in production today Okay, but what I expected very good Okay, so a little bit of background so 2008 we started with a single Java monolith and That's what we had about 2011 when we started looking at node.js as an alternative technology Got really excited about node.js And we thought hey, this will be really cool for microservices So let's go and build some node.js microservices We went off for about a year and we came back with a node.js monolith so then we had a Java monolith and a node.js monolith and an Incredibly symbiotic codependent relationship between the two of them So we learned a little bit from that we moved on 2013 2014 2015 and actually did some proper node.js microservices, but they were still very much sitting behind these monoliths and only really doing small pieces of functionality So last year We figured we'd learned almost enough to really tackle the monoliths and say right It's time to kill the monoliths and really move to a microservices architecture So this is the story of the journey that we are on we haven't finished the journey I can't say whether it will be a success or not, but this is the journey we're on So first off, why do we want to kill the monoliths? What's the problem with them? So typically with a monolith you're looking at horizontal tiers. You have your UI layer your business logic layer and your database there and That doesn't really represent a System very well because any piece of functionality you want to add any new feature Typically has to be added at all three layers So even though you separate horizontally You're touching all the layers almost every time you want to do anything. So you pretty much have them tightly coupled With the monoliths you end up with a very large code base It's difficult to keep in your head the entire code base in the context you're working on and Depending on the size the monolith it can even be difficult to load it into an IDE The the sheer volume of files and lines of code can make it very difficult to actually work on the code base Obviously with the size of the code base you're looking at It's going to be much slower to build it to run your tests and to deploy it to production You're very restricted in terms of the technology selection you make your technology choice day one when you start a monolith whether it's Java or Node or whatever programming language and you don't have Any real freedom after that because you're building in a single code base. So you're building in that technology stack and Then when you find a part of your system is running hot and you need to scale it You can't scale just that part you have to scale the entire monolith and that's not really very effective So in theory microservices to the rescue So what does microservices give you it gives you great agility and flexibility to change You're working with much smaller units of code that you can build deploy and ship independently Smaller code bases mean less context for developers You don't have to keep as much in your head You're working with a smaller service that ideally is doing one thing and doing it well. It's a lot easier to understand the code base With smaller components should come smaller teams. You don't have to have everyone knowing everything about the monolith You can have smaller teams focused on certain areas of the different microservices It becomes much easier to scale you can pick the microservices that need to be scaled Horizontally or scale vertically and just scale them leaving the ones that have low volumes of transactions at a much lower level of scale and you can pick the right tool for the right job, so if you need to implement it in a Typed language pick a typed language if you need a relational database pick a relational database If you need a graph database or if you need a functional programming language You have the freedom and flexibility with each new microservice to choose the technology that best fits your use case But you don't get this for free microservices come at a cost and as we've learned they come at a fairly high cost You're moving the complexity out of the code in the monolith and distributing across the microservices But you're also building a much more complex architecture overall You're moving from functions calling functions within a single process to inter process and inter service communication Suddenly you have to start hanging about network overhead HTTB buffers slow responses what happens if one service goes down and another service is calling it and They the call services unavailable You can very quickly find cascading failures coming up through your system But probably the biggest challenge with microservices is the shift in thinking and the organizational structures that you need to make So Conway's law Sells that an organization will build software that mimics the communication structures within the organization So if you have a large team all communicating together and you try and build microservices You're going to have the same experience. We had back in 2011. You're going to build another monolith To truly do microservices and do them properly. You actually need to start looking at an agile transformation You need to start looking at smaller teams With all of the skill sets needed to build and ship the microservices working independently of each other and If you don't make that transformation and that is a hard hard transformation to make But if you don't start by thinking about your organizational transformation you need You're going to be in trouble from the very beginning You also need a massive amount of automation You're suddenly going from one or two large components that are probably shipped in frequently and Could be hand-carved for deployment You go from that to 10 20 50 a hundred a thousand microservices The architectural structures are simply too complex for a human to be able to manually deploy them you need to automate everything and They really are not a silver bullet. They're not going to solve all your problems. They're not going to a to fix everything They work very well in certain use cases, but they're not suitable for all use cases So you need to really make sure you understand what problem you're trying to solve Before you decide whether microservices are what will solve them So a couple of words of caution before we move on So a colleague of mine on a mailing list recently Said this Microservices buzz is a lot like teen dating You're doing it because it's exciting and everyone else is doing it You don't really know what you're getting into but you want to do it anyway And my time you figure out what it's all about. You're already in a committed relationship So bear that in mind don't get caught up in the buzz So I think more controversial one from Martin Fowler who would be considered quite an expert in the field of microservices He says don't even consider microservices unless you have a system that's too complex As a monolith or start a greenfield development with a microservice architecture and that's a hell of a statement to make But when you think about the overheads and the work you have to put in upfront To set yourself up to succeed with a microservice architecture. There actually is something to consider in that So final word of caution before we move on If your monolith is a steaming pile of you know what and You don't think carefully about how you're going to break it up What you end up with is loads of little piles of you know what Okay, so assuming haven't scared everyone off doing microservices Couple of kind of rules of engagement to think about when going after microservices first off Vertical slices are the key this horizontal tiers Not what you're going to do You're going to take a piece of the system You're going to split it out from your monolith from top to bottom from presentation layer true a API true business logic to data storage. You're going to take it all and split it out How do you figure out how to do that? You follow the principles of domain-driven design and Boundary contexts That's not a technical thing. That's a process thing. It's a people thing. It's talking to your customers It's talking to your product folks. It's talking to your BSAs it's talking to your engineers and it's figuring out what does my system actually do What are the fundamental things that my system does and where are the boundaries between them? Is it an online system your boundaries are product catalog shopping cart? Customer addresses In our case with a mobile application platform system. It's things like projects Mobile applications back-end applications zero code applications each of these is a bounded context within the system and until and unless you identify the bounded contexts what you will find is Services calling services calling services calling services if you don't find those fracture points within the monolith and identify them correctly you will end up with a mess of inter-service communication and you really want to limit that because inter-service communication is Difficult it's costly you have the network overheads you have the problem with systems going down and You're basically just making things harder for yourself Now you won't be able to get away from inter-service communication all together The way you do that is you stick with a monolith So assuming you do break up into microservices. You are going to find services needing to talk to each other typically the first approach people take with microservices is HTTP and rest if you go down that route You have to code for failure. You have to assume that the services you depend on are going to be unavailable They're going to be slow that basically they're going to do everything they can to make your life a misery so Do not trust the other services use things like circuit breaker patterns to Guard against services either giving slow responses or no responses So that if it's service you depend on goes down It doesn't cascade the whole way back up to the user and before you even start thinking about my microservices start thinking about distributed tracing and logging so we've had a couple of very good talks in this room already today on elastic search fluent D cabana For tracing and logging start thinking about that stuff up front because if you have 200 services Each of them deployed across three instances that 600 instances if all of them are logging through their own log files Well, it's going to keep you up at night trying to find where things are going wrong Okay, so we now get on to the actual purpose of the talk which is the 10-step plan to breaking up the monolith Steps one to five are your prerequisite steps. You do them before you ever touch the monolith Steps six and seven are the steps you iterate on to kill the monolith and then eight nine and ten are Optional or follow-on steps that you need to consider depending on the complexity of your system So let's look at each of these Step one instrument you have to understand what's happening in your existing system Doesn't matter if it's a single monolith or if it's four components or ten components you need visibility Into what's happening in the system. It can't be a black box as I said you need centralized logging You need distributed request tracing so something like zipkin and you need as much Information about what's happening in the system at the moment as is humanly possible. You want metrics you want analytics You want alerting information when thresholds are crossed and probably most importantly You need information on API usage because as we start breaking up the monolith Excuse me, we're going to start creating new APIs and we're going to start routing old APIs to new services So you have to have visibility into what's happening at the front door at the API layer Once you have some instrumentation in place You then need to think about an intelligent router an API gateway that will sit in front of everything in your system a single point of access for all inbound communication so this router acts as a bulkhead between your clients and Currently our monolith and what will become your micro services It acts as a central point where you can do things like gathering additional API usage Where you can start as you as you move micro services out from the monolith and you want to start Rewiring old APIs that would previously have hit the monolith to route them to the micro service. You can start doing Root mapping and data rewrites So typically when you implement a piece of functionality in a new micro service You will do it with new APIs Hopefully cleaner better APIs But the request and response payloads will not necessarily match what your monolith had So the API gateway that the intelligent router in front is used to transform the incoming requests for the old APIs and Transform the data on the way back out So that you can continue to support your old APIs using a new micro service other things that you may want to do in there would be static file serving caching authentication Adding request headers adding unique request IDs that would travel the whole way through the system Which will be used by your logging infrastructure So that when things go wrong and you see that request ABC123 failed You can go to your Kibana dashboard and you can query on that request ID and see through all of the micro services That hit and the monolith What actually happened and where they failures were Now you're putting this thing in front of everything if it goes down Everything is down. So it has to be highly available It is a single point of failure. You need to think about that up front You need to make sure that it is as highly available as possible. So what your router in place? it's time to start thinking about versioning because You're going to be moving a lot faster in a micro services world You're going to be building services. You're going to be destroying services. You're going to be changing API versions So think up front about the versioning approach that you're going to use What we found is that the sem var semantic versioning approach works very well the x dot y dot z and With semantic versioning it allows you to communicate just through the version numbers whether or not you have breaking change and minor change or a patch change You then need to start thinking about The version that you're using in your public API's How does that map to the versioning of the micro services that you're deploying? If you make a change to a public API or you make a change to a micro service How do you root requests for the old API to the new micro service? Do you have multiple instances of the micro services running serving different versions of the API? Do you have conditional checks in your code of your micro service so that you can be backward compatible against your API? There's no one simple answer It really depends on what your system is and what your use case is but you got to think about it up front Because if you don't it's going to come back and bite you so stop and sit down with with your development team and Figure out what your versioning strategy is going to be and play through a couple of scenarios. How do I upgrade this? What happens here? How do we deprecate thing through all it is? Come up with a plan and a process and actually write it down and make sure everyone's on board Okay, so then the next thing we want to do is We want to come up with a template for our ideal micro service If we're going to be building lots and lots of these we don't want to be hand crafting them hand coding them what we want is a template that Takes all of the heavy lifting for all of the utility stuff That's not part of our business logic or our domain context and Gives us that in a standard way for every service. So things like your centralized logging configuration management the request tracing security analytics Monitoring service discovery all those things that every single service will need database connectivity all that kind of stuff build that in a template and Then generate instances of that template for each new micro service As part of doing that Figure out an approach for API generation. So you have the template you create an instance of it And you want to start creating APIs for a specific micro service Please please please do not write them by hand Identify tooling and technology that allows you to do API generation The reason for that is you not only want to generate your API You want to generate documentation on your API you want to generate stubs you want to generate contracts that can be tested You potentially want to generate clients that can be embedded into other micro services to allow them to call your one So think about tools and approaches for API generation rather than going the route of writing to my hand and then Automate absolutely everything as you start building the services You're going to need more and more automation and if you haven't really started thinking about the automation up front You're basically going to find yourself grinding to a halt To go back and have to retrofit that stuff So you need proper continuous integration and continuous delivery You need Visualizations you need to be able to see to say to your system as the number of services grows It becomes more and more difficult to kind of keep in your head. What's happening? so think about building out dashboards like some of Grafana or Kibana dashboards for viewing and Ultimately managing what versions of what services are deployed where and seeing the health and the state of them Invested time in being able to do one-click deploys into your test environments and into your production environments And that's not just one click deploys of an individual service That could be one click deploys of an entire product upgrade which could be a suite of services Well, yes, we can release individual services independently and it's one of the big benefits that's touted with micro services in reality very often for significant new product features You will be touching a number of different services So you will want to coordinate the release of a suite of services into production So start thinking about that figuring out how you're going to automate that how you're going to roll it back How you're going to verify that apply has gone out correctly? invest time in tools for an operations team if you have a SAS product To allow them to manage what's happening see counts of errors to see insight into what's happening logs stats performance all that information that Maybe kind of more manual at the moment You don't want to be SSHing onto a VM or into a container to run top to try and figure out what's happening You need to start centralizing that stuff and invest upfront in the tools and technologies needed to allow an operations team to successfully manage a micro service architecture and Make it easy to add new components That's actually been one of our biggest barriers we have a SAS platform and there's quite a lot of Steps and processes didn't need to be followed to add a new component to the system In terms of what user does it run out? What access does it need? Where is it logs? How is it? How is it monitored? There's I think it's been an eight-page document our operations team have did you have to go through for adding a new component? You have to codify all of that You have to make sure that it's as simple as possible to add new components because if it's not When you're under the crunch from your product team saying we need this feature You're going to end up putting it into an existing service because the barrier to creating that new service is too high That's how we ended up with our second monolith We didn't invest in making it easy to add new components upfront because at the time We didn't realize that we had to this. This is all lessons learned over this journey Okay, so at this point we're actually ready to start strangling our monolith And what that means is we're going to apply what's called a strangler pattern Which is we identify a piece of functionality Using our domain-driven design approaches and our bounded context We're going to identify a piece of the system that's well bounded And we're going to reimplement it as a microservice We're going to deploy the microservice We're going to add route mappings to our router to our API Gateway to start routing traffic to the new service And all of a sudden there's a piece of functionality in the monolith. It's no longer getting called Or at least no longer getting called externally Because as part of strangling we need to think about well, what happens our data Are we going to take the system offline for a while and do a batch conversion of data to get it out of the monoliths database And into whatever data store we're using for our microservice Because remember one of the fundamental principles of microservices is that it is a completely Self-contained stack it has its own data store microservices should not be sharing data stores with other microservices During that initial period when you're translating or migrating data You may make a decision to connect directly to the monoliths data store to suck all the data out and move it into your own one That should be a really temporary short-lived thing And ideally that code should not even live within the microservice. You should build a separate service just for doing the data translation You may for some reason Need to talk back to the monolith through some internal APIs to get access to pieces of data That's okay, too This is the real world. I'm not going to propose a Idealistic solution that won't work You may actually have to talk back to your monolith for some of your data But if you do Isolate those communication paths as much as possible both on the monolith side and on the microservice side To make sure that when the time comes and you no longer have that dependency That you can remove that code very easily Deploy early and deploy often Identify the first bounded context split it out and get it into production Do not be tempted to try and change the monolith into a microservices architecture in one big bang Because you try and do that and the only thing you're guaranteed is that it will be a bloody big bang And it will not work. Do not try and do that Identify one piece extract it get it live and iterate iterate iterate this might seem like a little bit of a no-brainer, but We all know we don't do documentation very well, and why do we need good documentation and microservice architecture? Well one of the primary reasons is as we move to microservices architecture. We're also looking at transforming our Organizational structures. We're looking at transforming into a single large engineering team that does probably a waterfall based methodology Into much smaller teams They're operating using an agile methodology and what that means is that not everyone is going to know very much about each microservice and You don't want to have Interperson dependencies to understand how to interact with someone else's microservice The service itself needs to have good enough documentation That one engineering team one agile or scrum team can use another microservice without needing anyone from that team to actually help them out So you have to have good documentation on your APIs Back to a point I made around templating Don't write the documentation Generate it if you're relying on engineers to write documentation You might as well go home now So figure out an approach that allows you to generate the documentation In the same way in using the same tooling and approach that you use for generating your APIs Make it as easy as possible to do API discovery consider using something like swagger and For anyone who has written swagger documentation. It's another damn good reason not to write the documentation by hand It's horrendous, but it's a very powerful tool So if you can generate swagger from some metadata that you put in code blocks with your functions Then you can make API discovery incredibly easy swagger provide some really brilliant tooling to allow Service discovery of other services and actually interact through a swagger console with the API in real time At the same time you're considering the generation of documentation as I said already You're looking at generating clients that can be embedded into other microservices to make it as easy as possible to call You're looking at generating mocks so that you can do Isolated testing and integration testing between a small number of services without having to stand up the entire system and Stubs if you do your Documentation generation properly you can actually do contract testing on your microservice almost for free Excuse me. Yeah, there's a little bit of initial upfront setup But once you have that set up out of the way and you have a process and you have tools in place for doing this generation Then you start getting a huge amount of benefits very very quickly Okay So here's where it starts to get really fun and by fun. I mean complex So HTTP is a good starting point for moving from monolith to microservice But at some point, I'm not sure exactly what the number is probably 10 to 15 services Point-to-point HTTP communication becomes a bit of a bottleneck You have problems with service discovery So when you only have a few microservices You can put in the host name of the service you want to talk to into the client service As that grows it becomes unwieldy on an unmanageable. You need to start thinking about service discovery through something like ETCD If you're in a containerized environment Kubernetes is using ETCD behind-the-scenes to do that for you Or you start thinking about moving from point-to-point HTTP communication to using same message bus technology And it's at that point that someone will invariably start talking about event sourcing and CQRS It's a whole other area of discussion That I'm not going to get into here, but suffice it to say It's an area that you should do some reading in Before you start on the microservices journey to understand what the different options are in terms of how services communicate How you manage things like consistency cap theorem eventual consistency It becomes a lot more difficult to guarantee consistency and to do transactional programming With a set of distributed microservices So if you have a strong need for transactional programming then I Would suggest moving this into your prerequisites and figure it out upfront And make sure that the architecture and approach that you come up with For inter-service communication and sharing data between services will work for your business use case so up to now I've talked pretty much exclusively about from the API layer backwards But what about your UI? It's kind of the elephant in the room this big hulking monolithic UI Possibly your UI is a part of your overall monolith Hopefully not, but if it is the first thing you're going to need to do and This may sound counterintuitive based on what I said earlier on But you're actually going to need to split the UI out Into its own horizontal tier before you can start addressing it in a microservices architecture with microservices principles it says that the everything for the service should live in the service and That includes UI elements, so that's okay. Don't worry about it. Sorry So How do you go from a huge monolithic tangled mess of UI code to a modular UI You need to start thinking about Frameworks that make it easy to plug components together Things like angular and react have some very nice models for encapsulation of functionality within components But that's a big big step and it's kind of at this point that you'll be asking yourself is the juice worth the squeeze Am I happy to have the one remaining monolith of my system be my UI? And the answer might be yes It may be a case that you know that there's a major UI Refactor rearchitecture coming up for some other reason Use that as the springboard for moving your UI into a more microservices approach that is a big big undertaking it's a big technical undertaking and It can be hard to justify to your business people what the return on investment there is unless your UI is In a state where the business are crying out for improvements and you can use that as a kind of a leverage point to move to a more modern technology that supports putting the UI elements into the microservices You're probably going to find this one as quite a hard selling point When you're talking to your business folks so a final then final thing then to think about is How we actually deliver software both how we deliver it to the clients that are using our platform and our Engineers become more and more efficient at delivering microservices into the wider ecosystem So drawing on some of the experiences of Netflix When they first started going down the microservices route when the first things they did was they put in this API gateway And it was a generic API gateway that all of their clients used And as Netflix started offering more and more services across different devices and different device types They realized that a single API gateway just didn't work Because you were trying to cater for the needs of many different device types needing different data Different processing power on the client side different communication protocols through a single API gateway So what they did was They split the API gateway into multiple smaller gateways targeted at the actual clients that are using them So if you have only one type of client if you're only serving a web application You don't really need to worry about this, but that's very rarely the case these days You will likely have at least web and mobile You may have some form of native desktop application. You may be playing in the IOT space You're likely going to have more than one client wanting to consume your API eyes And once that number starts to grow probably beyond two or three or you find yourself Putting more and more conditional checks in your API gateway to satisfy the needs of the different clients It's time to start thinking about breaking down the API gateway into a set of smaller gateways that are targeted at those specific clients Then in terms of what our developers doing so initially Small number of microservices as we're going through the strangulation approach. We're strangling our monolith We start with a small number of microservices. You can run them all locally As that number grows and depending on the size of your system You're quickly going to find that a developer setup a local development environment becomes an arduous Problematic process you're onboarding someone for three weeks just to get a development environment set up because you have so many moving parts At that point you want to start thinking about pushing most of your microservices onto a remote hosted paths something like OpenShift for example is a very good approach there and then introducing Inbound proxies that allow you to pull just the pieces you're working on locally and Develop on them locally with the rest of the system hosted Makes for a much simpler cleaner smaller easier development environment and then once the piece of development is finished and You commit and it triggers all of that nice automation and CICD stuff You have and all your good testing your build passes It automatically deploys that into that shared hosted environment and then everyone else is able to pick up on your work Which pretty much zero communication needed? So in summary It ain't easy and it's going to take a lot of time There's a huge amount of upfront work required if you want to succeed You're looking up months of upfront work thinking talking building engineering To set yourself up to succeed if you don't put in the upfront work If you just jump straight in with two feet like we did You will have the same results we did which was more monoliths The complexity that it's quite easy to visualize within the monolith Doesn't disappear when you when you break it up into microservices. It just moves You put a little bit of complexity into each microservice and you put quite a lot of complexity into the overall architecture They're not a good fit for everything if you're going to go down this route Don't do it Because it's team dating do it because you actually need to do it and you understand why you need to do it and Finally and Probably most importantly you will need to change your organizational structures if you want to succeed You will need to look at linking monolith to microservice from Sorry linking monolith to microservice with waterfall to agile and if you miss that step then a lot of the rest of it Probably doesn't matter very much and that's it. So If I haven't scared everyone off Do we have any questions? Yes Sorry, have we tried using a Grab QL. Did you say no we have not I'd be interested to hear a bit more about that though So we might have a chat afterwards Okay, so I've either put everyone to sleep or made everyone so depressed about microservices that there's no questions. Yes. Thank you So the question is how to detect that your development Patterns are leading you towards another monolith Your team structures your organizational structures Basically, if they're not set up For a microservices approach, then you're not going to end up with microservices. So if you're not looking at Breaking a subset of your your development team out into a small team to focus on building a microservice or a couple of microservices And then repeating that pattern over and over the agile transformation basically if you don't do that piece then That's a pretty good indicator that you're not going to be successful I mean, there's obvious things like the number lines of code in it the size of it Is it doing more than one thing is a crossing that those those boundaries of context within your system and Only did they creators and maintainers that a system can identify what the boundaries are So if you see what you thought was going to be a microservice crossing boundaries, then you're heading down the route of a monolith again Okay, so the question was there's obviously a lot of tooling and building required Do we have a build engineer within each team? to set up the build process Typically the approach we use is we've teams about six state people Usually four or five engineers one or two quality people and one kind of operational mindset person But we also have a separate team that's solely focused on tooling and automation I'm providing the the underlying foundations. I'm making sure our CI CD functionality is Working the way we need us on the the templating this needed for creating the microservices on The operational insights and dashboards that we need for deploying. We have a dedicated team That is responsible for a lot of that support work And then we have the right skills within each of the smaller microservice teams To be able to build on top of what that tooling team provides Okay Yes Okay, so the question is is there any particular a specifically test tooling? So is there any specific test tooling that we're using to help test the microservices? I Guess it depends what level of testing unit testing integration testing contract testing Availability Is there any particular testing tool we're using for availability? No, we have talked many times about they I don't know as a Netflix the idea of the Chaos Monkey testing where you have a tool That goes and randomly kill services The system is not quite ready for that. We're going to get there, but at the moment we rely more on monitoring Automated monitoring and automated deployment rather than something as aggressive as Chaos monkey type tooling that's killing processes Yes Which use cases aren't suitable for microservices? That's a very good question. I don't have a very good answer Sorry, yeah It's it's probably easier to ask which ones are suitable for microservices And typically it's if you're hitting a point and in particular in our use case if you're hitting a point where Developing and maintaining in a monolithic code base is slowing your productivity down to such a pace that You can't ship features and functionality. You're missing deadlines quality is suffering When when the size of your monolith grows beyond a certain point You need to start looking at breaking it up And that would be the kind of primary trigger that I would suggest that people think about it in terms of when to adopt microservices Because unless you have a lot of money behind you and a very very Understanding business owner and product manager They're not going to want to have you invest the time and energy and money In setting yourself up for microservices architecture for green field It's typically when you hit a point where productivity has dipped because of the monolith That you start looking at maybe this is the right time to look at a microservices approach And if the organization is open to the change required to do it Okay, so we have time for one more question Yes one big Q a team So our Q a team is fully integrated into our agile teams So each team is testing they the services and the components that they're responsible for We do still do product releases. We do synchronize the deployment of services. That's something that that we still need to get away from And typically when we are doing a product level release The teams that have been responsible for the feature development in that release will come together to do the end-to-end product testing So you need to have the Q a folks Fully integrated into the teams and working directly with the teams on the microservices They they kind of throw it over the wall approach is simply not going to work Yes, exactly exactly No, okay. We're done. Thank you very much on mute