 So, I'm going to talk today about microservices. Who here has heard about microservices? Who here has heard about the second most frequent buzzword in the last 10 years besides maybe Docker? The talk is going to be a little bit less traditional than I think you'd get other places just because I'm going to be talking about my actual experience in going through a project that has converted from a monolith to microservices. And I'm going to try to focus it a little bit on this audience, which I believe, and I hope I'm not making a bad assumption here, but I think that for this room, generally I'm speaking to people who are on QA teams that have been going through the convergence of microservices, whereas generally when I give this talk, I give it in companies where they're about to make the decision to go to microservices. So this is a little bit skewed. If I apologize if I'm all over the place and they get stuff a little bit jumbled, it's my first time with this format, but I want to try to express things in such a way that you can anticipate the changes that might be coming and that you can ask the right questions in order to get your thoughts together and get organized before the journey begins. The other assumption I'm going to make is that this is a going from a monolith to a microservice project as opposed to building a microservice project from scratch, i.e. yet to meet someone who's doing that. If you're working for a startup, you're QA number one, your company's probably starting with what cloud infrastructure do we use and how do we architect this and the microservices. My assumption here is that almost nobody in the room is going through that process. You're in a company, there is a giant ball of mud sitting in GitHub or your own Bitbucket repo, and it's a nightmare to work with, and your company has decided for right or for wrong that you're going to break that up into small pieces and let it work together. Now, if that assumption out of the way, that's how I'm going to conduct this, and I'm going to start to talk a little bit about why microservices, but I'm also not going to dig into that too much because that could take all day, and we could get into a giant religious warfare over why microservices. Are we overdoing it? Is this the right way to go? I'm going to dive into that just a little bit, but mostly I'm going to talk about here's the world we live in, and here's we as QA generally being the last to be told about stuff that's going on. How can we anticipate the world that we know we're going to be living in? So why do people choose microservices? Does anyone know what RDD is? Is Simon here? RDD is a blank screen. I don't know what just happened. Resume-driven development. People here about microservices, they think, oh, I'm going to do that, and they think it's great. For me as a director, I'm going to break this up into microservices because that's going to look fantastic as bullet point number one for my next job. So getting past that, when you go through microservice planning, you do like any other project. You go through planning, you have questions to ask, and then you go through development process, and then you deploy, release this project. In the end, I'm going to tell kind of an epic story about what happens when you have successfully engineered your project over a five-year period into microservices, and then you realize, oh, wait, there's one thing I wish we had done before we split everything up, because now we have to make a change to all of the microservices, all independently of each other, now that we've broken it up. Would have been so much better to have this done in a monolith. So I'll be telling that story at the end to tell you that's probably going to happen in your project, and here's how you can be a hero in making sure that it goes well. So planning, once again, it is possible, likely, that you are not on the forefront of the planning of these processes. Neither was I as a QA person. But these are the questions that are being gone over ad nauseam within the management confines of whatever company you work for. So at the director and VP level, here's what they're considering. Are you going to rewrite the software, or are you going to make incremental changes as it goes along in the process? Joel Spolsky, who has been quoted at least once today already, says that rewriting software from scratch is the single worst strategic decision a company can make. I'm not going to judge that statement's veracity. I will tell you that that's a fairly well-held belief. I have been through several rewrites, and what I can tell you is that from a developer's perspective, they're fantastic. You learn so much. You get to do something from scratch. You get to make something in your own image. It's almost like childbirth. You get to have this wonderful cathartic experience, bringing something new into the world. But the three times I've been in a project like that, I've learned so much it has made my career and it destroyed the company I worked for. That is a true statement. So I am not going to say whether or not it is always the single worst strategic mistake you can make. I could just say that three for three in my career, it was great for me. It was terrible for the company. So probably don't do that. Now the management chain is probably having a two-month set of discussions with the architects and everyone where they come to the same conclusion where they could just skip to the end and listen to me, but whatever. They're going to have those deliberations and they're going to go along. And someone's going to show them this chart right here, which is sort of a timeline of what happens when you rewrite. Black line is the rewrite. Green line is what you happen, what you have when you do it incrementally. You're going to change things in the green line. You're going to make incremental changes, add business value, incremental changes, add business value, incremental changes, add business value, where here the black line, the revolution as opposed to the evolution, the revolution is encompassing things like opportunity cost. If you're spending all of your time rewriting your software, you are not adding new features. You're simply breaking down old features and then trying to achieve parity. So this sort of describes the timeline. Let me put this a different way. Here's the business value over time evolution where you're going to take a nice smooth dip. You're going to lose productivity. You're going to lose a little bit of business value, but eventually at the end of the project, you're going to have increased velocity. You're going to have increased business value. Here's what happens in a revolution. You're going to break it down. You're going to rewrite it. Now, productivity dips significantly. At some point, you reach parity. What I didn't put in here was developer morale, which is way high at the top in a revolution, and then it just drops, drops, drops. Around here, everyone quits, and some new team has to take over. Around here, the management team gets fired. I got to apologize here. My son, Alex, if he's watching, he would say, dad, why the hell did you use Comic Sans on that font right there? And I apologize, Alex. So I changed it just for you. I changed it to Papyrus, just for you. I just wanted to throw that in there. It's an inside joke for my kid, but hey, we like fonts. What can I do when my company is busy making this decision? What can I do? Because at some point, it's going to be handed down. Do I just go about my business? What do I do? I suggest you sit in every meeting you can. You become a part of the plan as much as you can. I understand that that is limited ability in some cases, but do it as much as you can. Educate yourself about the decisions being made about the different stacks. Right now, you're running, well, if you're me, you're running a PHP stack with CSS and JavaScript on a MySQL database back end, and you're anticipating that some microservice teams are going to be using Java with Spring Boot. Some are going to be using JavaScript with Node. Learn all that you can about the stack that you think you'll be working with the most, or about all of it. Certainly start to put a mental picture together of how this stack is going to look when it's strung up together. Now, establish a rapport with the developer's products, stakeholders, end users, and by the way, you should be doing all of these things all of the time, but certainly during the planning, that's when you have a good opportunity to do your most anticipation and planning thinking. Where does it make sense to break these things up? What kinds of decisions are being made at that management level? Are they breaking it up by functional category? Are they breaking it up by, so that would be a vertical functional category being end user login flow, and then you get into, in my case, you get into, I'm sorry, I didn't introduce myself. My name is Marcus Merrill. I am the staff quality engineer for RetailMeNot. We have a site that makes coupons. We put coupons on pages so that you can save money, right? Coupon flow is one area where I can assume that we would have a microservice stack, but there's also another way you can think about microservice stack separation in terms of its architecture. You can think of it in terms of performance. All of the things that need to be highly performance should be in one service. All the things that should be highly secure should be in one service. Those are decisions upon which you can make these different, those are things you can make these decisions on and you should anticipate and have conversations with the developers in the know about what kinds of decisions are being made at this level. Then you can get to the kinds of questions that you want to ask. Is it testable? Right now you're monolith. You know how to test it. You know what problems you have. You know where you can't test it. You know where it's a white box, where it's a black box. But in the new world, are you getting in early enough that you can ask questions about can you even test it at the microservice level? At the microservice level, you're going from a state where you have a UI that's talking to hopefully an API which is talking to a database. Sometimes the UI talks directly to the database. We all know that. But hopefully you have some sort of NBC architecture but sometimes you don't. And in the microservice world, you have probably one team is running Mongo for God knows what reason. One team is running MySQL. One team is running something else. All different database platforms. All different API front-ends. You have some REST endpoints. You have, well, probably not some SOAP endpoints. But you have all sorts of different ways you can interact with an API. And how can you test that? Can you add endpoints? Can you have developers add endpoints for testability in lower environments? We have a service called the ledger service which keeps track of how often our users have transactions. They can save money. We can add money to their wallet. We call it. And in order to do that, we had to be able to add money to people's wallets in the test environment. So we added endpoints called slash deposit dummy account. And it would just sort of generate a new account with the user with transaction history. And that in one step, we can set up all of our test data for testing that we don't have to wait. We don't have to worry about it. We don't have to figure out how we're going to mock up anything. We just add this endpoint for testing specifically. Does it go into product? Hell no it doesn't. We are able to build these things in just so that it's in the test and staging environment. Do you have access to the code? Let's show our hands. Who has access to the code? That's the actual code. That's a great number. I asked the same question I think four years ago in Bangalore and got a very different answer. I am very happy that we as an industry seem to be moving in this direction. It's about respect at that point. Do you have trust with your developers? Do they have trust with you? And it looks to me like we do. That's a good sign. Are there unit tests? Cannot believe sometimes the answer is no. Can you help add unit tests? Can you encourage a culture of unit testing? These are things that you can just start to think about. While these decisions are being made, they're being made in a dark room in the back that you don't have access to, but you can start getting ahead of these things. You have basic authentication in lower environments. This is a huge one. My company, we have CAPTCHA on everything. And it got in the way of testing. And we got all these questions coming to me like how do you do CAPTCHA with selenium? I'm getting a few snickers. I hope the answer to everyone is you don't do that. Do not bypass CAPTCHA using selenium. It is a path to evil and ruin that will eventually lead to you doing the same thing that we did, which is just set a cookie in the lower environment and it will bypass the CAPTCHA. Are we in agreement on that? So hopefully every one of these microservices will have a similar kind of thing, but in our lower environments what we do is we produce a basic authentication header that is sort of authorized to work with our lower environment services. We don't even have to worry about that stuff. Yes, you should absolutely have tests in production that make sure your authentication scheme is working properly. That is now what we're talking about here. Testing the stack you're looking at right this minute. Now the really big question that's probably going to be a far too common if I ask for a show of hands. How many of you are integrating with a third party vendor at some point along the way? I'm not talking about rally or test rail or team city or anything in your testing infrastructure. I'm talking about in my company we had a massive stack of software that did all sorts of stuff collating user data in order to produce the greatest email we could produce and send them that would produce the greatest lift and engagement so they would click on our email links and at some point we had to talk to our email vendor who shall go nameless at this point. Yeah, but their API was challenging to use. It wasn't always working. They did not have a testing environment. It was entirely incumbent upon us to maintain our own data and state and delete everything that we created during testing. Made things extremely complicated. If you have a third party vendor right now before microservice and there's a really good chance you're still going to have that same vendor in microservices. What you want to do is encourage your developers to isolate the interaction with that vendor to one service or one endpoint or something like that if you can and then try to isolate that so that you can either mock out that one call or at least get a group of developers and yourself who are true experts in how to handle that vendor including customer support contract contacts and all sorts of people you can interact with if things are going wrong. The problem with interacting with a third party in the monolith is that you have 100 developers, seven of which vaguely know something about the interaction with this vendor, and QA usually hardly knows anything except it's not working or it is working. So dive in to the third party vendor relationships and become the expert and that is definitely a path to making this transition a little bit easier on yourself. So another question is is the contract for your microservice published by which I mean is it on a common server that you can see like swagger or something like that. Does everyone know what swagger is? Okay that's about half, not bad. Swagger is the ability for an API to publish what you can do who you can do it to and what you're going to get back in return for every endpoint in the system. It looks a little bit like this. So if you have an API that deals with pets, you can add a pet by using a post, you can update an existing a pet by using a put. I would actually swap those around but that's not my API. So you can get the status of a pet by calling this one endpoint. So this is the the diagram of how your API works and when you're dealing with a bunch of microservices suddenly you're going to have about six different swagger endpoints in your head. You're going to you're going to get to know these pretty well especially if you're involved in multi-service end-to-end testing. Get to know these swagger docs very very well. You can build you can start to build suites in a product like Postman and share those suites with other people in your team if you have license for the whole organization and you can start to build test suites within Postman not even touch the code that actually helps you quite a bit in learning how to integrate these microservices. So drilling in on one particular endpoint swagger will give you the exact type and specific specificity of what the endpoints will take and it will tell you exactly what you expect to return. So if your contracts are published that implies that your teams have established a rapport with each other where they have the ability to speak in a language that they can guarantee that they understand. In a monolith world a developer will make a change to the system over here which queries the database and then the developer over here will change the database schema and this thing over here will break and it causes strife and mayhem that you're probably all familiar with. In the microservices world you cannot change someone else's database except through an API which in theory should have a contract published a published contract which allows you to guarantee that you know exactly how these database changes are going to be made. This is sort of simple white box black box testing conversation but when you go to the microservices world it is it is changed because I I kind of hope that USQA people typically don't interact with the database directly. I feel like it's our job to stay on the API side but that's that's a little bit outside the scope here but if if you can if the contract does not allow you to alter the database in the way that you need in order to test this is where you can start to ask for changes to the contract to make sure that you can get to what you need to get to in your chosen microservice. Now next question hugely important have you versioned your endpoints? This is a humongous gigantic horrific problem when you deal with third-party vendors because I've found most of them don't. The email vendor who shall remain nameless you have something like your domain IO so they skip the v2 they say account slash create and then one day they decide they're going to change the contract and everything breaks not just your tests everything breaks so if you version the endpoints if you encourage your developers to version their endpoints that means that their contract is frozen in time and that you're guaranteed that as long as they're they're advertising backwards compatibility that your microservice and their microservice will play when not play nicely together and that it won't break it seems like such a simple thing and yet people constantly omit this this one step so let me go it again so the next step would be feature flags is everyone familiar with what i'm talking about when i say feature flags i'm sorry to make this so interactive but i'm very genuinely curious who knows how to use feature flags or what they involve three years from now i'm going to be back here and i'm going to see a lot more hands than this feature flags are the ability to switch software on and off piece by piece inside your service monolith or microservice this works for both but this is definitely something that we use this in fact is a screenshot from a very monolithic content management system we have a reasonable enough but it has allowed us to move somewhat faster now the story i'm going to tell i'm going to get ahead of myself a tiny bit the story that i'm going to tell involves normally involves everyone make sure you are ready to deploy your software at seven a.m tomorrow the story that i'm going to tell says the production code that we need has been there for a week already just flip your switch we've tested everything already just flip your switch that's the paradigm that i think the industry is trying to move to there are downsides to this but that's certainly the way i'm trying to push things there were certainly the way that that my company is going in the way a lot of successful companies i've heard about will go that they say this is already there just turn it on and the change will be immediate these are some tools you can use we use toggles it's an open source library for feature flags optimistically is some people you can pay i don't know how it works but i just thought i'd throw it out there i am certainly not a sales pitchman for them now here's another question that you should have for your developers are your processes between the microservices or within microservices are they asynchronous or are they synchronous are you dealing with this whole promise infrastructure in javascript the way um the way things have been going for a while why would you care about sync versus async we've actually heard about this quite a bit in the last couple of days with parallel testing where things will be out of state and you don't know about it i'm not going to get too deeply into this topic but i know that at retail me in the past in 2012 we had a system which through analytics every single time you clicked on a coupon there would be a blocking call to the analytic platform that says make sure you deliver the message of how they clicked on this software that message will contain how far down was the coupon when they clicked it how long were they on the page before they clicked it did they scroll or did they keep it in place when they clicked it it contains all sorts of information about the behavior of that user you want to hear more about analytics i have a whole talk about that that i did with aptly tools but um we used to make that a blocking call which means that if the server that was intercepting the analytics went down users wouldn't be able to use the coupons anymore and it's even worse than that because we would still process the event later and say they clicked on a coupon when they didn't so we would get attribution that we didn't deserve from our vendors that was doubly bad so we changed that to an async service instead of going directly to a blocking call we sent the message to a rabid mq at the time we use kinesis stream now we sent the message to a rabid mq immediately returned with a 200 and you as the user simply don't care what happened to that call we care it's our responsibility if we can't process it it's our fault but you don't care at all we did that we removed that call and immediately our pages became more performant more reliable and everything got a lot faster that was our first microservice by the way when we started peeling this monolith off we peeled off that one first and we said we now have an asynchronous service that is tracking your whereabouts and your movement through the system so that we can deliver better content for you but we did it asynchronously which means that the event itself that we're looking for during the automated test we had automated tests that would look and say did the analytics go and this test would sit there and wait for it when we had the blocking call when we peeled it off and made it asynchronous there was a up to a 15 minute delay before the event would appear in the ETL system downstream so we had a we we we simply made our test asynchronous this event goes on to a queue 15 minutes later if it hasn't shown up on the other end throw an error and that happens asynchronously but you as converting the microservices you will probably want to have at least a vague notion of whether or not things happen between services in a blocking way or an asynchronous way which means how can i test this in such a way i can guarantee that it worked or i can guarantee that i've handed off the data so that it will appear correctly downstream i hope that makes sense what does your pyramid look like boy is this a topic i'm not going to dive deep into because the test pyramid is sorts of all sorts of controversy but what does your pyramid look like now and under the microservices architecture what will it look like how can you influence this kind of decision as a hopefully i could be considered a leader in the selenium community i feel like the last five years of my career has been spent telling people to use it less not more it's a strange thing i remember seeing a talk earlier today where someone described having 4500 ui tests at which point i'm like please tell me you could make that lower if you built a proper pyramid had an integration in unit tests but um maybe it's fantasy my company we have 1300 ui tests i think it's far too many but for full disclosure that's just the world we live in unfortunately we're trying to trim that down think about your test pyramid is my point so once development starts this is where i get into the story um once development starts this is where you really will thank yourself for having taken time to ask all these questions go through this planning sit with your management team and figure out why they're making the decisions that they're making and then at some point you'll ask yourself did they make the right decision i mean you should always be asking that question and being a little bit critical of the companies that you're working for sometimes it's just academic but sometimes it can help you go into the next job interview or the next job and really be able to talk intelligently with a critical eye towards judgment and whether or not the right decisions are being made and how they were being made so tell the story i'll talk about pii personally identifiable information your email address is considered pii though sort of lower category than something like your name your uh your unique number in these united states we have a social security number i'm not sure if info says ever finish that project to hand out the similar thing here but um so you know what i mean though address phone number ways to identify you uniquely um so this story will involve a little bit of pii jwt is anyone familiar with jwt's hey 12 15 people all right i'm a little bit surprised because i've never heard of this thing pretty cool uh it is a way of just just sticking a whole bunch of uh information into a cookie that is encrypted in a universal way anyway uh i'll be talking about that then graph ql which is a facebook uh service that has been produced which is a way of producing a service with a query language that is relatively simple to use so here is what retail may not look like when i got there look like that with the database and they talk to each other like i said php css javascript talking to a mycicle database which we have slowly over the last six years converted so here's how it looks today here's where we were and here's where we started peeling things off we built an api which was marvelous we no longer had php talking directly to the database which i can't even believe i just said out loud is a thing that happened in the 20th 21st century then we built several other microservices and these were things that were a combination of features that were within retail may not that we built and new features we needed to add from scratch so we never sold gift cards at retail may not so when we decided to build gift cards we undertook an exercise in in way overdoing microservices which we have now retreated from thankfully we essentially story within a story we wrote the gift cards uh microservice three separate times once using uh i don't remember what once using python and lambdas with aws and now using spring spring boot member profile was something that we use to capture more user pii between gift cards and member profile retail may not used to only be interested in your email address as a way of tracking who you were and sending you the occasional marketing email we are now moving into the world of you want to buy a gift card from us we're going to sell you a macy's gift card for a seven percent discount and then give you a 10 coupon so that we can stack those together give you a 17 discount all we need in return is your credit card information so we need to start being a little bit more careful with your data ledger service is interesting because it only tracks a unique key that is tied to your email address and numbers so there's really no pii in there there's simply a key that meant that that goes with your email address and whether you added or subtracted money from your ledger balance so then we added rmnql in front of all three of these services so that retail may not anything needed to act to it would go through one sort of conduit to all of these different services and then if anything changed with the contract we would simply change the the query language that we needed to send to rmnql this is graphql actually so then we built the mobile api the ios native app the android native app i didn't even bother to draw the arrows on here because i don't even want to admit where they all go where should they go they should go to the api they go everywhere and i'm very reluctant to admit that it's going to change so it works but is it better this is sort of my interlude is this a better state than we were in when we were in monolith well yes and no the good news about the monolith was that it was working it was already working it was in a unified tech stack and if you were a developer on retail me not you had to go through onboarding one time and that was it problem was that it was hard to fix bugs because there was a very spotty unit testing apparatus and in order to fix a bug you had to go through mountains of regression testing to make sure you didn't break someone else's stuff especially like i said when php code is directly talking to the sql database it's really hard to implement big changes in the monolith like that because as aforementioned if you're not also building unit tests obsessively along with the new code it is difficult to know that you're not breaking someone else's stuff another nod to the database issue microservices the developers are going to use the best tool for the job so in some cases our microservices use mongo because we just want to write the data really quick we don't care about ever querying it again that's when you use mongo we also allow teams to release independently this is probably the single best thing about our conversion to microservices we had a hundred developers working on a single monolith and now we have about 40 who are continuing to work on the old monolith and the rest the rest of people have split off into different teams i think we're up to eight different teams and each one is responsible for their own universe and they make their own decisions they can release independently and as long as they do not break the contract or they release a new version of the endpoints that is either backwards compatible or they announce their changes well in advance of disabling the old version then everything works fine teams releasing independently has led to retention and morale gains that might be worth the cost of doing this that's been my experience the problem is it can be overdone like i said before under the gift card stack see at the bottom we wrote that three times and at one point i believe i could be wrong i'm sorry if i'm lying anyone working for retail may not be watching this i believe at some point they had broken the gift cards microservice into single endpoint services about five of them that communicated in you know maybe you get the point it was a monolithic microservice that was well overdone and i think we learned our lesson from that we have back way the hell off so now we try to be a lot more deliberate and careful in how we choose where we're splitting things off so here's the problem this was all done before we introduced jwts this was all done back when we stored our user information in eight character strings which was fine if all we're gathering on you is your email address we didn't store any credit card information we didn't really need super strong security and this was also pre gdpr everyone knows what i mean when i say gdpr right yeah i bet if you don't it's europeans new your regulation and how companies websites deal with private data right this is all before that this is what the model it looked like before sorry this is what the microservice architect looked like before and this is when how we need to apply jwt the only thing here this does not touch is gift cards and ledger because we were able to sort of like those don't store any information about the user they just store keys you could still argue that they need to be secure because if something else is unsecure and you get the key then you have all the information you need point being is that if we were we we were saying that we need to take our entire code base and slap a new layer of security on top of that code base wouldn't that be easier in a monolith world that seems like it would have been because we knew how that thing worked we knew what everything would have been affected we knew how to deploy it adding jwt's to 13 separate microservices at the same time proved challenging so my boss came to me one day and said how are we going to test to make sure this thing doesn't break production and after about three days of research i realized that we didn't even know how to release it never mind how do we test it so here's the answer to how do we release it that's how we release it yeah i drew that what i ended up doing was writing what i ended up just calling a screenplay i have a liberal arts theater background i decided i'm going to take this thing and i'm going to make it engaging i'm going to turn it into acts and scenes and casts of characters and i'm going to make sure that everyone involved knows what role they're going to play and that they are heavily engaged exactly as engaged as you would need to be if you were standing on stage and needed to memorize your lines no one needed to memorize anything but anyway they had a script they could follow so let's take this through every act was a whole thing that you could do a thing that had a beginning and a middle and an end that was an independent operation in theory you could do act one and then wait six months before act two starts so how do you implement a single security change that affects every single cookie that goes to the website and tracks all of that through 13 systems so each scene tells you who is involved what they do and then what the result is furthermore i didn't include it on this slide every single scene has a rollback plan so you know where you are in the process you know how to get it done you know how to test to make sure it worked and you know how to get back in case something went horribly horribly wrong so kasey reed thomas love it my favorite people your job is to deploy this thing to prod but the karen enabled flag is set to false karen is from if you fly a few slides back i had these out of order i apologize karen is our code name for the project that contained the jwt karen is incidentally the name of the ferry boatman who carries you across the river sticks when you're going to hell we felt like that was an appropriate name karen is the service that carries these jwt's to the new reality so you're going to use the feature flag karen enabled set to false and you're going to deploy this thing to production meaning all of the code you've been spending the last six months working on is going to be in production but since the karen enabled flag is set to false nothing changes for the end user so then kyle dress and me our job is to smoke test in production to make sure that that deployment with the feature flag off broke nothing and how did we do this we did this in the room together all sitting there at a table so scene two is the the sort of collector service is going to use this flag it's going to turn it on and for a select few microservices they will start to take these jwt's even though nothing is actually sending them we felt like the distinction was important because we didn't want to start sending them and then realize that the system couldn't even handle it we wanted to turn them on so that at some point you could throw tests at them even in production and make sure that the system could handle them properly we're sort of going back front here and then every single act and most scenes have a definition of done and this is i cannot stress how important this is to know that the reason you have divided it into this kind of part is that at you know at this point no user facing system is affected but we have deployed software it is in production we know what's going on so the process for this is i know you can't read this i apologize is you research and you talk to everyone you can talk to and you find out exactly what is your role here what is your role here what does your code do what does your feature flag do is that behind a feature flag have you considered the implications of whatever thing happens you research then you communicate then you meet about it then you integrate and then you test and then you start over so what did we learn from this whole process we had a rollback plan for every step we figured out the stopping points where can we stop where can we so right now we are actually not done with this project i had four acts on that board act four is delete the old systems and we can kind of do that whenever we want they're still running they're still in production the old system is still there we can delete that whenever we want now that we have six months in production we know it's working so each act is a standalone step the next key is consider external departments this is customer support operations info sec it dev ops everyone get a head nod make sure that everyone at the vp level understands your plan and that you're going to execute it right and then you test and tell it is boring we ran nine separate trials of this thing it was horrible it was a little bit fun we had ice cream and stuff right but on the ninth run somebody said can we can we not do this anymore because the first run was terrible we deployed this thing in production it immediately stopped working by the ninth run someone said can we not do this anymore and that's kind of what i was waiting for finally we knew it was working because it was boring and then yeah what's your sunset plan for legacy service how do you determine when something goes into life an old microservice and then don't assume anything but if you do assume something documented figure out how you're going to measure results and i would advise not including development timelines this is not a development plan this is not a plan for how you write software this is a plan for how you deploy software this assumes the software is written now when you start act four and you're you're starting you're only beginning act one and you're you're starting to plan act four yeah you can assume the software is not written yet that's because you don't have to worry about actually executing act four yet have a meaningful retrospective another thing you should probably always be doing some things we skipped up so once again an act is composed of scenes which are fully independent of a major feature and you have a definition of done the scene is a milestone composed of steps with a list of players and a rollback plan and then the steps involved you have to put a name on the step when i was running the meeting the final meeting when we actually deployed it eight o'clock in the morning there were 35 people in the room including people from customer support several vp's a couple of directors and every developer who had a role to play they had their laptop open and i stood like a conductor one of the proud moments of my life Thomas it is time to flip your feature flag Andrew it is time to test this thing it is time to you know so we were able to and we did it with the ceo a few feet away 13 services nine schedule run through eight teams 200,000 logins per day which actually is it's a low number zero reported defects this thing worked because we had been so engaged in the plan for releasing it boring is your friend but only at the end only at the end so i didn't even talk about selenium i'm sorry i forgot this is selenium conference and that i sort of help put it together but the point is if you're you're doing your test pyramid right and you're headed towards a team of doing microservices i would estimate that your use of selenium your genuine need for selenium is going to be significantly lower than it has been to date that is my prescription i'm not even sure it's true but i hope it is true because if you follow some of these steps for questioning you know that your developers are writing unit tests you have a plan for how you're going to write integration tests at the api level and then your selenium scripts are there to make sure that it's all tied together in the ui at the end in as few iterations as possible so once again i i spend a lot of my time as a leader in this community telling people to use selenium less here i go again anyway that is my conclusion i hope that this has has been illuminating but i definitely think that staying engaged my prescription for life in general but certainly in a case like this this is one of the things i've received the most accolades for is putting this plan together it was exceedingly difficult the only way we pulled it off is that it was my full-time job for two full months now had we not broken this up into microservices had we stayed in our comfortable monolith land my bet is i wouldn't have even had to do it so this is as usual for my career a fantastic solution to a problem that never should have existed but it's a world i can live in pretty comfortably any questions yeah so thanks a lot Marcus this is like kind of eye-opening for most of us to think about like don't just choose the technology because you want to choose because that's actually going to be beneficial for your product and business so thanks a lot Marcus so can we give him a round of applause