 We've had a lot of really fantastic talks so far, and we've got a lot of really, really exciting ones coming up about very inspirational web tech and how we're going to pull together and build a better future web. All of these are really interesting. And inspiration in this kind of way is very important. But what I want to look at is what happens when you actually try and do this, when you actually try and take exciting technologies and put them into practice to solve hard problems. We're going to take a look through a real case study for a project we built recently for the BBC, look at our actual design process in doing that, the architect we end up with and how it works, and the benefits and drawbacks you see when you actually try and build microservice architectures like these in practice. Some context first, I worked for a company called Softwire. We make products for other people. Pretty much everything for pretty much everybody. And some of the big clients, people like the BBC and Cisco that you will have heard of and hundreds of other smaller ones. But for us, maintainability ends up being a really key end property of these. We need to build systems that we can quickly and effectively change that can very quickly and as cheaply as possible give us the kind of features, solve the kind of problems that our clients are facing. Often we're going to be giving the resulting code we develop directly to our clients to support themselves. So we want it to be as understandable and easy to manage as possible. And often we really want to be selling later projects. We want to build software, give it to our clients and have them so impressed that they come back and ask us to build more things on top of it. And once that happens, we obviously have to understand the stuff we wrote before so that we can develop that further. So we're trying to look at lots of different ways to solve these problems as cheaply and as quickly and as effectively as possible. And microservices have been something we've been thinking about and toying with in various different forms for quite a while. The project we're actually looking at here is BBC Newsbeat. This is BBC News for Radio One in the UK. So this is part of the same BBC News division and it's got a lot of the same goals. We're trying to provide news in a way that is going to be neutral and trustworthy and reliable and give that same in-depth quality journalism that people trust BBC News for. At the same time now, we're looking at a slightly different audience. So we're trying to build software that's going to deliver this news, this content through to 16 to 24-year-olds. So we're pitching down to a slightly younger audience here. That puts us in an interesting place. This is an audience that by and large has actually rejected a lot of traditional journalism. They're much more interested in social media. That's where they get a lot, a huge amount of their content from there, as well as that looking at other modern new versions of journalism, things like Buzzfeed who provide a very different way of delivering content to their users. And this changes how we build software and there's lots of interesting things in the kind of polish and front-end aspect of this that we're not really going to talk about today. But we do have to think in great detail much more about mobile devices, much more about delivering engaging experiences that really connect with these users. These are the users that are going to get turned off really, really quickly if we start providing content that is very, very slow, that doesn't work in the way they expect it to. They are very much digital natives here. As well as that speed of content delivery really becomes important. One of the big things that social media is competing on here is getting news from all its different users into each other's hands very quickly. If you go on Twitter, you can hear much more quickly about a lot of cutting-edge news. And as much as possible, we want to try and win back some of that territory for more traditional journalism. We want to get some of that speed. We want to get some of the engaging accessibility that you see in a lot of these other platforms and bring it to the BBC. This is a rewrite of an existing site that existed for many years. That site was a desktop-only, very traditional, older legacy application. And we want to turn this into a responsive application with desktop and web and tablets as well. And also Android and iOS apps underneath here. So we've got these various different platforms that we're trying to support. And this is at the front edge of an ongoing modernization process in the BBC generally. The BBC has been building on a lot of the same legacy infrastructure for a long time. As I don't know how well the news travels generally here, but as you may be aware, the BBC doesn't have a lot of money to play with. Public services in the UK generally are getting squeezed and we certainly can't afford to do huge rewrites of things. So they're trying to progressively steadily move services one by one from their previous bare metal Java PHP applications onto more microservice-based approaches, moving onto AWS, moving towards Ruby. Updating and modernizing to support the kind of problems we're talking about here. We're not really going to be talking about the actual front end of the apps or the websites themselves. What I want to look at is the content pipeline. We have existing journalistic services that journalists write stories into. CMS, that they write up their news in exactly the same way that they have been doing for quite a few years now and the same systems that support every single thing within the BBC. These are strange. These have been going for a while and weren't built for the kind of scale that the BBC is delivering at the moment. And we need to try and build on top of these and build a system that is going to allow us to quickly deliver news, to deliver these polished, engaging experiences and get that content from that CMS right the way through into users' hands onto these devices. That's the core problem I want to look at here. If you want to look at this yourself, bbc.co.uk-newsbeat does have the website and the newsbeat apps are on Android and iOS. I want to just walk through the actual UI here so you can see what we're aiming for. The main pages look like this. We've got a stream of cards, various different articles. We have various different ways of displaying these and they come from various different sources. Some of these are editorially selected. Some of this is the latest news as it's published. And we've also got most popular tabs of popular news that comes from another popular news service. And then we've got a linked data platform behind that and various bits of topic information and tagging that provides many, many more pages like this. So there's a lot of ways you can dig into this information. Each of the individual cards here links through to an article which looks something like this. We've got the full spread down the side there. We have many, many thousands of these. When we launched the new site here, we have to support every bit of content that Newsbeat has ever published historically. And all of these are actually quite expensive to render as pages. We're building on top of this legacy infrastructure, which is no longer co-located with our services because we're moving from bare metal across into exciting AWS territory. So each of them is quite slow to talk to and they're also struggling under a lot of load. So generally quite slow to manage. And we have to make a great many HTTP requests to actually do this. So to actually render this page, for example, we need to talk to one service to get the actual body of the article. But then we've also got this most popular sidebar down the side here. And that requires us to make a slow HTTP request to another service to get the list of IDs for this and then a series of HTTP requests for each article to get the content that we're gonna need to display there. And then another series of HTTP requests to get the topic information for each of these articles, the little tags you might be able to see in the corners. Then we've also got the related topics, related stories that come in down the side of the article there, which again need another HTTP request for each one. And then we've got a footer at the bottom of the article again, which requires us to go off to another service, make another request to get the initial IDs and sequential requests to get the actual articles and then the topic data later. Building one of these pages can easily end up taking upwards of 30 HTTP requests. And this is something we all have to do server side. And this is something that happens before the very initial first bit where we get to start actually showing this page. So we looked a little bit of web performance in the previous HTTP2 talk. And a lot of that is talking about how you can better deliver the subsequent content that you're depending on for your page rendering. But all of that really assumes the very first page load is gonna be very, very fast. When you're building in these kind of environments, when you're trying to build on legacy infrastructure, but do exciting new, modern things, you end up paying quite a cost just in that initial request. So we'd like to try and get rid of that. We've also then got responsive versions of the site for tablet and mobile and apps. And these are showing the same content and the apps notably are then doing this by composing it on device. So they make JSON requests for the body of the article and JSON requests for the footer and then they compose that together separately, as you might expect. My key point here is that we've got many, many, many thousands of pages of different pieces of content that the BBC have published. The BBC is a prolific publisher. And we've got a lot of different sources to talk to to get hold of this information. And talking to any of them is quite slow and expensive and error prone. This isn't a crud app. We can't just hit a database, join together a bunch of information and get the data we need and throw it on a page. We've got a much more complicated, painful process to go to to put this together. Hopefully this still doesn't sound too bad, right? This is a new site, it's a relatively static content. Building this kind of stuff is fiddly, it's non-trivial, especially when you're this high profile, but it's a reasonably, reasonably solvable problem with a relatively traditional approach. So let's make it a bit harder. Firstly, we need to handle fairly heavy and quite spiky traffic. We need to be able to launch this site to preferably take about 2 million daily active users, which for comparison is about the same size as Slack at the start of this year. And that's what we want to be able to support on launch day and on average in days in future. This is gonna be very spiky traffic as well. That's something you see a lot particularly in news applications like this. You're going to have millions of users suddenly turn up. On the week we launched Ricky Gervais started tweeting a whole bunch of links to various articles we've voted and we wrote and we started trending on Twitter. And you see as soon as you do that huge spikes where massive amounts of traffic come through and just the BBC's own audience. When BBC News tweets links to Newsby articles, BBC News has well above 10 million followers and a large chunk of those are going to click through and jump straight to the site. So we want to go handle this out of the box. Still manageable. We also have to be able to publish content quite quickly though and we touched upon this a little bit earlier. This is an audience that really wants to see cutting edge news right now. They want to put that in their hands as quickly as possible and that's something that they really care about and these fights against some of what we were looking at before it's much easier to handle this traffic if you throw in loads and loads of caches that are going to last quite a while but we can't do that. We need to be able to publish articles and immediately get them out. We also need to be able to publish updates and immediately get them out. The standard way as a journalist that you write breaking news is you write a very, very short intro paragraph and you publish it and then you update every couple of minutes and you keep doing that as more information comes in. So we need to be able to get these updates out very quickly. We also have to worry about redactions and this is more of a BBC specific problem. At any point if we discover we've published something that is false, that is wrong, that we need to update, we have to be able to push updated versions of that for any bit of content that we've published, into user's hands, preferably within maximum one minute and preferably quite a bit quicker. So we have to be able to invalidate all the caches right the way through end to end within a minute for everything and we'd like as much as possible to really focus on this, getting our content into our user's hands really quickly. Again though, manageable with heavy caching and just shorter lifetimes for this but we can make it more difficult. The backend services we're building on are legacy and old and struggling. They're supporting everything inside the BBC and the BBC is big and high profile and if we break them, it will be very embarrassing. It becomes international news if BBC news goes down for any substantial amount of time and I personally don't really want to be responsible for doing that. That's not something I'm particularly keen on putting on my CV and more practically even if we thought the risk was gonna be okay, the BBC are not going to let us release any code which we cannot convincingly put bounds on the amount of load it's gonna put on backend services. We have to be able to ensure that anything we build is not gonna be heavily polling these backend services for content and preferably we'd like to make requests to the backend services only when there is new information to pull through. Not because we've got lots of requests and we're polling and quickly trying to pull updates but the absolute minimum possible frequency. On top of that, even though we're gonna be as gentle as we can with these services, there are lots and lots of other services being built on top of these and they're quite fragile. They are often going to get slower and slower to make requests. Certainly waiting a couple of hundred milliseconds for each of those HTTP requests is not gonna be uncommon and that's often gonna get even slower. We're gonna have actual timeouts as we try and talk to our internal infrastructure here and despite this, this is the BBC. It has a reputation for being visibly reliable for publicly being there, for steadily being something you can trust and particularly with an audience like this, that falls apart if the site just goes down if we just start throwing errors. So whenever possible, we want to serve up our best efforts of the news. We want to serve stale content if we don't have the newest news. We want to serve stale content while we're revalidating other content. We want to guarantee that no matter what happens, we do our very best to give you the closest we can to the news. So this gives us some interesting conflicting problems. On top of this, of course, we're an agency. The BBC doesn't have huge amount of money or time to throw at these kind of problems. So we're trying to put this together in as quickly and as efficiently an approach as possible. We've got a few months and about five people to put this whole thing together and launch it. So this puts us in a difficult place, okay? What I want to look at now is the design approach we went through and design ideas I'd like you to think about if you're trying to put things like microservices into practice. If you're trying to take exciting new technologies and put them into reality, into your system, perhaps to update systems in this same kind of environment in these kind of large organizations, how we can think about those problems and how we can solve those better. We're going to walk through a series of approaches that are going to start terrible and that we could then steadily improve to actually solve these problems we're talking about here. And we're going to see how the eventual architecture evolves out of this. This is hypothetical. We didn't actually build a terrible architecture that was obviously not going to work, release it and then start steadily updating it. But this is the process we went through as an initial design phase to make this work. And I should point out, again, this is not actually initially soft-wise idea alone. A lot of this is stuff that the BBC have been working on. They're trying to modernize already in this direction and we're developing on ideas that the BBC themselves have been thinking about and they deserve a lot of credit for in various places starting to pull together these same kind of architectural concepts we're talking about. So let's start with a really simple basic architecture that's obviously not going to work. But this is traditionally how we might have built an application, right? You want to serve content on the internet. You make a service. It takes requests. It loads all the data it needs from some backend sources. It puts together some HTML or some JSON and serves that straight back out. This, I think, pretty obviously is not going to solve almost any of the requirements we've talked about. But it's going to come down most interestingly in once we get to the actual heavy traffic. Trying to scale this. This is going to scale at best, really, relatively linearly for this front end with a number of traffic, with the amount of traffic coming in. But most of those requests are doing exactly the same process as all the others. We'd like to make this much, much more efficient than that, if possible. So we need to find some way of transforming and improving this architecture. So that it can support more traffic. And again, we can still use relatively traditional easy solutions to this sort of stuff. The obvious thing to do at this point is to stick a massive great cache on the front of it. And we actually do this. We run Varnish in front, which is an excellent system for solving this sort of stuff. And we can just stick some simple time-based expiry on here. Every minute, let requests through and hit the actual backend service. And Varnish has a lot of little levers we can pull to do some more complicated behavior. To solve that, to solve problems here and tweak this. But it still doesn't actually solve most of what we're trying to do. One of the really big bits that breaks here is when we get to cache misses. We're inevitably going to get cache misses here. When you first publish an article, we're going to get a wave of traffic coming through quite quickly. That is all going to hit until we have our first rendering of this page. That is all going to get blocked on this backend service. Similarly, whenever we get cache evictions, which, if we're going to update every minute, needs to be at least once a minute, we're going to end up with requests unless we do some quite fiddly fiddling. We're going to end up with requests synchronously blocking on this service. And as soon as we do that, we're in a big problem. If we are waiting synchronously on this service, then we have to wait for all these HTTP requests, some of which are sometimes going to fail. And we're going to get requests that fail. We're going to start serving slow news to this audience that really cares about fast news. So, something's going to break as soon as we have synchronous cache misses in this kind of way. So, we'd like to change this architecture and transform it again to solve our problems better. We'd like to transform it so that we never have requests coming in and synchronously waiting for this data loading process. And from this, we start to move towards a microservice approach. So, we now simply split this into two services, okay? Our first step here, we're going to pull out data loading. And just on a timer, we're going to do that totally asynchronously and put it into some data store. And let's not worry too much about the details of this. And that, say, is going to go and get the content for each article, get the list of latest articles, each bit of data that we care about and put it in some data store. And this gives us a simple background async process to manage this. And then on the front end, when a request comes in and comes through the cache there, that's going to hit our service. And that is just now going to load the same bits of information, but from a data store that we can co-locate in AWS. It's going to be nice and quick and we have much more control about making highly available, this sort of thing. By doing this, all of a sudden, you never have a request coming in that is blocking on the actual data loading part. Just this one very small change means that we don't ever have requests going in that might fail because these backend services are down. As long as we've rendered, as long as we've got hold of some data at least once, when we go to the data store, we'll just get the latest stale data. And we don't need to dive all the way in to microservices and if this was all of our requirements, this would be sufficient. But this change gives us enough of a structure to solve that problem. But we still have more problems. We always have more problems. Here, critically, it's very difficult to work out what exactly that timer should do. How often do we go and we reload content? We can't handle the backend services, so we can't have that timer go through and update all of our many, many, many thousands of articles every single minute. If we do that, we're gonna send at minimum thousands of requests through to our backend service. Probably many multiples of that. That will take down the backend service. That will take down BBC News and everything else we know and love. And it will be very bad, it will be sued. It's gonna be a public embarrassing big problem. So we need to find a better way of managing this data loading frequency. We also can't make it only one hour because if we do, we don't get the kind of updating requirements once. We're going to be an hour behind the times with updated stories, with actually getting the news into people's hands. We do now because we're using this different architecture have more control, because we've pulled this out into a separate service. We now can change it in new ways. We can make this timer do whatever we like. We can update different articles at different frequencies depending on how new they are, this kind of thing. We've gained more control than you have with a simple cache. But still it doesn't actually solve our problems. So we need to change this so that we can get those same guarantees on frequency of updates without hammering the backend service. What we really need to start doing here is moving to a more event-driven architecture. So with BBC have actually been worrying about this, the legacy infrastructure is difficult to change. They can't easily go in and make wide-changing content services to make the changes to the content services, to make these scale better, to make this just work. But we can easily do things like add notifications. So BBC content services now links up to SNS, the Amazon Simple Notification Service. And every time a new bit of content is written, it throws out an event saying we've now published Article 12, for example. And we can hook that up to SQS on a queue. And now we can have a queue that this service just waits on. We have a queue with a list of all the bits of content that have recently been published. And our service now just reads off the queue. And we can scale these up to render these in parallel or do whatever we like. And this loads in our bits of data and sticks them in the database and does so as frequently as we need. We don't have to worry about updating articles that haven't changed and polling for that. We only make requests when actual updates happen. So this event-driven architecture now, with just a small change, lets us start pushing stuff through. And again, this is a simple change for us to make at this point because of the start of a microservice approach here. Because we've separated these into separate services, we can change this back-end part. And the front-end hasn't actually changed at all. Half of our system is totally untouched by this change, despite a reasonably-sized change to our underlying behavior. We've still got some problems here. Critically, the front-end is kind of basic and doing a lot of re-rendering. So we get a request that comes in. We say, I want to show this article. All right, it's going to make 30 requests that now go to our data store that holds the loaded data. But it's then going to have to make all of those requests, going to have to spend the time doing that, follow the same process, build all that HTML and serve it up, even though nothing's necessarily changed. So it would be much better, certainly from the point of view of cheaply running this with our millions of users coming through, to have that instead only done once when the content changes. We like to change this architecture so that we don't re-render all the time. To do that, we need to move the rendering. And just looking at the architecture and thinking about it from this perspective makes it relatively clear. We've got two blobs and one of them says render, and we want it to happen in the other blob. So we move it to the other blob. Now, content changes. We get a notification saying article 12 has changed. We go to the content services, get all the content we need to render that whole page. We render the HTML and the JSON for it, and we throw that into S3. We just build the whole page, pre-render the whole thing, and save that. And then when actual requests come in, come through the web or through the apps, through a cache, because they actually get a cache miss, all we do now is we look up the totally pre-rendered bit of content that we've got straight out of S3 and we throw it back. Our front-end now gets way quicker. It doesn't do anything. It doesn't have to think about anything. It's really just a lookup service. It's extremely dumb. You could almost scribble it out and just read straight through to S3. At the moment, in this, it's really just mapping URLs and not doing very much else. Again, though, this solves new problems, but it doesn't solve all our problems. Yeah, we're still doing more requests than we like to the back-end content services because we've got one big blob doing rendering, and that means it's re-rendering all the common elements. So if we have a big article page, and we're told the article, the journalist has updated this article, we've got some new information, we go and we get the new body of the article, but we've got to produce the HTML for the whole page. So we have to also go and get the list of all the latest stories and rebuild the footer that brings all of these together and all the information for that. All the most popular articles bring all of this together and put a sidebar for that. We have to glue together all these extra bits of content, and doing so requires us to make many, many, many more requests than we'd need. More problematic when you have articles that haven't changed, we end up not actually updating the surrounding content. If you've got an article that hasn't been updated in a week, you've pre-rendered the whole thing, the latest footer at the bottom is not the latest anymore. So we need to separate these out. We need to take this and split up the individual components of the page and then later compose them together at runtime, something more like this. And here you start seeing a much more microservice approach coming out. Now, on the back-end, we get notifications for individual types of content that need to be updated. So when an article is updated, we re-render the body of the article alone, the HTML or the JSON for that, and throw both of those into S3. We also have a queue that manages when the latest, where all the latest content should be updated, both the content that becomes part of the front page and of the footers on various articles. And that, again, has a queue. Here, actually, we just control that queue by throwing in events on a schedule here to update that every 30 seconds or so. But we have a lot of control. We're using SQS to do this, and we can just chuck whatever we like in it with whatever control we need. We gain a lot of easy power to change this. Similarly, for the topics feed, which we renders all of the many thousands of different topics pages we have available, and then we've got a whole bunch of other renderers for things like the most popular articles for various bits of other static content distributed around. So we can build many, many small renderers here, and each of these are small, dumb components that take events, go and look up some data, stick it together into some static content, and throw NS3. Then on the front end, we just compose these together. You get a request in for an article, and you say, okay, I know what an article page looks like. It's got a header, it's got the article body, it's got the most popular sidebar, and it's got the latest footer. We stick these together, and that is a far smaller number of requests to S3 than we were making to our data store before. We're down from 30 requests to three, and all we do is take those strings, really, and concatenate them together. So we've got a very simple process to glue together these pages, and this is something you'll see a lot of news organizations talking about. That's the new way to build and cheaply update for content-heavy pages when you have to manage this amount of content. There's still problems as this. There's still stuff we'd like to improve here and do better, and the obvious one that sticks out for me as we've started breaking up all these services is this large service we've still got in the front that is doing a bunch of stuff. We've got multiple problems here coupled together, and that means when a request comes in for the app, for example, which doesn't care about composition whatsoever, it's we still have to go through some of the logic for do we need to compose together these kind of pages, and we still have risks where if we start getting bugs in the composer and that starts crashing in the composition logic, the apps will start failing where they're actually totally unrelated. So we like to take this and make one last change to actually split those out. That looks something like this. Requests come in from apps and they go to a broker, which is really just look up as a service, and that reads through to S3. I want the HTML for this article, read from S3, whatever. We then, of course, go to the composer and say I'd like this entire page. That says I know what an article page looks like. I need this bit, this bit, this bit, makes all those queries through in parallel to the broker to pull together those bits of content. This also got some internal bits of caching to make that quicker for shared bits of content like these footers we're talking about. And that just knows how to do composition. We end up with these three very small DUM services. This is our resulting architecture. This is the core structures we're building here, this small set of DUM services. We've got these renderers. They are very, very simple and DUM. And also these really are the kind of things that you would now start to move on to Lambda. This was written about a year ago now or released about a year ago now. So it was a little less mature, but I'm looking forward to the next talk in Serverless and how we could start actually building these infrastructures on top of that. These are very simple, very DUM services that don't do anything. They take an input, they throw an output somewhere. And that means you can do cool stuff like compose them together. We could have the article render throw a message onto the latest feed queue saying a new article has been rendered, therefore update the latest feed and you can start to chain together flows of these. You gain a lot of flexibility because they don't know anything about their environment. They just listen to a queue and output to S3 and everything else is irrelevant. We've got a broker which is really just S3 lookup as a service, very little else. It does little bits of mapping and logic and nothing else. We've got a composer which knows what pages look like and is a templating engine as a service. Each of these are small standalone functions that don't do very much and don't break very easily. They're very easy to build. This is the kind of thing, each of these that you could almost do as an interview question. You could plausibly hand these out to people and say, could you build one of these in half an hour? A lot of this stuff is very, very simple and that puts you in a cool place. Plausibly, you can actually delegate all your software development to interview candidates. You're in a place where quite easily you can build very, very simple services and the overall architecture gives you the behavior that we're looking for. Each of these services has a very simple implementation. They're doing very little, but we end up solving relatively complicated problems purely from the architecture itself driving them. Technology under the hood will skim through quickly. It's not particularly relevant. The architecture is the important part here, but this is Ruby. We are autoscaling and a lot of the autoscaling actually comes out very nicely, simply because we're just building on standard AWS components. The renderers, for example, just scale on the size of their queues because we're using standard tools there and that's really, really easy to manage. S3 and Dynamo for lookup into S3, SQS and SNS for the actual event-driven parts of this. Elastic has to take the edge off some of our internal load between these components, particularly the composure and the broker and CloudWatch for monitoring, which, again, very standardized components makes much easier than it would be. If you want to monitor the renderers, you can just look at how much stuff is waiting in the queue for each renderer and really easily that gives you a basic idea of what's going on with that. You can pretty much just turn monitoring on right out of the box and know what's going on and we build extra monitoring on top, which, again, CloudWatch makes quite easy, but having small components glued together by very standard tools gives you a load of benefits here. It's all built on BBC's Elephant Framework, which we've been working to evolve with them to actually bring up to the scale so you can build entire applications on it. This is now open source, so if you're building with Ruby, you can try this out yourself and do the same kind of things that the BBC is doing to drive their modern application infrastructure. What is interesting here is how this actually plays out. The end benefits for us, having tried to solve these tricky problems on a budget, on tight timelines. It works in practice really, really well, right? Performance is one of the hard bits we were trying to face, getting content into users' hands very quickly, being able to handle this very spiky traffic, and we get that really easily, and we get that not by building complicated implementations. The architecture just makes it kind of work. If you move all your work away from your users so they're not allowed to touch it, then they can't add that much load to your system. Requests that do come through the cache just don't really have to do very much. They just hit very, very dumb, simple services. That makes it really easy to go fast. It's not that we've done very clever optimizations to make this stuff fast. It's not that we need to be brilliant. The overall architecture makes it easy to do this. Similarly, robustness. By building an architecture like this, it's very hard to make systems that go wrong. These services are all very stupid. They don't rake very much. We end up being about as reliable as S3, which is certainly enough for our needs. And everything that comes through doesn't end up ever actually blocking on the rickety bits. We've built a really quite resilient, quite robust system for not very much money on a legacy rickety mess that we can't trust. And you can do that by focusing on these kind of architectures where your front end is stupid. The actual modules themselves end up being very nicely decoupled and encapsulated. Quite a few changes as we go through that architecture there. Half the architecture, at least, doesn't actually know what's happening. When you split the composer and broker in half, the renderers don't change. They continue exactly as they were. We've got lots of independent things. And this gives us maintainability we're looking for in future. The BBC, as they take on and support these services themselves, can independently pick up any one of these modules, see how they work, change that, separately from the rest of the system. We don't actually have other than the composer and the broker. Really, very many direct interactions between these components at all. They all go through standard AWS steps. And that gives us some cool things. It gives us easy monitoring, like we already talked about. We also had to republish all the historical NewsBeat content. And doing that with this architecture is really easy. If you want to re-render every single article, you throw the IDs of every single article on the queue for the article renderer, done. That pummelled it all right the way through and you need to do this steadily to not hammer the back end. But you don't have to build a separate tool or any kind of separate processes to set that up. You just use the existing infrastructure. And because you've got standard tools like SQS, where you can just poke in and change them, you've got a lot of access to exactly how that works internally. We also use the broker, for example, to serve up some static content. So various bits of config that the apps need. We just put in S3 and the broker can serve that out for us because it just understands S3. It doesn't need to know that this wasn't rendered. We wrote it in manually. It only understands S3. And finally, again, the actual behavior we're looking for comes out of the architecture. Not out of microservices themselves. Microservices have not just made us scalable. We've come up with designs with modularity that allows us to scale this. And then microservice approaches allow us to nicely manage that, give us the power to do this better. And we get things like the resilience, the serving stable content just for free. You just serve out of S3. They just kind of works. This is not all roses. There are downsides to building these kind of event-driven architectures and you do see problems with these that other people talk about. So Twitter does similar bits of infrastructure and talks about similar problems about how you pre-render users' timelines and this kind of thing. This is something you'll see a lot in other services trying to be, trying to support read heavy content, but there are problems with it. Just because it's common doesn't mean it's easy. Critically, everything has to end up very cashable here. So in our designs, articles had a little bit at the top. It said published five minutes ago. That is not content. You can stick in a cache. And if you're not thinking about how this model actually works and you accidentally do include that in a cache, then 10 minutes later, it will be wrong. And if that article doesn't update, it will stay wrong forever. So you have to think about which individual small bits of content you're gonna end up re-rendering on the client side, so in the apps or in JavaScript for the actual web application. You've also got similar problems with data migration. Here for us, this worked out relatively well, but if you're rendering a bunch of JSON and you're caching this and then you decide do you want to change the structure of your JSON? This isn't like a normal web service where you can just change how you serve up JSON and then think about versioning on the app end, on the client end. You actually, if you want to change the structure of anything, have to re-render everything you have ever done. You have to go back right the way through the content and you're gonna get similar problems if you try to make large structural changes to your HTML. So you need to be aware of the kind of challenges you're gonna see there. And finally, code sharing particularly becomes a bit of a mess. Something that a lot of microservice proponents have talked about in a lot of interesting ways and there are a lot of nice theoretical solutions to this. You can think about actually just duplicating code and refusing to directly share anything with places. You can follow a rule of strictly moving everything into standalone services. And in practice, certainly for this kind of case, a lot of that doesn't work. Eventually that will break down at certain points. Mono repos are another interesting approach. What we've actually looked at here instead is moving to just simple Ruby gems to share stuff and trying to minimize the amount of stuff we share. And that worked out reasonably, but it is a little painful in places. I don't have perfect answers to this and I would be very interested in talking to people who think they do. But there's a very difficult trade-off you have to make when you start using microservices in practice as the extra complexity you're going to allow in various places and how you're going to balance that against the simpler but perhaps slightly messier approach. Hopefully this is kind of interesting use into the look into the reality of actually doing this. It's not quite as shiny and magical as seeing all these cool technologies and the nice things they can do. And when you try and start actually using these technologies, you'll find out that's the reality too. Just trying to move to a microservice architecture isn't a nice, wonderful place. Microservices won't just be scalable for free. When you try and build these architectures, they won't just be modular for free. If you want to build a better future web, you need to focus actually on the design process. Don't think about just adding microservices. Think about your fundamental problems. Think about how you're going to iterate and develop your design to solve them. And then think about where interesting new technologies are going to come in there and allow you to build these better designs and build a better future web. Thank you very much. So what was your approach to testing all of these microservices and making sure that all the infrastructure and everything is working as you expect? Okay, so that is a fascinating topic that is sadly too difficult to actually fit in the size of this talk. But the decoupling of these makes these really, really nice, especially because they're just taking an input and throwing out an output most of the time. So we've got standalone component tests for each and every component in here, which runs a fake AWS environment and throws something onto our fake queue and then asserts on the output in S3, for example. That becomes really nice. We've then also got similar component tests for each of the apps in the actual front end for this and the Selenium tests on top of there with a fake S3 content. And then some overall end-to-end tests that spin up the whole thing in a real AWS environment and spin up a fake phone emulator and start trying to actually use it and check whether when we publish new news it comes through within our kind of constraints. There's a lot of complexity to bringing all of that together, but even just having the component test is actually a much nicer experience than you get when trying to develop this as a single model of the application. All right, I have to ask you this one more question because I really like it. If you can't trust legacy systems, how can you trust them to send events? Ah, that is a good question. We probably can't. There is definitely going to be times when the back-end legacy services are going to certainly give us delayed events, certainly going to take a while before we hear about new news that's been written. And that's definitely a bit of a problem. In this kind of case, so when you're working in larger organizations like this to try and deliver the future, you just kind of have to accept that some of that stuff is going to be the case. We get really the best possible result we can on top of that here. As soon as new news becomes available and we get the information about it, we end up rendering it. But yes, we can't do it perfectly. And if we're simply not going to get events about new news, there's not really very much we can do about it. Hopefully, though, in the future, the BBC and I suspect other organizations in this position are going to be steadily evolving in this direction. And a lot of this comes down to culture and commitment to improving in this way. But the BBC, especially, is very keen on moving in this kind of direction. And that should, in the long term, actually solve these problems. All right, thank you so much, Tim, for telling us all about how to do Myoka services for realsies. Cool, thank you.