 Hi, thank you for coming. I'm Jan and I would like to talk a bit about microservices and how to split your monolithic project. I work at Chomex as a part of our CMS team and I will use that CMS as an example about why, when and how to make that split. For that I would like to talk just a bit about microservices and then I turn out my projector. We streamed videos on demand mostly to Africa and some EU countries and our platform is quite fond of microservices and our CMS is a platform in general where our content team can enter some metadata about uploaded movies. It's title, description, images, stuff like that. And as we were writing that platform and the CMS we actually decided that we do not want to split that at the beginning because in Django it could be quite convenient to it's very easy basically to make some administration. You write your data model, you write like 10 lines of Django admin and it's already doing what you want and it's quite convenient to make your API 10 points with Django also so you can use it. In the data model you can use Django ORM maybe some common methods, but you have to be sort of very there not to match the match too much together, not to drone in 1000 mix-ins. Because if you do so it would be borderline impossible to make that bit later. You will end up with a mess over the code just on the multiple places. And as was said on another conference, if you can build a well-structured MOLIF, what makes you think microservices is the answer, right? Because it's still the same as. And yet we felt that it's in some time, it stopped being convenient for us. For the first we felt that our Postgre stopped to perform real well because we ended up with our movies having lots of related object on it. Next two reasons are sort of tied together when your main entry to your system are users filling lots and lots of form and there are a lot. They will pretty quickly find some edge case. You think you put all the restriction, you type check it all, you never do. You're not able to. And if it would be just the data management, it would be fine. They will write to you, you find the ORM movie, fix it in like 10-15 minutes, later put some restriction. But when your API is also in the same system, you probably have yourself a platform-wide audit. Also as that code bloats, you are constantly in each other way with your team. There are tons of managed conflict, maybe something is already on staging, but didn't pass the QA, so you have to either cherry pick or postpone the deployment. So for that, we decided it's time for us to split from Django and started to thinking about XSEC. We went with Falcon as our API framework. Quite light, powerful, easy to write, great thing. We use L6 search as our database and varnished for routing and caching. I would like to touch that a bit on the end of my talk. Just a side note, caching can be quite important for us because in peak hour, just one of our microservice serve about 8 million requests from cache. So about that Falcon, when we thought that we want to split from Django, we didn't felt the need to stay with Python at all. We also tried Golang because those two languages at the time had quite powerful L6 search libraries, which was in standard in 2016. And write some endpoints, run a benchmark and results with Go actually blew my mind and still does. It's 20 times quicker. Falcon also break 10 times quicker, but it's a bit overshadowed, right? And yet we went with that Falcon. The reason for that is simple. Personally, I suck at Golang. And with Falcon, we were there pretty quickly, it was quite a joy to write it with Go. It felt heavy-handed, it took a long time, and to migrate the whole service would be endless until we get good, which didn't seem nearer. So yeah, we went with Falcon. I mentioned that our post-bred didn't perform very well, but we only have somehow about 200,000 movies. And movies with all of its images, its description will call that an asset. And that's not a large number. Postgre could handle it pretty fine, unless you see our data model. In the middle, you can see that asset, that dot structure. There are some images on the right end, some contract on the left. And from your APIs, you actually want to return that asset as one object. So front-end can ask only once. Postgre would make lots and lots of joints on that. Somewhere around 100 for one front-end request. So we end up with like 32 million related objects to one asset, approximately 150 to one video. And if someone is to write me, OK, this asset doesn't have a description, it didn't show at all, I don't actually want to go to this. What I would much like to do is take a look at just a JSON, this whole Elastic Store data. I can find it more quickly. Also performance got better life and times, eight times on some endpoints. And you get quite powerful full text, which came in handy. So that would be Elastic Search. I think we have our TechSecs completed, so let's get to migrating it. It can be quite a headache, of course. And I would like to go through some common obstacles that you might encounter during writing it. First thing would be cache invalidation, because there is always a problem. We want to keep Postgre as our write-altered to use in Django still, but we want to serve our clients from Elastic Search. So how to do it when so long from contacting and this new asset or just update this typo in existing one, we would want to stream that change to Elastic Search and drop cache on that one object. We made a plan and then we decided that, well, we are not the news. We are not, we do not need that change in seconds. Instead, what we did is we wrote a Python script that takes Postgre, assembled that asset and feed Elastic Search index with it, slept some AngularS on it, we serve clients with it, AngularS. This happened every 40 minutes, so half an hour later, new indexes made, AngularS are swapped, then we can drop caches with some fuzziness, so drop will not hit us that hard. Next, next thing is, this migration can take some serious time. I split it my second microservice from CMS as my first task is showmax, like on my fourth day or so, and it took between two and two and a half months. Quite a lot of time, would be quicker now because I know this is so much better, but still, at this amount, it would take still. And during the whole time, your production, your product guy doesn't let you sleep because he really shouldn't. Instead, he comes with new feature requests and during that month or two months, you have to implement it twice, to your soon to be dead system and to your to be microservice. And it wouldn't be dead code, because nobody's dead like you, right? And if we were still a monolithic service, deployers are quite easy. Let's say that previously mentioned feature requests would need some data model change. Okay, then I change my model, I deploy it, someone review it, I'm fine. Now I have to change my model, change that script that fit elastic, change elastic search models, then microservices maybe even some middleware. So I have to make a plan for every deploy bigger, which is usually pretty straightforward. But what I think is slightly worse is it's no multiple code reviews and some context may be lost because I need to tell my colleagues, okay, you should review this, then this, then this. And I think there is quite a chance that they can overlook something, but only thing how I think we can tackle this is microservices in Monoripo because you will still need to deploy all of those services, but your peer that is reviewing it, have it all in one context. And the last thing that you might think about is routing. It's not really an obstacle because it's pretty straightforward, but what I think is worth mentioning, your front end guys should never be forced to change a code just because you are switching technologies. They, of course, they shouldn't change your parameters or recall it. They should even change your URL and it's pretty simple to do so, right? You just change your VCL, point it to different Docker, you're done. Your front end guys will never know, but you still want to break to them, right? And with this, let's say we have it written and we are in the QA phase. I am signed myself for a ticket, say, you do a QA, go to a cafe, and then someone came and asked me inevitable question, what can break? Well, all of it because I just wrote it. And I didn't think that we are able to QA this in a reasonable time until my colleague wrote what became this in, like, three hours. We call it log replay. It don't see your logs from Kibana and replace them against your locally running instance of this microservice. Then you can compare latency, but much more importantly, the last column is different. Compare status code and body over response. Is it not different? Great. So with this, we discovered our bugs pretty quickly. Fixed them, went to production. So, after we went to production, did we get anything out of it instead of the service being 18 numbers quicker? Well, still yes, because... Oh, I just want to mention the last thing, sorry. Good news because we are now thinking about how to generalize this tool enough to make it one of our open source projects. It's currently quite a lot of work to generalize it. But, okay, we are in production. We are quicker, but is there more? I can easily micromanage my resources because each of our microservices takes different amount of requests. One of those is responsible for returning details of that asset. So, user browsing the catalog, it can get called quite a lot. Second one, it came into play only when someone actually clicks the play. So, lot less requests. And I can accommodate to that with my containers. It's easier also to test in the bug. Is there anyone working regularly, if you can go? Okay. Did you try to write some integration or functional tests to it? Is it pain? It's incredibly painful experience to me. Okay, we have to talk later. It's also easier to debug. And by that, I mean know your code is doing just one thing. Maybe if you split it correctly in the beginning. Maybe in tons of different contexts, but it's still just one thing. So, if something broke, you know much better where to look. Also, it's easier to migrate your data because with Django, when you run some migration, you deploy it. Okay, migration is deployed. The first container is up, but all of your other containers doesn't know anything about the change. So, they can easily break. You have to stop the containers or do like free deploys, which is not really convenient. But with Elasticse, you just feed the script and it's really much easier for me. And let's start waiting for deployable master. By that, I mean that there are less merge constraints because all my colleagues are working on all those services. So, yeah, I think even in production, we got much better performance. And still, I think it's worth the wait to wait for your platform to mature. Don't do it from the beginning because you even have quite hectic tempo at the start. It's much more convenient to do it in that one context. But then it might not be so convenient anymore. And then do what your platform wants, not do it for the sake of doing it. I think that's all I wanted to say. If you have any questions, please make so. Or you can get me up in the halls for a few more hours. You can write me a mail of if you want to know more about our open source services or how we do things, there is a link to our tagline. Thank you. No, no, no, we have... Oh, sorry, repeat the question. Because we find a difference because we use the primary and the secondary. It is, it is. I thought I mentioned it, maybe I forgot because this is my first talk and I'm really nervous. We use Postgres still as a write-authority and each half an hour we take Postgres and done LSD search in this order of it. So, write-authority is still a Postgres because even before a year it would still be possible for LSD search to lose some data. LSD 2.0 still did it quite frequently. So, yeah, it's not our main data store, it's just what we serve clients with. Every half an hour. Because this is how we cache. Because those data can change for half an hour so why not cache for half an hour, right? You had a second one? I don't see any questions. Okay then. About LSD search we actually think about how to stream those changes frequently and drop the single cache object. We have all the tooling but it's more like what would be more cool than what do we need? We never actually need this. So, if there is something else? Thank you. Have a nice day.