 Okay, so let's start. My name is Juan de Carrillo. I'm a software architect in Lidl Digital. I have around ten years experience. I was working in the Czech Republic for around five years, three years in Beno, then in Prague. The last five years I have spent in Germany and mainly for first in the last six, seven years in e-commerce. Okay. The company I work for, it's well known for its supermarkets, okay? They have, it's a discounted. But what many people don't know is that we also run many online shops and that it's also a very important part of our business also as a part of our strategy. This company Lidl belongs to this group, okay? We like, it belongs to the Schwartz Group, it's a company that runs the Lidl Group, the supermarket, the the line part, Kaufflan. Kaufflan has also a few projects in this e-commerce area and it's quite big, like it's like around the size of Amazon worldwide. It's like depending what which numbers you look at, it's like the fourth or the fifth biggest retailer group in the world, okay? What we do on the line part, it's the line shops. We have also a website where we sell travel packages. You can book a hotel, you can book plane tickets, you can book it all together. We have also like some small areas that work kind of independent inside our non-food online shop, for example to sell wine and some alcohol. So we have also some things also in the area like of food because it fits also to the business. Something like you can order like some packages and then they sell you like some food to cook, like for the weekend or with all the ingredients, the receipt, the paper. We have also something to send flowers online. Not very profitable, but they love it because there's things that we have been in the customer with that. Something to print your photos on a cap, on a poster and then like a number of services that we try as a proof of concept and if it works, then we develop it further. Around like in this e-commerce part, like when we count, everybody working in IT, logistic in the warehouses, so we have around 2,500 people. We have in the part of the IT, we have five locations. In Germany, we have offices in Neckarsum, south of Germany, close to Sturga and in Berlin. We have also an office in Barcelona, very international and two more offices in Sofia and Bucharest, okay? This means that when we are developing something, when we have a project or a team, the team is usually split across all the locations, okay? In reality, it's even bigger because it may be also that we work with some external partners. For some particular topic and then they have different locations, but it works very well so far. This is a little bit like to show you how the warehouses are structured. This is also very important to understand the business of Lidl. So Lidl is at its counter. We sell products online, very cheap. The profit march is very little, so we try to do as much as we can by ourselves. So that means like running of our warehouses, but also running our own infrastructure in our data center because everything that we can say for every order, for every product that we sell, then it's it's something that we can invest in the future. If we were trying to outsource everything, it wouldn't work because we are not like a different kind of brand that can have a big march like Apple. In Apple, if they lose like one Europe product, they don't care, but in our case it's very important. And this affects also the way we operate our software, the way we develop our software and that's what we are running in production. Right now we have around 30,000 products. They are being sold simultaneously in several countries in Germany, in the Czech Republic, in Belgium, in Netherlands, and also in Spain. And there are more online shops to come. We have around 8 million customers already. This is the revenue that we have at the moment per year, but this is also incrementing very rapidly. And this is how we started. Okay, so this was our online shop with the current design. So this is something that we started developing like around 10 years ago, and it has been growing. At the beginning, 10 years ago, nobody knew if a leader was going to be accepted as an online platform. So that's why we started very small. We started with a very small team. We didn't want to invest too much money. And we started with a small application like self-developed that let us grow until now. But the problem that we were facing with the time is that we want to move faster, to try new things, to run AB test experiments, or to implement new features to catch up with this online sector that is very dynamic. It's difficult to scale this monolith to one team. One team works amazingly. You don't have almost overhead, but the problem is when you try to grow. Right now, this is what we have for the setup with the development. So we have five countries that work, five locations that are contributing with developers, business analysts, and other roles. We have around 20 teams to develop all our software platforms. At some point, we noticed that it wasn't working. So the main pain points were like we also tried with out-of-the-box products, some commercial solutions to make the role a lot faster in every country. It was some ACP product. And there we had like four teams and then we noticed like, hey, it doesn't work. Like when you have a big code base and you have four teams, you are all the time like having a lot of overhead, a lot of discussions to implement something. It has to go through many people. You need a lot of release management. And then we tried to make a break with what we had before and then we called this project Mineshift. Mineshift is not only the software that comes out of that. When we talk about Mineshift, it's also the new structure that comes out of that. So this structure, this organization with people, this new way of working where we have a smaller unit focused with a concrete purpose and improving some part of this platform. This is what we usually have. We have a monolith. A monolith, as I said before, it's great when you have only one team. The struggle comes when you have more than one team. If you do it properly, you will have an e-commerce monolith divided in these layers and you have a front end, you have back end, you have a database. Inside these layers, you can also identify the different domains. If it was properly done, you have to like proper developers, so they have structured the software in components and these components are grouped in domains. One domain might be the product handling, okay? One sub-domain of this product handling can be the price or can be the availability. So if it's a monolith and it's a big application, you can see these domains and these components. This will be the splitting layers. This is somehow how we split so far the development of the online shops until we started with these projects. Basically, we had a team with front-end developers working very close to the business department and they made some HTML mock-ups and then they just throw it to the back-end developers and then they integrated with the business requirements to implement it. This was working when we were small. At some point, it didn't work so well anymore. When we think about microservices many people think about the back-end. The back-end is very easy. You can have for your whole application, you can have a simple service for the product or even smaller for the availability, for the stock. You can have a back-end service only for the search. We have seen it many times. Actually, we started like this a long time ago. There were pieces that were not so easy to run with the monolith or with the database. So then we started creating some stock service where you were handling that in real time and then it was integrated to the monolith but a separated thing. What we have seen discussed many times is to have some kind of aggregation layer, some back-end for front-ends and then to have a single patch application or some repower where you have the whole front-end. It can work. Everything has advantages and disadvantages. The problem is that if you want to implement some future in the product or in the search, you probably have to touch two different repos. You need to coordinate that with two teams. So this is the problem. It looks good in the back-end, but it doesn't look so good on the front-end. So what we can do is to split this big monolith in these domains, the whole thing, and then one of these vertical domains will contain all the layers that we had until now. It will have the front-end, the back-end, the database, and it will contain all these components that are required for this business domain. If we make this split, then even if you have the front-end and the back-end together, you have very focused teams. You have a team that is very focused on the product. What is important for this team? For this team, it's important that when the customer finds a product, it doesn't matter how he found the product, if it was through some external affair or if it was through the search or some campaign, some landing page. What is important for us is that the customer finds all the information that he needs to assess if he wants to buy the product or not. Of course, we want to sell the product, but we want also to show the customer as much information as he needs to reduce the return. When you buy something online, you can always return it. According to the law, I think it's similar in the whole European Union, and this is quite important, for example, for wine. For wine across the whole industry, the supermarkets they throw away around 10% of the wine. This is crazy. This is the thing that they didn't handle when in the warehouse or they didn't coordinate well to sell it, and then it got spoiled. In the case of e-commerce, from our experience, it's like 6%, and when they return the bottle, you cannot sell it because you cannot make sure that it wasn't manipulated, so you need to destroy it. With mode, it's even worse. With modes, there are some big players like Zalando, they claim that 40%, 50% of whatever they sell, it's returned back. So if you can improve this product and page so that the customers are sure that this is what they want, if they can have high-quality pictures, if they can know that it's the right size, then you are also making a favor to the customer and to the business. For the search, it's a different mission. The mission that you have is like, you have something in mind and then you want to find it. And then you need to work also with different technologies. So the technologies that can help you on this vertical are very different than what you have from the search. In the search, you need to know how a search engine works. You need to know what can you do with the text, which attributes can you extract from the text and then show it later on the search. In the checkout, it's also different. On the checkout, what you want to do is to make it pleasant, to make it fast for the customer to go through the checkout and enter all the data that we need to ship the product. And then you have very different business requirements. What we gain doing this is not only that we can scale with the software, that we can scale with the organization. It's also that you get teams and then you have the business and you have the developers that are very focused in one particular domain and they can deliver the best user experience and features to our users. Another advantage is that if you divide that in this smaller domain, so you know how many resources you need on the infrastructure. So you could have an OpenShift cluster and then you could say, okay, so for the product detail I need that many instances of the frontend because I have some campaign going on. I need that many resources, these three resources for the backend, but maybe I just need one instance of our database. For other applications, it might be different. It might be like, okay, for the search, I have most of the rendering done on the client side, not that much done on the server, and then we will have only one instance of the frontend and then three of the backend because this is where I need more CPU. The same for rating, the same for every vertical. This applies for resources in our data center as for developers. Okay, so you can have more developers for one particular topic where you want to invest, where you want to grow because you know that it's not working that way and then you can have less resources for another. One thing that we are aware that it's slowing us down is like when you look at the numbers of Internet companies, so you see that because of the infrastructure and the culture and many other factors, they are able to deliver things to production very fast and this enables not only like to deliver something, some of the things that it might work, but also to do AB testing. AB testing means that I want to test a new feature, I want to test something new, and then I run an experiment with one subset of my users, for example 1%, and then 1% gets to be shown the new feature and to use this new feature and the other 1% that I have as a control group gets the regular feature. If I'm able to deliver that to production easily without any big pain, then I'm able to test more things, I'm able to drive the development of my application through data and this is something that Facebook, Amazon do very well, there are other big players, for example auto, a big e-commerce player, that it's also doing okay for the size and this is where we are a little bit behind, okay? We are a little bit behind because of this monolith because we have so many people working on this piece of code because we have parts of manual QA and this was also a reason to divide the application in this self-contained system, microservices and enable this experimentation. The customers at the end, like when they want to visit a page, they are not only interested on the part of the product, they want also to have the navigation, they want to also be able to add the product to the basket. It's not that you have a page and the page is only one vertical, it's a little bit more complex. For example, in this case we can see that there are different verticals, so one would be the product information, the product vertical, another one would be the part of the checkout, because here is where the checkout really starts when you add a product to the basket. We can see there on the top also the mini-basket icon, this is also a part that is provided from the checkout vertical because they have this information and then we have another fragment that comes from the search, okay? So this is one of the things that derive from this splitting in these domains. Here one thing that we are striving for is that we don't force the teams to commit to any technology, okay? So we want to be more agile, we want that every team decides what are the best technologies that solve the business domain problems, okay? In this case, in theory we could have an application for the team product running in view, we could say that the search is angular and that, for example, the team checkout works with React, that would be possible because at the end, like what we see on the browser, it's only a DOM object and you can combine DOM and have different applications with their own HTML, CSS, JavaScript. How we are going from one place to another, so we have to remember that this platform was running for around, it was developed for around 10 years, so you have 10 years of a huge team implementing new features and a business department like trying new things. So how can you shift from one thing to another without taking so much time? One of the ways we are doing that is incremental integration, so we are not waiting until we replace the whole thing to go live with everything, so the approach that we took is like we want to try our things as soon as we can, so we think, we thought at the beginning, okay, what is the smaller domain that we can take online? In this case, it was the rating, so we make a small proof of concept and after a few months, it was live, okay. I will show you later how this is integrated. Right now, we have gone live with several verticals, so the navigation for the header and the footer is live, the rating, the product and now comes the search, the checkout, so we are doing quite well. It's quite a lot of effort at the beginning to take the first piece of this application online, but then it's very easy because it's a loosely coupled, so you can deploy it independently. At the moment, we have 10 teams working on 10 different domains. This was also quite challenging because we were used to have developers that know the software very well, that know the business domain very well, and now we started with new teams. We had luck in some teams, we have people that were with us quite long, and they know the business domain, they know how it was implemented before. In other case, it's people that need a little bit of time to ramp up because they don't know the business requirements, so it takes a little bit longer time to do that, but in general, they are doing a great job. About the team responsibility is something that we are shifting. Before you had a different setup, you had people that were specialized on the frontend, and they were doing the frontend for all these domains. You had people that were focused, for example, from the business domain, they were focused on the A-B testing, and then we were doing the A-B testing for everything. Now, in every of these 10 teams, we have everything that we need to develop independently from other teams. That means that we have a product owner, he's the one that has the say about what to implement and when. We have also frontend developers, backend developers, and then people that can integrate that in our infrastructure. We are a developer with some experience in DevOps, so it's quite simple, actually. This is a little bit of our infrastructure, so we have a likely SAP phase on the backend. The focus of Mineshift would be this part that is facing the customer, the shop systems. Once we get the order confirmation for an order, then it goes to SAP and then we forget. This is more or less what we do. Here is a high-level diagram of the software artifacts that we have. We see here that we have a part of the backend systems to maintain, for example, the product information, the stock, the pricing. We have the CMS to add, for example, landing pages or to maintain the navigation. And then there are all these arrows that you see here are synchronously. So we don't have any synchronous call when we are running the software in order to render a particular fragment on the frontend. So you will talk only to these pieces of software that you have in your own vertical. The rest of the data that you need to render that comes synchronously. So these are the CMS, but the systems are applications that only our internal users see. And in this column, you have all the verticals, all the applications that our customers are facing. So at the end, they are integrated in some integration layer. We use for that varnish, we could have used EngineX or something like this. And then the page is returned to the customer. So this is somehow the loop. So we get the information of the product. We maintain the navigation, the landing page, the content. And then once the user has bought something, then they check out in an order and the order is passed to ACP. We have gone with this microservice approach, but we are not trying to make these parts as small as possible. So for example, for the checkout, we started only with one team for that. And then we created a single page application for the whole checkout. Right now, we divided this team and we have four teams that are going to be working in different domains. For example, order management, customer data management, like address, the basket. And then we need to split these applications. But in other parts, it's already losing support. So every application that you see here, the product detail, the rating, the search, the recommendation, the service, they are all totally independent of the other. So you can run them on the computer and then it will render this fragment of the page and the data will be distributed through a message blocker. In this case, we are using RabbitMQ. There are some use cases where we think that it makes sense to use Apache Kafka, but right now it's RabbitMQ in all these places. We still have some ugly places from our legacy systems. For example, here in these verticals, we still have a shared DB. But we are addressing the topic and thinking like how to split it further between the CMS, the navigation part, and the content pages. In order to migrate the data, so we are not doing a clone of what we had before. So what we are doing is like every domain has its own database structure. When I say database, it doesn't have to be a relational database. It depends on the domain. In some parts, we have just a very simple key value storage. With Redis, we don't need to purchase the data. We can always collect the data from the other verticals. In other verticals, we have a search engine running out of that, something like a legacy search. Others require a little bit more of data integrity, and then we have a little bit of a relational database. It depends on the domain. But what we do is like we are doing this migration per domain. We are going away from this big database and have this divided. OpenShift is one tool that we are hoping is going to take us farther in order to maintain the infrastructure, in order to give the developers more autonomy. I don't think I have to explain to you what OpenShift is. This is something that we don't have in place yet. We are finalizing that, but we hope that in March we can go live using OpenShift. So far, we are doing that using Docker Compose on virtual machines. It works, but in a few months, we will start using OpenShift in production. This is a snippet of how it looks, the integration. So this would be, for example, what you see here is a part of the product detail page. So on the product detail page, you want to render the health. So in order to render the health, the only thing that you need to do is this HID include, it's like a server-shiding include, where you say that you want to render this piece of HTML coming from the navigation vertical. And then varnish will take care of that under the hood. It's a very simple thing. You need to have a little bit of discipline, for example, to have in these fragments to have the proper CSS, like self-contained, the JavaScript, the CSS have to be invoked in reference, at least in the HTML. But once you wrap your mind around that, it's very easy to continue. The reason why we use HID includes and not a single page application is that for us, for our business, the sale is very important. It's very important for us that the products that are well positioned on the Internet, that they get a hit when you look for, for example, when you look for a bet or a mattress, you get in Germany some results from a little bit. This is very important. And this is why we cannot do just rendering on the browser. And this pushed us to use this HID include. But at the end, it's very nice because if you look at this code, the vertical that is including the others, know nothing about the technology. So you don't care if it's Angular, you don't care if it's React, you don't care if it's PHP or Java or both. You just have an endpoint, you include it and then you forget. Other options would be to use Ajax. So we can always use Ajax, retrieve some fragment from some endpoint and then add it to the DOM. Another option would be to share the state, for example, for the basket or for things that are related to the customer session on the browser and then have a little bit of DOM manipulation to add this so that you don't have to do this pimple going to the server to render this information. Another thing that works very well is to share web components using NPM. So what we do is like, for example, we on one vertical or the product vertical, we have a component to rent a product grid box. This component will be exported using NPM. It will be stored in our Artifactory or Nexus and then it will be imported from other verticals like the search to be rendered later or together. This might be a problem. If this NPM component requires some synchronous call because you can have a timeout, you can make your page slower when you are waiting for other pieces of software you can have a problem if there is some issue with the network but this works great. If this data that you need to render this component is contained in your vertical. In case that we use that, so what we do is like we always share this information, this view model through the message broker and not synchronous with rest. This is something funny that we have experienced so we basically open the door to use new technologies and at the beginning with the first teams they were introducing Apache Knight, Apache Kafka, Zookeeper a lot of technologies that were a little bit of overkill for these problems but then they soon realized that it could be more pragmatic and we could do the same with less effort. This was a great experience for us because we didn't know how it was that we didn't have the opportunity to play with this microservice landscape. Our experience is that when you have a new team they realize that it also helps if they use tools that are used already in our ecosystem even though we don't forbid them to introduce new things they tend to stick to things where they cannot get help from other colleagues. But at the end the team is responsible for the development to get ready what the product owner needs to be done and also if there is a machine production then it's the team so they are responsible to make it work. Some lessons that we have learned so far so we are working around one year in this setup the first lesson would be that it works it's much faster when you have parallel teams working in different domains the dependencies might be a problem at the beginning so you need somebody to deal with that so you either have two developers discussing with each other how to do that between two different domains, how to do the interfaces or you have some software architect that has a little bit of overview and then defines these interfaces this cut between them it was a little bit of a problem for the business people to understand which team should be doing that we are still working with that trying to teach them providing a better documentation so that they know how the whole thing works that's it, it's working it speeds up this works only if the business takes part of that so one thing that we were experienced at the beginning so the business were not playing with us so if we wanted to try something new it was difficult if you want this to work like long time not only when you make the first version you need to involve also the business and then you need to have the same structure on the business as in the product teams I would like to show you a little bit of how this is done so here you have a small overview of a few of the verticals that we have of the applications so every application it's usually a REST API first so we have to document the REST API so we have we are using Swagger UI where we document our endpoints this would be the private API we have done the same for the public API, the endpoints that are facing the customer and as you see so we have one vertical for the details one for the navigation one for the search, one for the content there are many more that are not listed on this page these are some technologies that we are using so we are using some kind of feature flags to switch things online and offline in production or in our integration environment we are using PrivateMQ, we are using Prometheus for inventory, Grafana to aggregate this information from Prometheus this is a tool to microservice to manage the stock Grey Lock, it's basically this ALK stack Elastic Search Locktash, Kivana and a little bit of tooling if we go to the product so we can see an overview of the product that we have here this would be the the HTML fragment that you render for a for a product detail page this is built in Vue.js to make the server-side rendering we have Nax.js something similar like Nax.js in React it means that the first page that you render will be always rendered by the server there are some advantages so it's the page speed it's a metric that Google used for sale it's basically the time that you need to render a page and then to make it usable it's much faster if you render the first page on the server because then you don't need to wait until the JavaScript is parsed and then integrate it and then you render that here you see the product information all this information comes through our REST API from the storage that is contained in this cell service and if you notice here you don't see any buttons to add the product to the basket this is like this because this information, these controls they will be part of the checkout team but if you look at the code what you would find here is that using the server-side rendering what we are painting is this include this include renders an endpoint in this case in our legacy system and it prints only this formula to add the product to the basket and to the watchlist for a particular product for a particular productivity this would be the endpoint that it would be rendered by the checkout as you see only the buttons that are required to add it this would be the same endpoint after it goes through varnish varnish is our integration layer in this case the one that is putting together this checkout includes and as you can see the buttons have been added but only to this fragment and if you go toward our customer then you would see that here it has been the whole thing integrated so you have the whole legacy system it's a Java application with JSP but here this is being rendered by UJS by our vertical event sheet and then this part it's rendered thanks to another checkout include that renders another endpoint so we are composing things this would be the first level of the checkout include by the full varnish handles up to four levels but you can also configure it so far we have a fountain use case to do more than two nasty levels this is some documentation that we use for the business guys or for the team so it's basically the interfaces that we have but written in some so when you go to when you try to teach somebody using these these things so this is something that a developer can understand very well but when you talk to the business department they don't get it they cannot think of this so it's better to describe it a little bit with some text so they know what are the domain interfaces from team to team or from vertical to vertical this would be the mission so this is some snapshot of what we have so as you see every team has a very mission so they don't try to fix all the problems that we have they focus only on one thing here you can find some of the guidelines that we were using to create our software so one is the self-contained system architecture so I think the authors of this website didn't invent that it was there already but they wrote it down and they made a good job specifying what this is about then some guidelines to be the the front end the back end some guidelines the 12 factor up manifesto so how to make software that runs without problem being cloud native and this is another link to go through these micro front ends documentation in this website you will find that they go a little more into detail on the technical part and they use a slightly different approach using web components so web components are part of the new generation of browsers you can still use it in other browsers that didn't catch up yet using a polyfill the reason that we why we don't use these custom elements from these standard web components is that the server side rendering is not that good yet so we went for next but I think something to consider for the future thank you for your attention and if you have questions I will be happy to answer you yeah so the our preferred setup is for developers for two front end, two back end but at the end they do they are more full stack so we encourage the developers to pick up tickets to pick up user stories to implement even if they are not the best doing that because we want to spread the knowledge we want them to understand we have a test automation engineer we have a manual tester this is something we want to go away from but we need to invest more time in the test automation then you have a product owner okay so so we so the guideline is like this self-contained system we also call it vertical so it should belong to only one team and one team can have one or more verticals and then inside the vertical they have as many micro services as they need but they usually the current setup is that for every vertical you have at least two back end applications one to handle the imports to have a little bit of monitoring to check if something goes wrong one to have the REST API and then the part of the front end so we have something like three artifacts per vertical three independent three independent deployments but if the if the team wants to break it down they can do that for example for the product we are implementing a micro service to handle only the the price and the availability okay so it's something that it breaks down we also are aware that one team so ideally one team should be responsible for one piece of software if you split the application in too many small micro services then you have also extra overhead to coordinate that so we try to keep it so it is consistent so for so when you have this setup you have a team but then these teams they have every member of this team they have different strengths okay for example you have the front end developer backend developer you have also assigned somehow some designer and these people they have some common guidelines okay so there is some centralized guidelines like when you write CSS code you have to use BAM you have to like not to break other components for the designers is the same so the designers have some corporate identity style and then they are responsible for following that okay so basically it's not organizing the team so basically the boss of this designer is maybe the one that is deciding the look and feel for the whole project or all the projects in little digital and he's in charge of saying okay so a battle should look like this okay this is how it works it might happen that we implement the battle twice but these guidelines they are across the company and we are responsible to follow up that so every application should be somehow highly available so we are using the common approach for example for example okay at the storage level we have always a cluster we have either a MySQL cluster or we have a Redis cluster so we do that at this level like this a small cluster for one domain and then for the for the backend application we will have something like a blue green deployment or some rolling deployment where you have at the same time applications running from two different versions and then you are shutting down one application and then creating any one we have also some load balancer on the top and this load balancer checks also for all the instances for a particular domain and if some node it's not healthy then it takes it out from the load balancing so this would be the way we do that we don't have anything like a big infrastructure teams that take care of that everything knows that the application have to be running the whole time and this is how we do it yeah that's a very good question so at the moment so we have in the legacy system we have jQuery as a framework and then we use VUJS for the new verticals okay in the ideal case when you load the page you should load only one bundle of these javascript css and this is handled by narch by webpack so it should be okay okay we have measured there is some overhead the overhead is not so much in the rendering part because when you render a page the page provides the html and the css the browser can render that but you also render the thing for this VUJS application and this requires the browser to pass the javascript even if no function is involved you have to pass the javascript and this is extra overhead that makes it a little bit slower to the customer so the problem is not so much how big it is the overhead that you have when you are rendering css or javascript at the moment it's only one application one framework it's okay there was an example okay thank you