 But before I start a little bit about myself, I'm Karan Honawar. I work as engineering manager at IKEA IT, the same IKEA where you buy furniture. We have IT as well. Been working with data and analytics for over 18 years now. And right now I have the assignment to develop this platform for machine learning operations. I'm quite passionate about AI and automation. I mean, that's been my sort of bread and butter over the years. I'm an open source enthusiast now. I must admit over the last three years, not really have been active in the open source space before. And almost an MROPS evangelist, so to say. I mean, been speaking about this, been trying to develop this for the last one and a half years. And it's quite a new topic in many parts of Europe, at least. And a concert junkie. In fact, I'm thinking if I should go to the disturbed concert tomorrow that's happening outside Vancouver. But probably not. You need a car here. Also allowed to play badminton. And happy to connect with any of you in LinkedIn. So today, we're going to do a quick intro to IKEA. I mean, I'm hoping most of you know what IKEA is, have been to the store. But just to give you a context on what IKEA is, then we move into some sort of basic reasoning. Why a platform? You know, why not have those data scientists do what they like, something that they've always been doing. Then we discuss about the data personas. You might notice that my wording changes to Inka now. So we are a retailer for IKEA, one among many, but the largest. So it's a setup where InterIKE as a company owns the brand. And we are like the franchisee that have stores. And then we move into the platform construct. Try to go into a little bit of detail on some of the components, what open source choices. And then we move into Q&A. So our vision. I mean, IKEA has a very simple but a very powerful vision to create a better everyday life for the many people. And this is for giving furniture, home furnishings for the many. And it's the very context of being as low-priced as possible, but at the same time with function and form. This forms the key of how we operate within our furniture business. And it drives even how we develop our IT applications, platforms with the same mindset. As a company, I mean, we are about 42 billion euros. We are over 177k work coworkers across the world working at the stores, the warehouses, the pickup points, and so on, which are approximately 482. This number is definitely wrong, because we keep opening new pickup points and so on. But this is when I had this right a week ago. And then, of course, we also have what's called as meeting places or shopping malls in some parts of the world, particularly in China, which is a business model of having IKEA store within a shopping mall. But what's also important for us is how we are able to have gender balance within leadership. We have a target of a 50-50 gender balance between men and women. We also have circularity, which means we try to keep renewing our furnitures, which means we buy back furnitures as well. We're trying to drive a whole sustainability agenda with trying to increase the life of all our furnitures or giving it a second life. Then, of course, we believe in causes. And we work a lot with, for example, donations towards the impact of the Ukraine war. We also work with the refugees, trying to get them employed, and so on. So these are quite core, and as a co-worker, I'm proud to say I'm one amongst those 81% that feels included with the IKEA brand. But now moving to MLops. Why a platform? I mean, we've been doing machine learning for a very long time. I mean, data scientists have been developing models, all over the place. But now, why are we suddenly saying that IKEA, we need a platform? Because we need to accelerate our machine learning lifecycle. Today, it takes too long from ideation to actual model deployment. And that costs us market share. That costs us sales. Because add-on sales come from machine learning models. Let's say, recommendation models, personalization models that we can put on IKEA.com to make our omni-channel journey more successful. Then of course, standardization and collaborations. I mean, you just notice the scale of our company. And that means that there are many people working in many different ways to a common goal. If we do not standardize the way of working now, it will become very difficult to be able to have machine learning at scale, which is another reason why we need to do this. Because imagine a model. That model will not work for all our 28, 29 markets, or even 42 markets. Because they need to also be market wise. So the sheer scale at which we need to deploy AI is quite large. So the same recommendation model that I have in the US will not work in Europe, will not work in Asia. If it works in Italy, it does not work in Sweden. The range is also slightly different to every country. Even every store, there is slight variations in range with local range, which is essentially some products that are sourced locally based on what people kind of consume locally as well. And then of course, the observability and explainability. This is getting bigger and bigger. We need to be able to observe AI. I mean, it's no point having the world's best recommendation algorithm that's not trained on our latest products, which means it's throwing out stuff which is out of stock. It's throwing out stuff which is not something we want to sell even or are selling even. So impacting the brand. And then of course, moving towards explainable AI, which means we need to more and more look at, why did this AI classify it like this? Why did the AI predict something like this? And that's due to regulations. So if you've been around the conference, you've heard that there is an artificial intelligence act coming from EU. And when being headquartered and working primarily in EU being our biggest market, I think we need to be aware and one step ahead of the regulations. Moving ahead, how did we come up here? I mean, it's not been something that kind of popped in my mind and I went to the management and I said, hey, we need a platform. There's a history behind why we're developing a platform and how we went about it. And it began in summer 2021 when the first draft of the EU legislation was officially released. I mean, it was in the rumors that something like that is coming, but the first draft came out prompting a question, OK, how do we need to deal with this? Because a few years back when GDPR was released, most companies were caught unaware. I mean, after it was live, suddenly, OK, this is important. Oh, how do we manage this? So we did not want to be in that position, which means we started discussions with responsible AI. We started discussions with our data scientists, trying to get them aware, hey, this is coming. I mean, how are you dealing with this stuff? How are you making your models? What sensitive information are you using? You know, some sort of interviews and so on. And then we said, standardized thinking, how do we do that? And we took a hackathon approach. We did two hackathons with our data science teams, asking them all to solve the same problem to see how differently they work, to understand our landscape of consumers. Who is working with code? Who is working with a GUI? Who wants what? So we start to get an idea of how people work in our company. And then, of course, nothing works without aligning it with strategy and architecture. So we need MLOs. How does it align with our data strategy? How does it align with our technology architecture? Where does this fit in? Organizationally, where this should sit, and so on and so forth. And once that is aligned, then you need, OK, what am I going to do? What am I going to do to develop this platform? And that's where you start small, with what's called a minimum viable platform, which is essentially having three basic functionality. Can I create a model? Can I deploy a model? Can I observe a model? Nothing more. Everything pretty manual. And then we start looking at, OK, what are the things that are required? Once our data scientists start using it, they say, hey, this doesn't work. This is required. I need something at scale. I need GPUs. I need, you know, I'm doing generative AI. And then we start to evolve with what other components we need, right? From what's called a MVP to an MSP, which is a minimum sellable platform, which means that a consumer is able to use it. And then we move to a phase called MLP, which comes 2024 and beyond, where it's a scaled platform with 100, 200, 300 models hosted over it, which is our sort of end goal, right? Going ahead, looking at our consumer landscape and what personas we have, research data scientists. So all of this work that you saw over the last two years pointed us to some distinct personalities in this organization that we work in with different needs, quite unique in nature. So the first one that we found was research data scientists. I mean, they work with Academia, the latest model coming out of MIT or the latest paper that's published. They kind of, you know, present at these conferences, their latest LLMs and how LLMs work and so on. So these are like PhDs and stuff who are research oriented. They are code first, they work only with code. They do not want an interface or an experience layer. They just need high compute and they need the right packages to be able to train models and then deploy them. Then we have what's called as a business data scientist. And within this, we have a subset of, you know, business analysts, reporting analysts and so on. These guys use data, but in a repeatable manner, like reporting, they just need, okay, what's my next sale gonna look like, right? They sit, work in our stores. How's my marketing campaign going? So these are now not data scientists sitting centrally in the data science or data analytics, they sit in all the stores. They're like our marketing managers, the sales managers. They are also personas. They also use data. They also need ML. It may not be complex. They may not be working with neural nets, but they need to be more accurate on how do they predict sales? How do they predict footfall? How do they predict the effectiveness of a marketing campaign and so on? And then we have something called a citizen data scientist, which means these are people who are not trained in math stats, but their job requires them to do this. I mean, IT operations, working with operational data, working with incident volumes, problem management. That is also requiring specific models like anomaly detection models or time series, right? So these are citizen data scientists and these both profiles need a GUI because they are not code first. What they want is drag and drop. How can I do this? Click next, next, next. That's all I would like to do. I just need a model. I need a report, right? And then we have the engineering data scientists. Within this is also a subset of machine learning engineer. These are the guys that actually work with production models. They develop and deploy production models. They are sort of our core, something that we rely on to give add-on revenue and sort of business to IKEA, where IKEA.com or all through the stores, right? Moving on, consumer journeys. And this was very important for us. I mean, we said basic develop, deploy, observe. We said that. But we need to make all these phases independent of each other because a platform is not mandatory. I am just a choice amongst many in the world. So we use Google a lot, which means we have GCP and in GCP we have Vertex AI. You can just search for Vertex AI and do your machine learning. Why should I use the platform? What do I offer as a journey? An experimentation and training journey. I offer a model deployment journey and I offer an observability journey. So what does that mean? You can come to us and go through the entire lifecycle. You can come to us, just take a development environment, do your experimentation, take the model out and do what you want with it, right? I mean, you are dependent on us, but if you have a choice, if you think it must rely within your application somewhere and not with us centrally hosted, you're welcome to do so. You have your own training environment, your own data, you developed a model, but you want to deploy it in production and you don't know what to do. Then you can come to us with deployment journey alone. And then of course, there's an observability journey. I've developed my model, I've already deployed it, but you know what? I need to be able to observe to the logging, to the monitoring with it. Welcome, we can help you with just that, right? And it's also headless by design, which means that those that work with APIs and the layers there are not impacted by the experience layer. So they're like completely different. I can just swap this experience layer and scrap it if I need to. Or I can build a much more complicated UX that fits them, but nobody observes the difference. So that's a concept that we work with. And what we also work with is both centralized and distributed. Why? Because if we have to, for example, deploy models in China, we can't host them in EU, we can't host them in US. They must be locally deployed in China, right? And for those that don't know, there's laws around AI machine learning kind of with customer data in China. It's called the China Cyber Law. So moving ahead with all of this in mind, how is the platform construct like? I mean, any MLops platform, any machine learning first of all needs data. And that's not my problem, but we have somebody that looks after data and they follow the DMP or data marketplace with a fair principle, which means it's findable, accessible, interoperable as well as reusable. So me as a data scientist, going through the MLops platform, I come, I have access to data, I request the development environment, enter the experimentation platform and pull all the libraries that I need and start to interact with the feature store and develop the model, right? And once I develop the model, I go into my deployment phase, trying to deploy the model and then trying to get inferencing and all of that. And once that is done, I go to the observability and operational monitoring, which means that observability deals with the drift and all of those and monitoring is more inframetrics in terms of IO and so on. And then of course, if I now own the model, which is in production, I need to be able to manage it. I can manage it via an API or through the backend or I can manage it through the front end. If I am someone that likes to, you know, retrain the model with a click of a button rather than, you know, put in some codes. Now, I will, in the coming slides, start to take you through each of these phases and trying to deep dive in them. I will not be able to go very deep because of course this is a large platform we're speaking of, but I'm around for discussions the rest of the day. First of all, in develop, I mean, these three are quite central, right? I mean, your code repository like a gate, all the packages that you need, scikit, so on and so forth, some shared libraries. So this is something that we offer as a standard. If you take something from us to develop, these basics are already in place and it's like a self-service way of getting it. Next up is the actual development, right? The ingestion and ML pipelines. Now you are starting your work. What features do you require and what features do we need to provide to make sure that you are satisfied as well as our sort of job is done in terms of logging and so on? So first up is then the integration, development environment. It could be a VM as simple as that, right? It need not be much more fancier than that, but it could also be, for example, connected to high capacity compute, which means if you're doing something generative AI, we have GPUs on private cloud that you are able to use. Similarly, now you start your experimentation. There are two key things we need to log here. One is the experimentation tracking, which means what experiments are you doing and registering. Both of these together will then become your model registry in the future. Why is this required? So that you have full control of the different models that you've experimented and you have full confidence on what models should I be able to promote and what models are my backup and so on and so forth. And then of course feature engineering is the interaction from the experimentation platform as we call it to the feature store. So moving ahead, now we actually speak about the feature store, which has a simple, simple goal. It needs to be able to define, maintain and orchestrate features. And the first step is of course the actual feature store, which is a repository where you can share, discover, reuse features, right? So we try to bring this centrally. I mean, if you look at some market products, they have feature stores, but we chose to build a feature store outside because we want one more important thing, that this is a central feature store, no matter you use a custom platform, no matter you use Vertex AI. And we tried to set it up as an inner source project so that the feature contributions are also acknowledged. And then of course, lifecycle management, this is about, okay, if we need to retire a feature, what version the feature is in, or what is the metadata behind the feature, what columns of information was used to derive the feature. So this is important so that the one that is using or pulling a feature is able to understand the consequence of using it, right? And then of course, feature catalog, which is actually even feature reporting in a way, which basically is showing, okay, which are the most popular features used, how many models are using them. On one level, it is a graphical interface. On another level, it's also a catalog with an API. How do I call that feature, right? And then we have, of course, serving, which means the actual process of using, pipelining the feature data into the IDE for feature management and feature experimentation that you saw on the last slide. Now I move to deploy. So I have now developed a model. Now I would like to deploy it. So I use all the features on the platform. I've built a fantastic recommendation algorithm to make sure IKEA gets 90 million more in revenue, let's say, that's how we wanna call it, right? So first thing I need to do is containerization. And we provide the feature for containerization. Why? Because when you're working in a VM with Jupyter and so on, everything works. But you can't deploy that. You need to be able to isolate it, package it, and make a container, docker, anything, and then start to push into what's called as a deployment execution. There are many ways models can be deployed. Currently we focus on only three deployment styles, canary, blue, green, and shadow. Of course, we are looking at the flow first and then the depth, which means at some point in 2024, we can see what more execution styles must be offered. But this is what we have right now. And of course, you can also, during the deployment execution, be able to say, okay, this is a distributed model. Where all should it be? What regions should it be located in, right? And this is a separate process for China. So this will not work in China. And then, of course, the actual deployment pipeline, when everything is decided, you push it through the CI series to be able to go live. And then critical governance. And this is, of course, under development right now, but considering the regulations coming, should we be able to audit the code? Should we be able to see what data was used? Was there any sensitive data being used? How are the weights and bias looking like? Am I recommending a product to my consumer or customer based on his gender, based on his income, based on his race? I shouldn't be, right? It should be a fair model. So should I be able to audit it? Yes. And how do I do that? By making sure everything is locked right from the beginning. That's why we have experimentation tracking, experimentation registers. So when I come to deployment, I need to be able to access these or someone that is auditing is able to access these and say, yes, you are developing the right model. So moving ahead, it's deployed now. So then we have what's called as a task orchestrator and that task orchestrator has just one job. That is to make sure that the ML pipelines are working and the right data is hitting the right model, right? As simple as that. And there you have two types. So one is called batch inferencing. This is very important to us because we have a lot of models which are mathematical in nature. I mean, a data scientist might question if it's a data science AI model, but it's a model. It does maximization, minimization, optimization, let's say, of stock, right? I mean, how do we manage stock the right way? So those run in batches. And then of course you have the real-time inferences which is the recommendation algorithm I just built. And then what is also important for us is logging of each inference, logging of each prediction that comes. Why? Because then this goes into the next phase when I wanna do the drift, I wanna do all the observability and then accuracy and so on. And this is what we call inference platform or the components within that inference platform. So now I have deployed the model entirely. Everything that I want is done. It's working with the inferences, the data is flowing, it's working 90% accurate. Now I need to observe the model. So the first thing is operational monitoring. This is essentially like I told you about the infrastructure where we look at two key things, model performance, which is like response time, and then of course the platform availability and model availability, which means that the model is there and model is functional, right? And then of course the model observability as such, which is of, this is slightly broader because drift, adversial and outlier detection, quite central. Adversial, very important. If there is a model deployed in production out in the world, if there is malicious intent happening, people trying to hack into the model, adversial helps detected, right? And then of course retraining pipelines. If there are drifts, we should be able to automatically or give the data scientists the control to automatically re-trigger retraining or trigger retraining. And we are working with a concept called digital trust scoring, which essentially kind of is looking at your inferences, looking at your drift and everything and giving some sort of a score, a custom metric signals, like you know, signal like red, amber, green and says model is good, right? I mean just as a feature, so that someone that is looking at the model that is not developing the model is also aware of how this works, right? Because it's not only about the data scientists, it's about the larger community that works with engineering because that model from data science is sitting in an application built by other engineers. So this is one component in many. So these two come together to form our observability layer with alerting and so on, of course, I mean connecting it to for example, on-call management solutions, because if the model fails in the middle of the night, someone needs to be called because Europe might be closed, but the US is working and if US is closed, China is working. So we are 247. Now I start to move into the managed piece, the last box that you saw in the whole shebang construct that I showed you. So model registry, I mentioned it briefly before. Primary aspect is to make sure every model is registered, every version of the model is there and the data scientist is in full control of what he's been working with over time. And then of course, metadata repository which essentially brings together what data was used or what columns of data was used, what versions of data was used, what features were used. So it's like a one-stop shop giving model description and basically telling the data scientist this is how you develop the model. Now the focus on manage starts to move more and more into the space of the business data scientist, the citizen data scientist because they have done click, click, click, next and deployed. So this is where the experience layer is kicking in. This is where the whole front end with UX is working. And then of course, the model lifecycle management, this is key. I mean, you are code first or not. Basically you need to know how your model is performing at what stage it is and so on. And then access management. This is important to be able to share models, right? I mean, I have developed a certain model. Maybe you can get inspired from it. How can I share it with you? It's my model, it's sensitive, but yet it is I can't lecture property. How do I actually share it across the organization? So moving ahead a little bit more on what's in the experience layer. So we spoke about sharing models and that happens through a role-based access space where you as a model owner, not only do you have full details of the model, but you're also able to share the model with all the artifacts and packages and so on so that the next person that is working with the model can just copy paste, right? This is where we're trying to industrialize machine learning as well. Because 90% of the models that we develop are actually simple models. It's the 10% which is niche, which is convolutional neural network, generative AI, I don't know, all those which are really, really new, but 90% is bread and butter. Those should be able to be reused. And then of course we have the modeling environments as such I briefly mentioned, a self service, next, next, next. It's like a flow where I say, okay, I want a VM with these scores or these CPUs and I want it with a PyTorps, I want it with Jupyter Hub, I want it with VS Code. What do I want, right? It's like a buffet of options that you get that you just click, click, click and then here it is. Scripted, delivered on your cloud or on our cloud doesn't matter wherever you want it, right? And then we have something called as data marketplace interaction, which means that in this development environment of yours, you're actually able to browse the data catalog. You're able to see what data exists in our environment. You can bring your own data, of course, but what if there is more data that you do not know that exists in the data catalog that might be more relevant for you? How do I interact with the feature store? Is there some feature that I can pull in? So all of that interaction happens on a UX. And then, of course, one click deployment, that's the feature, but that's for those people that have already developed the model. I need to deploy it. I need an interface for it. Here we go. Click, click, click, next, next, deploy. And then one click observability. I mean, these are feature names really that we work with internally, but they, people like them. So we stick with it. Is I've deployed the model. I need to observe it. So cool. Where's your model located? Perfect. Next, next, next. Now starting the observability, which means we start to lock them centrally and start to show them in their model owner space and they start to interact with understanding how the model works. And then, of course, all of this is built on open source, right? So now comes the key part on how did we develop this? What are we doing? So what open source projects did we choose? And why did we choose them? So let me start with why did we choose them? And this is obvious for all of you sitting here working a lot with the open source, but not for us. This is one of our first large scale open source based initiatives that we're running. So for us, how do we choose the right open source? That's with the community volume. Everybody knows it. Obvious metric. More people contributing. Better the project space. Then you know this is something that's running, right? Interoperability is essentially, we looked at what projects are focusing on open standards. Because if they are, then if it shuts down, we still have a way to migrate to another project with open standards. But if it is a little bit more custom, then it doesn't work for us. Then, of course, contributions and updates. I mean, new projects come all the time and the first phase is what we've called the hype phase where everyone is speaking about the project, but then what's the reality? How many consumers do they have actually that have focused on and done something with it? Have deployed a production grade using these systems or these projects? And then, of course, license type. I mean, would be foolish to look at something AGPL, but we look at MIT, PSD standards. I mean, I don't need to deep dive into this, but essentially the idea is we have organizational guideline on what kind of license we use and that's the license we use. And then the architecture fit because now we're speaking of actually many open source packages. They need to speak to each other. They need to work with each other. They need to come in a cohesive environment because if they don't, then I'm stuck with things that don't work with each other. So looking at all of these and, of course, in a very democratic process together with our engineers, we started to look at our current preferences. Kubernetes, I mean, simple. There's nothing, we're never working with ML and if you don't have Kubernetes, then you're an idiot, right? Then code and storage. I mean, the front end is built on React and then a lot of Python, of course, and Postgres SQL for all the orchestration storage and so on. And then some cloud natives things that we work with, Istio, Argo CD, Knatives, Prometheus, you might relate to where they add up as well. And the frameworks for machine learning, MLflow for the experimentation platform, Seldon core for the deployment. I hope you guys know about these projects, right? Because I haven't found anyone with these projects here. I was so looking forward to. And then Alibi, which is also from Seldon for the observability, adversarial and all of those that we spoke about. ZenML for orchestration. The advantage we get with ZenML is that you can place, for example, MLflow below ZenML and if tomorrow you want to replace MLflow with Kubeflow or something, you're able to do that. So that's like the orchestration layer for us. And then Feast is our feature store. All open source. But the UX and engineering is IKEA. How all of these work together, only we know. And of course, to conclude, it's foundational for us to work with MLops to transform our data science practices to start off, be the leader in AI and machine learning. With this, I say thank you very much and move to Q and A. What are you using K-nades for? So it is used for orchestrating the Kubernetes. So because we have multiple locations as well on Kubernetes. Anything more specific do you want? What are you looking at? Because it's almost like an abstraction layer, right? It's simplifying a bit of the deployment. I mean, when there are multiple deployments happening, we need to be able to manage it, right? And K-native allows us to do that. That is what is the value from K-native. So this platform looks pretty super mature and it seems to be cross region globally. How long did it take to evolve to its current state? Was it in a bad state at some point? There was no platform. So we built it bottom up. So we started mid-2022. So we are not completely done yet. So if you saw the timeline, there's still a lot to be done. But what we have is the basic flavor on the flow. Now we are gonna start working on the depth. So yes, it takes time. It's a three year timeline to be able to get to everything that we want to, which means it's a platform with 200, 300 models deployed. It's a three year journey. But the feature engineering part and data collection, that's usually the toughest part, right? There's legacy systems, data that don't talk about. Yes, so that's why I said the data part, we have other teams. So my responsibility is to develop the MLOS platform. So there are different teams that deal with, you know, getting data out of legacy systems, replicating them onto cloud. So we have all those solutions as well that allow for those sort of data. We have a data catalog. We also have a lot of data in BigQuery, which is kind of easily adjustable. So we have a lot of Google ecosystem, so it helps us. And a lot of the legacy data that sits with, for example, Oracle databases, or those that sit in-house in our data centers are also kind of replicated onto the cloud when they're critical in nature. Thanks. Yes, behind you. I know you have mentioned the UX part as IKEA, but is there any component from the open source that you are using for the user experience itself, or at least to automate the workflow of user experience? No, the user experience, so we have something called a SCAPA design internally. So we use that, so it's completely different. It's simple, but... Simple and different. So if you've gone to ikea.com, that's SCAPA design, and that's what you will see with what we have also. Yes? What level of awareness does your data scientists have of Kubernetes and the scheduling that goes on there? That is the reason we had such long sort of time to market, because they don't, right? So, I mean, if you look at the person of an engineering data scientist, they have some DevOps, but rest of the personas, they don't, they're there, like the PhDs in data science, or those, they have no clue. And that is why we need this platform to be able to automate and orchestrate a lot of things for them. So once you've containerized your model, who owns setting up the manifest for getting it to deployment? So it is owned by the data scientist. But what we do is we simplify it for them so that they're able to understand it and own it. And that's why we have all this model owner space. We have a step-by-step deployment flow. We have all the UX so that they understand what they're doing. Because even for example, containerization, that is also not something that they're very familiar with. They're like, here, on my VM, it works. Why am I not? Yeah. So we are at that stage. So that's, if you ask me, what my biggest challenge is, is to actually get our data scientists to work in a standardized manner, to get them educated a little bit on the DevOps concept that is essential for productionization. Because a lot of them come with these fantastic research backgrounds that create data science. But they haven't worked so much with even Cloud, for instance. So we have a journey to make. And then that's the biggest challenge that comes once you have the platform in place. So who's setting resource limits and reservations? So we manage, there is some automation that manages it, but so that's why we have this concept of bring your own Cloud, right? Which means that if they're hosting this centrally or on their Cloud. So they are able to, in their Cloud, manage the resources themselves. So they work, so we have what's called as a, it's a PDEC setup, which means that data scientists are working with engineers in a product team, right? So they have these engineers that are like DevOps engineers that are able to manage it for them. But we have enablement as well to be able to manage it for them. So we also have a concept of FinOps that we're developing, which means they actually start to see the cost. Which means then they start to say, why the cost is? I have set up my resources all wrong. I have just given too much compute. This model does not need it. But it's a journey to make. We want them to own it. They're not in a place to own it right now, but we need to push it. Because if we don't, then we lose money, right? What's the point of having, for example, a fanciest model, but it costs like a million dollars to maintain? Doesn't make sense. So that's also a journey. But they want us to do it? We want them to do it. That's how it is. To be very honest. Thanks for presentation. Are you planning to open source any of that IKEA box in there? Oh, I can't take that call. It's my open source office. So the UX piece, like I said, it's a project called SCAPA. Well, engineering is the orchestration part. And is IKEA contributing to any of the rest of the tools? Not today. Okay, so you guys are basically using it? Yes, I have to admit it. But we have started an open source office the last year. We have now guidelines on contributing to open source, but the legal is still sitting on top of it, right? It's a journey to make companies. There are private contributions, of course. Our engineers are making contributions privately with their private accounts. But as a company, that framework is in development. I mean, for very many years, we were only using bot solutions. And this is probably one of the first projects where we're using so much open source, right? I mean, we're using open source other places as well, but maybe one project here, one project there. But this is a lot of open source. So we are opening our eyes to open source now. And hopefully that changes over time. But if you ask me today, no. How large is your team that's running this stack? So right now it is 11 people working and I will have six more people soon. So it's like 17, 18 at the end of this year. You have clusters on-premise and in the cloud? Only the GPUs are on-premise. Rest is on Google Cloud. And the GPUs are on-premise because Google charges me $18,000 a month for 12 GPUs. And I can fix them at a much lower cost, like less than 30% of the cost when I do it on a private cloud, so. Cool, cool. Yeah, and the team is also spread across. I mean, it's mostly Europe, but yes. We have office in India now also. It's a big platform and it's a big team that needs to develop this platform. Yeah, I'm familiar with a lot of this stack and there's a lot of complicated engineering. Yeah, and you never know. So the other challenge that we have, because every model is unique when it comes to deployment. So the scheme on the fly for the model, it's a lot of challenge. I mean, we say one click deployment, but if it's not a very standard model, one click deployment does not work, right? I mean, if you've not packaged the model, containerized the model properly with all the dependencies, because you're supposed to do it as a data scientist, then you just, hey, this doesn't work. Your platform doesn't work. That's the reality we live in as a provider. On top of Argo CD, right? You've put a UX in front of that. Yes. And that's writing to Git and Argos is pulling it up and deploying it. Yes. It's doing, but still, if the basics are not set in place, right at the beginning, then no matter how much automation, how much user friendly it is, it will not work. It will never get deployed, right? Yeah, yeah, that makes sense. Curious, have you ever been requested to put another, its abstraction layer like the IDP to make it much more simpler? Not yet, but it might come. I do not know. The countries that I mentioned to you, the marketing managers and so on, those people are not yet here. And when they come, all those will start to appear. Right now, it's limited only to Sweden, which means the people outside IT working in stores and so on. And we know them, right? Because we are in Sweden as well. But once we start moving it out to other countries, US, France and so on, not only will there be a requirement to simplify it, there will also be a requirement to have it in different languages. Thank you. That's another journey. Maybe I will come back for it next year. Sorry, just a question. You mentioned, you just started in 2022, was it? We started in 2021 with the discussion. We started working with it actively in 2022. And we are still at the MSP phase, which means we are not at the phase that we want to. We are in that mid phase. So somewhere 2024 is when we'll be done. Well, I find it hard to believe because you have a feature store. You have containerization. You got UX for data scientists. Usually the bottom level of prioritization. And you have metadata for your models, versioning and data lineage. These are ideal things. Usually people don't build them in two years time. So do you think it's... It depends on how many people you are working with it and how independent you are working with it. It's still quite amazing. It is. I mean, I have to say I have some of the best engineers in Europe. At least three of my engineers, one of them is actually even training ML Ops on one of the training platforms and so on. So that helps when you have the right talent working with you. And then of course the right sort of focus to do it. And it's not very mature. Like I said, it's there. It exists, which means we use it. It's available. You can see your model. You can see what versions of the model. You can do retraining. But how efficiently is it working? That's what it is the next phase, right? How do we scale it? How do we industrialize it? We have not gone there. So what is there today is a functional platform, right? So what we follow concept of we develop something, we throw it out there, right? And look at feedback. So everything is developed and it's out there. And like I said, sometimes it fails. Sometimes it just doesn't work. So, but it's there, right? Yeah, that's the amazing part. It's comprehensive. It may not be perfect. It might drop some of them, like one of these. Probably. But I think at some point, I mean, there are so many other things to do. I mean, like I spoke about, you know, how do we serve the same model of different versions of cross different countries, right? So those challenges we've not yet addressed. The capability exists, but even we are afraid to push the teams to actually do that. Because even we do not know how the platform will perform. When, for example, the same recommendation model is in like 20 versions across, you know, for let's say 25 countries. Even we don't know how. Because we've not done that much stress testing here either. Thank you. Peace stage. What was your MVP stack like? So MVP stack was basically one way of requesting. So no UX, nothing. Nothing was there. So we had IDE request, which was just like a package and VM that you get delivered together with, let's say Python 3.9 or something. Everything very, very fixed. So you had a list of items. You don't have a buffet of things to choose from. And then for deployment, we actually, for experimentation, we were actually using a licensed product called weights and bias. And for, for example, seldom, we had nothing in observability like Alibi. We just had shadow deployments. So, and then the observability was like very minimum. We had like Prometheus, which is mostly looking at the, what do you say, model metrics, not looking at the actual observability drift and so on. In MSP, all of those have come. So it was very, very basic. I can develop a model. I can't develop all kinds of models. I can develop models that are developed, let's say, hugging phase, for example. I can do that, but I can't do something like generative AI. Wasn't possible. We did not have the kind of compute and so on. So I would say half of this you can take away. The top you can take away. So even, we did not even have a proper database. Postgres, it was all, nothing was really saved. So it was like in the air. There was no traceability either. It was just like, okay, can you do this and see how it works? Please don't do it with production, right? MVP was never to productionize a model in production, which means you don't give it to the end consumer. Yes, you deploy it, but in a test environment. Could you tell us a little more about how you get like real-world data from web browsers, for example, and send them to models for inference? Real-world data? Yeah, like user data from web browsers, and then get them to. Fastly edge, so we have a lot of data. We have 3.8 billion visits to ikea.com. So we capture the data with, of course, consent. So when you click accept cookies, there's a lot of data flowing through. Session data, there's a lot happening on Google Analytics also. So we use Google Analytics quite heavily to track the website. So we have a lot of data. Then, of course, we have a lot of, all the sales come through, and that is real-world data, right? I mean, so 42 billion sales, we have data for 42 billion sales. What products were bought? What products were paired? And what products were paired in different markets? And who are our consumers if they have an ikea family, if that's like a loyalty program? So then we have the purchase history. So we have a lot of data. It's about how can we use it ethically for the benefit of the customer and our own benefit as well. Data is not a problem in ikea. We have a lot of data. But using it effectively, I hope this can help us use it effectively, which is the purpose. Do you have any, in this platform, I guess, in terms of real-time model inference, for example, like displaying results of a user's like product recommendations? Yeah, the inference, it comes through. Yes, it's logged. It's not shown on the UX. It's just logged, and it basically then correlates and does a drift and shows. It doesn't show every inference. It's logged in the backend. That's all that is happening. But it's there. There have been asks to, you know, do the inferencing during the deploy. So after you've deployed it, you can run inference tests. So that we have not done, for example. If there are no further questions, I thank you for your time and thank you for your questions. And I'm here the rest of the day and tomorrow. So if you wanna discuss more, happy to catch up. If you wanna discuss even more, happy to connect you to my engineers as well. Have a nice day, everybody.