 Great. All right. So very excited for today's topic. This is a topic that I have shared in previous Conferences, but it always keeps evolving and it keeps evolving really really fast So today we're going to be diving into the state of production machine learning in 2023 there's going to be a broad range of topics that we're going to cover on a very high level Because this have been topics that have been covered in previous conference talks So what you will find is you will find actually references that will allow you to dive deeper into each of these topics So pretty much a slide will you know consume your entire weekend on a rabbit hole learning about it? Hopefully for the good a little bit about myself. So my name is Alejandro. I am currently director of technology at salando I'm also a scientific advisor at this to for ethical AI and governing council member at large at the ACM So as I mentioned, there's going to be a couple of links that you can dive into So this is from previous talks that I have done but also interesting references that you can check So you can find the slides on that top corner. So yeah, please You won't have to take all the pictures and keep them. You can actually access the slides So what we're going to be diving into today is five key Sections starting with motivations and challenges. Why do we care about production machine learning? Then we're going to dive into some trends that have been emerging in industry and also On in academia and in society as a whole We're going to be then diving a bit deeper into the technology and then we're going to be looking at the organizational Considerations themselves as well. So by the end of this talk, you should have an intuition of the, you know, very high-level societal Implications and considerations, but also how that reflects into the technology and then bringing some insights that you would be able to not only Add into your tech stack, but also into your teams and into your organizations. Hopefully So starting with the motivations and challenges. So one of the things that has become crystal clear Is that the life of the machine learning models does not finish once they are trained, right? Like that's Only the beginning and it's only when you actually start consuming those models and really getting value out of those machine learning models And more specifically machine learning systems as we will see today You'll start seeing challenges that you are not going to to see when you are in your experimentation phase, right? You're going to see things like, you know, outliers You're going to see drift from the moment that the model hits production. You already see a degrade that has To involve certain considerations, right? So as part of that this is going to be one of the core principles that we're going to be fleshing out But we need to ask the question of well, why is production machine learning so challenging? What are some of the key areas that make it not only difficult but also different to traditional software, right? So some examples that may be different to traditional software microservices are that there is specialized hardware, right? So when it comes to the productionization of models, you have to involve not just special accelerators like GPUs, TPUs, but also in some cases very large amounts of memory, right? For example models that Require a lot of RAM or a lot of VRAM You also require perhaps special compute and as part of that that involves complexity in the orchestration of your models themselves as You know, you reach harder scale Larger scale. There are complex data flows, right? It's not just about the model itself and the inputs and the outputs, but it's Potentially the impact that Considerations can have up the stream or down the stream, right? There are compliance requirements particularly when it comes to machine learning It tends to be also very very tightly closed into the domain use cases, which means that often there are a lot of Compliance requirements and in some cases even ethical requirements When it comes to the productionization of machine learning systems and then another area is the reproducibility of components, right? It's not just the deployment of the of the application of the code, but it's that combination of code Plus environment plus artifact, right? And then the the versioning of that and making sure that you're able to have that reproducibility so that You know, you can introduce that determinism into your into your environment, right? So if you want to dive deeper again, you know This is the links that you can find for for talks that actually in this case talk specifically about the challenges But then going one level higher, you know Why is production machine learning so challenging at even the societal level, right? And this is part of the point that it goes close towards that use case Specific area. There are challenges that you may have heard already particularly in high-profile cases in the news Around algorithmic bias, right? Whether it is discrimination Due to undesired bias within the within the models and that in itself again, you know, there is a very interesting Field that that you can delve into to understand about that explainability interpretability bias There is also the challenges that you have in traditional software, which is basically software outages, right? What happens when when your actual infrastructure falls down? There's the misuse or the challenges that come with the data itself And then there is an element of cyber security Which we are going to be diving into and that it's exciting to see that there's a lot of topics that are coming up Now especially in this conference There are a couple of exciting talks on cyber security and then of course, you know, it couldn't be a state of production ML without the LLMS But this in itself You know doesn't really change the whole challenges that machine learning introduces But I think you will see as part of the examples that I will be giving in this talk It makes them a little bit more intuitive, right? And it makes the need for this production machine learning considerations more clear So I think that's that's the one thing that has been you know brought Beyond the hype that we will benefit from and this includes complex architectures That will be described as this Data-centric view of machine learning that involves multiple components, right? You have seen most likely when it when it comes to the world of applications on LLMS That you would see the use of machine learning models in ways that are very creative, right? The machine learning model interacting with APIs interacting with databases and then bringing the prediction together with that that combination So so it will kind of like give you an intuition of some of those challenges now In order for us to tackle those challenges It also involves some considerations of the skills required for those Challenges, right? And this is something that that has now become a little bit more ubiquitous and Standardized and understood is this intersection of the skills between software engineering data science and DevOps or Platform engineering, which is basically the skill set of machine learning engineers or MLOps engineers So this skill set is something that has now become even more prevalent within data science teams as as their requirements and Productionization grows and we will actually touch upon that in the organizational shapes in a bit But then there's also not just an intersection of skills But it's an intersection of domains themselves, right? So you have the intersection of the knowledge required within the machine learning expertise But also the industry domain expertise and then as we will see as well that policy expertise, right? It's how do you make sure that of course you're doing it correctly from a technological sense? But also that you're aligned with the industry requirements and then aligned with the higher level considerations But one thing that that we will see as well is how how to think about this right because right now It's very very abstract So for that now let's dive into some industry and domain trends, right? So we're gonna go a little bit deeper still very high level We have been seeing that and this is something that I really liked how it was verbalized by the Linux Foundation Is that we started with this description of AI ethics, you know, perhaps in like 2015 to You know 2018 then we went into Responsible AI which is basically okay. Let's discuss the higher level then let's discuss the best practices And now we're talking more about accountable AI, right? How do we actually? Hold people accountable Introduce whether it's through regulation through policy through standards things that allow us to understand what best practice It looks like and what should be the bare minimum in some areas So in this case, you know the way to think about it from a hierarchical perspective You of course have those very high level principles and guidelines that give you that North Star But then from that you have to get a little bit more concrete What are those industry standards those regulatory frameworks those even organizational policies to be compliant for? certain requirements, but then similarly what is absolutely critical is not just to have those North Star principles But is to make sure that those open source frameworks that are now becoming ubiquitous within industry and academia They are also by design aligned for those principles, right? Because you know in essence we can have as many roundtables as we want and we can all agree that discrimination is bad But if the underlying infrastructure and the foundation is not Enabling this by design and it's not built with those considerations in order for them to be empowered then you know, we're not going to be able to achieve what we are setting out to and Similarly, if we actually see it from an organizational standpoint The thing to emphasize as well is that large ethical challenges or even large compliance challenges So should not fall on the shoulders of a single data scientist or on the shoulders of a single software engineer, right? Because of that it is not just required for individuals to be responsible, right? Because also one thing that you know, we have seen in the past is that you can have a situation where you know a group of Individuals are all responsible or ethical, let's say But then the outcome as a whole may not be right and that doesn't mean that people were just sitting there thinking How can I build the most? You know racist algorithm that I can possibly do right and you know when we've seen that in the in the high profile cases in the news I hope that that's not what you know the people are thinking but I most likely it isn't right and and that emphasizes that It's not just about the individuals. It's about the compound So what that means is you know, of course from a personal perspective having that sort of technology best practices Being able to work in the areas where your competence is relevant Having those, you know areas of professional responsibility like the ACM has Code of ethics and professional responsibility that you know, we always kind of recommend them point to because you know It's just from the ACM, but then brought brought it to this You also have the team right and how you make sure that there's that cross-functional Skill sets that balances each other how you have the key domain experts how you have like the relevant Alignment within that and then at an organizational level. It's also important. How do you introduce the relevant touch points? human touch points that ensure that the you know respective domain experts most likely in several cases non-technical Will be able to provide that, you know human Decisioning right and and ensuring that that accountability can be distributed as opposed to just you know ensuring that there's just a single single individual accountable and of course, you know one thing that has now become a little bit more real is And that you know, it's it's really interesting to see is that before we were talking about how regulation was playing catch-up But now it is tech companies playing catch-up with some other regulation that is being rolled out So recently we actually submitted a contribution to a consultation for the European unions AI Act which is going to you know come to force in a couple of years So now actually companies are thinking how do how are we going to to roll this out? Like how are we going to introduce the processes now similarly with you know the the the other regulation that the other regulations in the EU But we're seeing similar things in other parts like in the UK Again, we submitted for their current initiative for they call it the innovation first AI regulatory framework and for that they're also looking and thinking how can we achieve? The best practice is being rolled out within the organizations whilst encouraging innovation, right? So it's it's really kind of having a balance of both areas and again This is to emphasize back to the point that I mentioned but making more of a call to action to the people in this room, right? That of course we can have all of this, you know regulatory frameworks all of these principles But really it is you know the foundation that now actually runs a Large percentage of our society this open source frameworks Not just machine learning frameworks, but also just general software frameworks that are required for you know Individuals like ourselves to be involved in this discourse, right? Because we are really guiding and supporting How this best practices or how this? Mitigations of bad practice are formed and rolled out in society. So I mean and the ways to get involved as well I mean ultimately are are as simple as just attending conferences like this but also getting involved with open working groups like from You know the Linux foundation or the ACM So that's I mean that that definitely would be a call to action for anyone that would like to get involved in those so now actually let's go a level deeper into the technological trends and You know some of the the tools and frameworks that have been growing within the ecosystem So yeah, I mean just as a refresher, you know back in the day, you know, this is how it started right like simple You know you could just pick and choose and it was it was easy, but now it's a little bit harder Right, we have a very very large tool set and I think since the since the rise of LLM's It's like one every hour Pretty much, right? So so so yeah, so so the question of how to navigate I mean not what I do want to highlight here is a bit of a plug Of one of the the frameworks that we actually maintain so this is one of the awesome production machine learning frameworks and get hub and Actually, we are celebrating its fifth year and we just broke 14,000 stars So, you know our call to action is not to just you know go and add more stars unless you want to but To actually add PRs and add anything that is missing We are a little bit more strict on what is added now because you know, otherwise it would be like enormous But definitely if you're interested like this would be a great also way for you to discover New areas right like diving into into tools But yeah, so so now putting a little bit kind of like into a shape this this this set of frameworks If we want to like get an understanding of what is the anatomy of production machine learning, right? So the way to the way that like I'd like to think about it So let's think about basically this this sort of production Blueprint where we have the training data the artifact of the train models and then some inference data We start basically on the experimentation where we train models through that training data Doing sort of like whether it's an workflow manager or a notebook ETL or just a notebook to generate train models, right from that we want to Do something with this train models. We want to be able to make them available for consumption So we would be able to do them either manually by you know create it like publishing our Jupiter notebook Don't do that or by properly, you know, production icing your your machine learning models Into, you know an environment with basically in this case your offline or real-time or semi real-time machine learning models ultimately also introducing observability and monitoring things like drift detection outlier detection and In production observing basically inference data, right like running inference on on unseen data data points ultimately with the The objective to be able to make use of that inference data at some point whether it's for training data or for analytics There are some some some relevant use cases and of course the metadata that actually interoperates around this is something that you know would Make the whole picture of this You know anatomy of production machine learning So one of the things that we have also been seeing is now looking at this as just an architectural blueprint If we look at the question of well, okay What frameworks can I pick and choose there and what we're seeing is that there has been a conversion? Or yeah convergence into the concept of a canonical stack, right? It's basically having elements or or sections or little kind of like placeholders That serve for particular Outcomes right like your experiment tracking your data version in your experimentation your model registry your monitoring, right? And as part of that you have an ability to choose different frameworks So now what we start thinking about is how do we then? Encapsulate those components and think about in a way standards that we expect frameworks to be able to have so that we don't end up in a World where you know just everything is completely different and we end up kind of like with different standards that create more standards Right and we're gonna touch upon that this actually is a very interesting tool that you can use to just you know Pick and choose your your your frameworks But now there's actually starting to be a little bit more convergence Which is interesting to see like they're starting to be a bit more preference to certain tools and certain combinations of tools Also depending on the scale of the projects Another trend that we're seeing is that people are starting to also realize or well I mean realize and also put a name to it that when we talk about production machine learning We no longer talk about the production model, right? We actually talk about a production system and what that basically means is that we stop thinking about this model centric machine learning And we start thinking about this data centric machine learning, right is the question of how does your data flow? What are the transformations of the data as they go through your system and of course, you know here is an example of a architecture of the Facebook search so I actually cut the yeah, here's the diagram. So here you can see that there's actually an offline and an online Sort of section so basically training the the embeddings and then being able to use them and you see that there's multiple stages, right? As part of this there's not just going to be multiple machine learning models But there's going to be multiple versions of those models multiple Relations to the training data and multiple components that are not machine learning related, right? So when we think about this machine learning system, it's important To to understand like what does that mean in terms of intuition because when you look at something like this Facebook search I mean, maybe it's a little bit abstract This is what we can go back to what I was suggesting in terms of LLM's providing a more intuitive picture And I think right now when we start seeing this age and chain architectures of how people are thinking to deploy a LLM that then interacts with let's say another API or that then interacts with a database So that in itself is a in a way partially a data centric machine learning system where you are expecting multiple Multiple flows of interaction, right? And I mean there is there is of course You know increasing complexity depending on how large is the system? But the way to also think about it is that each of these components will also Introduce the challenges that we you know revise at the beginning and will benefit from the production machine learning considerations that we will talk about right all of this Monitoring metadata management Every single component is something that you'll have to consider as part of your your machine learning system But yeah, so this is just another topic that you know, you can spend an entire weekend Well, I mean people spend their entire PhDs and careers just just on that But you know all of these areas are are are definitely very interesting to dive into so Another thing to take into consideration So as part of this machine learning systems We also want to understand what are some of the relationships between the components, right? And we I mean probably most people here have actually come across the concept of a model registry Right like an ability to be able to you know keep track of your trained machine learning models But when it comes to production machine learning, we actually introduce a new paradigm that has to bring new sort of like New new considerations, right and let's actually see that intuitively let's say that we have a data set So instances for data set a so we have basically all of these instances That we then used to train a model, right? So we run an experiment we train model artifact a one, right? We train artifact all the way to AM, right? So we have basically a data set that we are using different parts to train basically different models within an experiment But then we also may have other model artifacts that come from different data sets that in itself is your artifact store Right, but then what happens when you productionize your models, right? You productionize your models Let's say you productionize your artifact AM And you are productionizing it with certain configuration, right? Then you may actually productionize it again with in another environment with a different configuration, right? And then we you may actually productionize a combination of these models as a pipeline or as a as a data flow Component, right? So as part of you know, this introduces considerations that in your traditional artifact stores, you're not fully capturing and as part of that you do have to you know make sure that those things are Considered when it comes to your production environment, right? Because if something goes wrong with model AM, right? Then something will happen as part of your pipeline then something, you know We'll have to be debugged and you'll have to consider with that Second model and you will have to understand kind of the whole picture then bring in the relevant experts You know who trained model B1, right? Who trained model a AM? But yeah, ultimately, this is just for an intuition. So hopefully doesn't confuse you a bit more for the next section And this next section is is saying basically, okay We so we have multiple models in production that have sort of multiple considerations in this Sort of machine learning system now as part of each of those components You will have also to introduce the best practices around how do you keep track of them? Right is how do you know when something goes wrong? And this is basically by introducing things that you know in traditional software would be just monitoring, right? And traditional monitor the traditional software monitoring would be things like what is the request per second or what? What is the throughput of your service? What is the current CPU usage? What is the GPU? Well ram usage, right? And suddenly it crashes you start seeing that the The chart and then you see that there's like a consistent chart and then you realize that there's a memory leak And then you know you go and address it, right? But in machine learning there are further considerations when it comes to monitoring for each of these components, right? It's monitoring of things like statistical model performance, right? Like what is the accuracy of your model in production? What is the precision? What is? The recall of your model in production and in order to answer those questions There's an element of data labeling, right? Like that you have to know what are the actual what are the actuals in your production environment? So that introduces basically the questions of how do you then you know bring that into your production environment and monitor that? There's also Things like explainability, right? How do you make sure that whenever there's a prediction? You can explain what happened as part of that prediction that in itself is another you know area of You know Research that has some really interesting approaches and then also the question that we were discussing about well as part of your inference data You may also want to get some insights, right? What is what are the distributions of your production data? What is basically perhaps use cases that you can bring into the organization from your inference data, right? So so so those things are our considerations that go beyond the traditional Monitoring of software and then similarly how do you introduce observability on top of that, right? So things like SLOs So service level objectives alerts SLIs for your for indicators so that you don't have to just be checking on the model in the dashboard, but you can have you know Pages for your teams so that they can be notified whenever actually there's a problem So so and then finally things like you know drift detection outlier detection that you can introduce as part of your your stack But again, so so you know each of this each of these areas is probably like you know a deep dive in itself But you can actually check out in one of the talks that we gave last year on Production machine learning monitoring which is interesting itself now another consideration To take into account is the challenge of security, right? So this is this is something that comes up a lot when it comes to traditional software But in machine learning, it's not something that is discussed as much. So when you think about security the first question is where is Security relevant, right? Like what part of the machine learning life cycle do I have to think about security? Is it on the data processing? Is it in the model training? Is it in the model deployment or is it on the monitoring, right? And the reality is that it's basically like across all right I mean, well, that's supposed to be like a red line across all but you can see it Well, but yeah, so it's basically saying like every single part and every single stage of your machine learning life cycle is Acceptable to vulnerabilities, right? And it's something that now the community is starting to think and ask the question of well What vulnerabilities? What does that mean, right? so as part of that we actually have Started doing some initiatives exploring the security risks of machine learning, right? And actually, I mean if you if you do want to get involved we are running a Committee as part of the Linux Foundation on machine learning security where we are really trying to explore What are some challenges within within security examples of this would be challenges and risks on the model artifacts, right? Basically a potential injection of within within the binaries, which I think there's gonna be actually a talk on that Yeah, on poison pickles. I think it is so do check that out Challenges of accessing to the model being able to reverse engineer the models and exploit them Challenges of dependency or supply chain attacks being able to inject in the dependencies Challenges within the infrastructure that the model runs, but ultimately, I mean, of course This is a long running list the main thing to to to think about in this in this specific area Is that as machine learning practitioners? There is an element of security that in certain Extents you don't have to a hundred percent, you know care about all the time and try to kind of like, you know Address it every single time just because we can't make every single data scientist also cyber security experts But ultimately is to think about in the from a holistic sense similar to how it's introduced in the software development lifecycle, right? So how you have traditional as SDLCs that we will see in a bit So yeah, so that's that's basically some some trends when it comes to the to the Security side now as part of the last area loudest talk about teams and organizations So how does this look like when it comes to my team to my organization? How can I roll this out if I was to bring this to to to to kind of like my area, right? The first thing is that we are starting to see a trend Where basically this concept of SDLC so software development lifecycle, which like organizations tend to adopt and tend to roll out basically now you don't find organizations or but I mean not very common to find organizations that are putting stuff in production without an operations team like a DevOps team with CI CD with like, you know Testing etc etc, right? So but when it comes to machine learning and this concept of the machine learning development lifecycle It's a little bit different because you cannot just roll out something that is the same across every single Use case right because different use cases will also have a different level of risk And also different use cases will have a different tech stack, right some machine learning, you know teams maybe much more analytics Heavy as in from from analysts that are performing, you know that are using Like analysts stacks something that is a you know perhaps more higher level Versus something that is a little bit kind of more machine learning engineering that involves productionization that involves real-time Inference so there are different considerations at the at the tech stack level But also at the domain level in terms of the risk, right? Some may actually involve heavy compliance requirements others may not right so when it comes to this ML DLC It becomes more of a framework to adopt Best practices that are relevant for for the for the context now as part of that You know we talked about a little bit in terms of the components But something that we're seeing as well as part of trends in the organizations is the concept of metadata itself, right? Now we have different components different systems different frameworks some that are doing our our model versioning some that are doing our model Artifacts some of that are doing our model serving How do we make sure that we have lineage across all of this so that you know that if something went? Wrong in production even at a compliance level you can actually go back into the training side to understand You know perhaps. What is the the linkage between both and this is harder said than done ultimately because when it comes to metadata You know it also often involves standards to be able to ensure that it's Homogeneous enough that it can be processed and that can be handed over in a way, right? So what we're starting to see is that there's standards that are that are trying to standardize Standards right so what we want is to also make a bit of a cold call to action To not do that right and try to kind of like contribute to potentially existing ones that and try to even as open-source project leads if there are present in in in this Room to also think about how can I standardize and and and bring? alignment into this broader ecosystem So but it's actually in itself is an interesting one and again There's a talk that it was actually two years ago on meta ops So I mean it sounds more boring than it actually is it's actually pretty interesting or I'd like to think that but yeah Do check it out if you're interested and then finally the last couple of things to mention is that? You know people in the in the data space Would have heard of this you know buzzword of data ops and data meshes So we're now starting to see a bit of a convergence between this ml up space and the data mesh Concept, which is basically thinking about even ml ops not as a single sort of centralized Data lake where everything just gets put there But as something that actually interoperates and is closer to the domain and that serves kind of multiple different domain specific Expertise but also has an ability to like interoperate on that so that that in itself is another interesting area that I definitely recommend diving deeper this intersection of data mesh and MLops now when it comes to the the products themselves We need to also start thinking and having a mind shift of machine learning as products themselves right not just as projects So ultimately them having roadmaps that don't you know that that that involve that sort of feature improvement incremental improvement Perspective that has been ubiquitously adopted in the software space right is really kind of seeing this this this machine learning Machine learning initiatives as products that would also involve that product Mindset when it comes to that like refining delivering value iterating and that similar thing is also reflected into the mindset that we're seeing within Organizations is thinking about their machine learning also as this product roadmaps right so you have the investment at the Infrastructure level the investment at the tooling and then the investment at the actual value delivery for businesses But how does this map into each other and how does this actually you know become kind of this product mindset? How do you iterate from from those things and then finally this is the same thing for teams? We're starting to see this concept of squads coming into the machine learning space this cross-functional You know feature driven iterative combinations of you know researchers But also engineers that are delivering value as they would with with a product and similarly see starting to see the the rise of this Machine learning product managers and program managers that you know organizations are starting to to really standardize the words now A couple of final things from an organizational perspective is that also you have to take into consideration that when it comes to all of this Elements that we are talking about this will come as your complexity increases, right? So it's not a big bang that you should just start with all of this complex infrastructure and bring in the full wrath of Kubernetes and and you know bring in like everything like scalable for a billion users from day zero But instead it's actually thinking about this in an iterative way right as you start bringing more You know a few models You know the common thing to see is a combination of data scientists data analysts as more models start coming in We start seeing that machine learning engineers start coming into the picture because otherwise You know data science is just end up with a lot of the operational burden And then as there is an increase of that there is this sort of more specialized machine learning platform engineering roles and then you start seeing also that increase of those elements of Automation standardization control security observability and then similarly the way that we think about it in this sort of like product mindset When you have basically the the group of data scientists You may have them focused on a particular use case But this this product mindset is to think about how you can enable Those data scientists to be able to deliver value in a way that Increases without having to also increase the headcount right like not not requiring that linear growth in terms of the number of people With the number of models or the number of the amount of value that you're delivering and as part of that that is when you start seeing that automation right like pipelines are are introduced so that the Science experts are able to start you know creating value and repeatable value and then as as it starts becoming kind of like more and more sort of As you start increasing increasing this sort of like product perspective Then you start being able to increase even the the the level or the higher level in which it's operating right so then it's actually Data analysts or even business stakeholders that are interacting with this data and machine learning products to to carry out the the outcomes So yeah, so that's that's ultimately yeah the the main Areas that I want to highlight just to wrap up one thing to remember and this is very important Right is that from all of the problems in the world, right? There's a very very small chunk that actually are solvable and should be solvable with machine learning, right? So when you're running with a hammer everything looks like a nail, right? So we have to remember that the first question is whether machine learning is actually relevant Most often and statistically the answer is no, right? So yeah, that's just something to keep in mind And also the last thing is that we have to remember that as practitioners, you know We do have a big impact and we have a lot of potential to drive value and change within this space, right? Because large and growing amount of critical infrastructure is is depending on machine learning systems that we are, you know developing that we are maintaining And always the impact is going to be human irrespective of the number of abstractions and and and data products and machine learning products That's always something to to to remember So yes with that said it was a lot of content, but I'm glad that everybody's still awake. I think So thank you very much, and I'll take some questions now and maybe more in the pub later. Thank you Amazing thanks so much. I know that was that was riveting honestly So there's definitely a question. So I'll verming Mike, but there's also stationary Mike Hi What do you think it's the optimal team size in the future when you are running machine learning products in the company? Is it more like a Hierarchal structure or a small independent teams? No, that is a great question. I mean, I don't think there's there's a Silver bullet number But what we started to see is that there was more of a ratio and what we start seeing is is basically that ratio of Not even to say number of data scientists to number of machine learning engineers to number of machine learning operations or platform engineers, but also to the Outcomes that this are providing so it's similar to when it comes to the questions of how many You know software engineers or how many engineers? Would you want in an organization? I think it also becomes a bit more of a ratio that ultimately is to the infrastructure and to the overhead The the trend that we tend to see is that machine learning engineers are included once the data scientists are just getting Overwhelmed on just doing operations and engineering and they are no longer actually doing data science So yeah, I would say it's more of a ratio from what we see than a specific number Thank you for the great talk my question is regarding the new generative AI that's emerging and all the services, you know AI as a service that appears and The high quality of what we see the routine that's a threat for AI researchers and data scientists in term that We can't build like regular data scientists cannot build something like GPT So just to make sure that I got your question. So it's your question that you're seeing a lot of innovations in the large language model or just like generative AI and is your question about Researchers not being able to compete with that like amount of hardware that is that is happening Or is with the use cases that are coming out with the use cases I mean In the past when you need a new product you hire some researchers they build it and now you just get GPT right, okay, okay, well, I mean so I I Yeah, I'm a bit a bit critical of all of this like generative AI hype because I mean there is certainly like an element of Yeah, huge potential and huge value And there definitely will be a lot of transformative elements around it but something that I have been seeing that is becoming a little bit more clear and more prevalent is that what is gone like What is converging towards is not that you will end up with a complete automation of just researchers will just not exist and will not research like researchers or engineers or or even like full-stack engineers that are creating products, but is that those domains will evolve and the Practition the practice will act will be Acting at a significantly higher level, right? So they will become experts in building something along those lines and we're already seeing that right in those sort of complex Agent chain architectures that I was talking about that starts delving into the concept of this data flow machine learning where I would say the the the more boring and Traditional best practices of MLOps becomes even crystal clear You know critical right and that means that the jobs for machine learning engineers machine learning operations platform engineers I mean they're they're going to be there and also like for data scientists. I think But yeah, it is exciting There are you know still quite a while to go to really get it right. I mean from what I see but Yeah, that's my perspective that it's that it's a little bit more of kind like an evolution of profession. Yeah Thank you for the presentation. So Can you recommend for example three frameworks that help we've running machine learning pipeline in production? You know top top three your favorite So, I mean that would be a hard one What I would point you to is is to have a look at the two things that I showed the first one is the Canonical stack because that will actually give you a lot of guidance for each of the frameworks that are recommended and that are Very popular And I will also recommend you to check out basically the the list of MLOps frameworks Yeah, I don't think I mean Yes, how long is a piece of string? I mean what I would recommend actually there was this article that we shared in our newsletter Just last week which was called MLOps at reasonable scale and they actually had a practical MLOps pipeline end-to-end that was kind of like balancing on effort together with Scale of scalability. So, yeah, I mean that would be a good one to to get started with So, yeah, sorry for not giving like a very specific one, but you know, this should give you enough For you to find the ones that are right for just playing around or actually bring into your environment. Yeah, thanks Okay, we've got two minutes. So Can you do one minute on each? Hi, so I'm wondering I didn't see you talking about experimenting with models in in production What do you think about practices there? Is there any any good framework to follow? So with experimentation what we see is mainly to have basically those considerations of experiment tracking artifact management Lineage of metadata so I think you know really kind of following those Suggestions of the tools of the of the of the canonical stacks gives you an intuition of what are the best practices that? Need to be enforced within them because for example the lack of having that Experimentation management tool or or or model tracking tool would mean that yeah, you just don't have that best practice So yeah, I would say probably along those lines. Yeah And then final Hi, thank you So I feel like in recent times there is this move from just data scientists to machine learning engineers doing like end-to-end So I'm just wondering what do you think about that? Will we move from machine learning engineer, which is able to do both a data gathering machine learning Developing the model and then implementing it or you'll see all the different positions still existing Yeah I mean so like with a full-stock engineer in the traditional software engineering that does you know front and back end I think you know in earlier stage products and even earlier stage start-ups You may find individuals that may actually master both and be unicorns and be able to do all of them But what we are starting to see is more Consolidation towards specialized skillset so still with the data scientist, of course getting a little bit more of that engineering acumen That is definitely a trend, but ultimately the machine learning engineers being the ones that focus more on productionization But still with the knowledge around the machine learning specialization So not being as much of an expert as a data scientist is for data science But same the other way around so yeah, I mean a lot of us to a full-stock engineer in software development Okay, I mean we can if anyone has a really Urgent question or you can maybe catch Alejandro in the break between the keynote, which is by the way on LLMs So nice segue. Okay. Well, let's give Alejandro a warm applause