 So today I would like to start by asking you a question. So do you remember this time when maybe you were four years old, you were really like dinosaurs, and you also liked drawing? So maybe one day your parents asked you, can you draw a giraffe for us? And you said, like, yes, I will do so. So you run into your room, you took the crayons, you took a piece of paper, and soon the masterpiece was ready. So then what you did, probably run back to your parents. However, maybe they were no longer there, which was confusing. So you went looking for them around the house, and you found them in a kitchen. And they took the drawing and said, oh yeah, that's really nice. Thank you for that. However, you have never seen the drawing being put on the wall in the living room for all of the guests to see, which was confusing to you, right? So this thing that you got back then is also something that you may feel right now in your job, for example, as a software engineer. And this is also something that sometimes I experience. So my name is Paulina, and today I will tell you a little bit about the intricate art of making your internal clients happy, and all of this from the perspective of infrastructure team. So what we have on our plate today. First, we'll start with explaining ourselves what this internal client is. Then I will tell you a few words about my team. Then we'll proceed to establish what infra team is. And then we'll move on to the challenges and something that you are probably most interested in, so solutions. And here I will also mention what worked best for my team. And at the end, the summary. OK, so we are probably familiar with this pattern. So we have your company. In your company, you have some team X. And this team X provides some sort of product, some value, to somebody outside of the company. Then this person is your external client. But it may also be the case that your team does not provide value for somebody outside of the company. Rather, your team works for another team. And then this team X becomes your internal client. You may wonder, what are the differences? Are there any between external and internal clients? So first up, usually external clients ask you to do something. They want to get some value, some product. But with internal clients, it may not be the case. Internal clients, basically with them, you are the one responsible to learn what they need. You need to create this value for them and then try to convince them that, yes, this is something that will benefit you. Next up. So with external clients, it is pretty straightforward because they pay you for the value that they get. But with internal clients, it is not the case. Obviously, the employer pays you, but there is no direct cash flow between your team and your client's team. And what comes with that? When the external client asks you to do something and they pay for it, basically they will be most likely interested in their progress. But with internal clients, it may not be the case. They may be more focused on their own priorities to deliver to their client than learning about some tools that they may use or maybe not. So a few words about my team. I work at Brainy, where we deliver our products to over 300 million unique users monthly. And these users come from all over the world, so Europe, the United States, but also, for example, from India or Indonesia. And at Brainy, my team is called Machine Learning Infra Team and the eight people with, as you can see, very diverse roles. I am a machine learning engineer in the team. And my team belongs to the department of A services, where we have five teams in total. And this makes around 50 people on board. So as I mentioned, I worked in Machine Learning Infra Team. But what this Infra Team is, the story starts with our CTO, who had worked with a couple of different companies, and he observed a pattern. That whenever you have a given domain, it usually turns out that the teams in the domain develop their own toolboxes, which may be. For example, a set of scripts that make their life easier, or maybe they try different tools and find which of them work best. And they collect all of those things into their own toolbox. But what the CTO also observed is that you can take in other efforts. So have a dedicated Infra Team for the domain. And then this team can see what are the common patterns between the teams and develop a toolbox that will be generic enough to fit many teams, but that will also bring all of them the value. So what happens next? Basically, this toolbox can be given to each of the teams, and the teams can use it. And what is there for us? What we want to get from this, basically? Those teams can focus only on their own work, their own priorities, and they don't need to reinvent the wheel. And what comes with that is that no reinventing the wheel, no time span on that, so the costs go down. Basically, in the end, it benefits the company. And as you might expect, my team follows this pattern. So we are in constant communication with each of the teams. So we got the requirements, we get feedback from them, we know about their projects. But what they get from us is value. It can be a standard, it can be a specific tool, or maybe something else. And if you want about specific responsibilities of my team, as you may expect first, it is research because we need to learn about different options that we want to train. And out of that, we may propose standards. And sometimes, we are also developing our own internal tools, most likely in Python. Then when the standard is in place, we can also help with adoption, or we can help with architecture design, or in general, we can advise the teams. So now, let's take a moment to consider what's maybe a challenge in such a team. Think about it for a moment. OK, maybe something already popped up in your head. When I was thinking about it, quite a few things actually, a lot of things. And when I was thinking more about the topic, those things fell into three categories. It was timing, communication, and adoption. So let's see what exactly is here. When it comes to timing, I think it is important to consider the context of my team. So as I mentioned, right now we have 50 people on board. But back in January of 2021, it was more like 10 people. And in our company, the first machine learning initiatives date back to around 2016. And my team was created in April, let's say I think around this time of 2021. And as you might expect, there was a rapid growth in the department happening back then. So throughout the recent months, what we faced when it comes to timing. Basically, we were often too late with our associations because some of the teams already came up with their own ways of doing things. So it created this thing of being behind with what the teams do. Sometimes we are in time. But we were in talks with a specific team, and they said they will need a given solution in a few weeks. It's specific four weeks. So we proceeded to work on that. And we were almost ready, like one week to go. But suddenly, we learned that the priorities of this team, so our client changed. And they no longer need this solution. OK, but there's also the third scenario, which is too early. Sometimes we tended to anticipate the needs of our clients. So we were asking them, OK, so what will you work on in two months? And they say something, something, and we'll need such tools. Great for us. We have time. We can develop such a toolbox for them. But sometimes it ended up that those tools seemed artificial because they were just too early. There was nobody right there to take it and use it immediately. It may be a few weeks or even months before such a tool or standard will be used. So that's about timing. But let's consider the changes of adoption. And let's be specific here. Let's imagine a scenario that your team developed an internal library, internal tool so as to make the management of EC2 instances on AWS especially for the specific case of machine learning. OK, so you may think that this is something that may be useful for everybody in the department. But it turns out that's not always. People were not very eager to immediately adopt it. And here we need to consider a few questions. So first, were they aware of it? I think most of them were, at least to some extent. But still, there were people who were not. Maybe they did not attend meetings where we talked about the solution. So, yeah. Then they could not adopt it even if they needed it because they were not aware that it existed. But if yes, then we are on the right track. But the next question is, do they trust it? And here the situation may be, you know, it may be that they believe that they have more experience in the area and they think that maybe this solution that we propose will not work for them. Or maybe they have their own tool already in place. They don't want to switch. But if they trust it, again, very good. But another question is, do they have time? I believe it is crucial because if they have their own priorities, their own tasks at hand, especially when the time is very tight for them, they would rather invest in delivering their own tasks than learning about some tool that may benefit them in a few weeks. But maybe not. Who knows. But if they have time, great. So, is it the perfect scenario? Not always. Because sometimes they may not be involved enough to learn about the standards, to try them out in their own work. Or they may need help with getting started. However, they never reach out for such a help. And we may not be aware that there is a person who would like to check given solution. But they just need even like 15 minutes intro to that tool. And that's about adoption. Let's consider communication. I think we all know that communication is the key, whether you're a software engineer or in a completely different field. When it comes to changes that we face there, the obvious one is basically deficits in communication. And it may be from both sides. So we may not reach out to everybody, or we may not talk enough about the possible solutions. But on the other hand, our client may not let us know about the parity changes, all of the things that they suddenly need. The other thing is something that I found quite interesting. And this is misunderstanding the role of infrared team. And what comes here? People do not know why we need standards. What are possible benefits from that? And sometimes they may feel that we, as infrared team, are pushing some solutions on them. And then our team is being seen rather as an obstacle. And it all creates this thing of us versus them, which is not very healthy for the department. And another thing here is that sometimes there are people who would like to voice their opinions. So for example, we, as infrared team, would like to propose a standard. So we ask the community, hey, we have such a tool. Is it something that will be OK for you? Maybe you see some problems, possible problems with that. And people may not voice their opinions, for example, because they may not have time. Again, they may be focused on something different. And if they want to voice their opinion, yet they don't have this time, later it may turn out that they are not OK with the decisions made. OK, it all seems like there are a lot of different things that may go wrong. But I believe that instead of focusing on challenges, problems, it is better to focus on solutions. So I guess that right now you expect magical solutions. But I will not give you magical ones, rather possible solutions. So something that we tried. And I can tell you if it worked or maybe not. And again, here we will revisit the same categories as before, but I will add one more, which is the category of involvement. I believe it somehow emerges from the things that I already mentioned. So let's not wait. Let's jump right into it. When it comes to timing, we all know those scenarios. And I believe some of those can be solved with communication, which we also tried, obviously. But as a team, we realize that it is not enough. It is not enough, and we need to try something else. And what we decided to do is to embrace it. Embrace the fact that these situations used to happen, happen right now, and will happen. But we may take advantage out of those. First up, if there is a team that already developed their own solution for a given problem, maybe their own library, maybe another tool, they tried something they liked it, what we can do is take it and evaluate. Because maybe this solution is something that can be distributed to the other teams and may benefit also them. Obviously, we need to check if this is generic enough. Maybe we need to tweak something. But this is a great starting point. And it also feels more organic once you are using something that one of the teams already tested and something that worked for them. OK, but this is only one thing. The other thing that is also something that we need to remember about is that, OK, we may deliver something that will not be used for the next few weeks or even few months. But maybe somebody will come in those few months and check out the solution on their case. And maybe they will like it, then cool. But if not, it is OK to iterate on the solutions. If we get feedback after those few months that, hey, this does not work for us, can you change it? We can do it. We can iterate on the solutions, find some new tools that will work better. Or maybe in the market there is this new shining tool that is a lot better than something that already is a standard for some time. Then let's check it. Let's see, because maybe it makes sense to iterate and change the standard in our company. What comes to adoption? Again, here is a consider specific case. So at some point, our team was supposed to find a solution that will allow machine learning practitioners to easily train machine learning models. So we did research. We contacted with our clients. And it turned out that stage maker pipelines is something that seems OK as a tool to be used. And we also, as a team, developed example project structure that we proposed to the teams to be used for such machine learning training pipelines projects. OK, but what we could do? We could take that and throw at our clients. And our clients most likely wouldn't be happy about it. Maybe they would not adopt it. What we decided to do instead are four things. So first, we decided that we need to dive right into it. What it means? It may mean that we need to do a demo for our client so that they know what this tool is about and they can get started from that point. This may work for some cases. But in our case, it was not enough. We decided that we need to do something more. And that's when the pilot program came. What it means? Basically, we took one of our team members and said that, OK, you need to go now to our client and work with them for two weeks, help them with the adoption, be another extra pair of hands that can help them. And we did so. We performed the pilot. We helped the team to develop more like incorporate the solution quicker. They were quite happy with that. But what we got out of it is not only that some team adopted it, but we also get a lot of feedback in real time as the pilot program was going on. We learned about the things that they struggled with in this topic, but also in other topics. And we got a lot of insights in what their work is like, what are their projects. And it is also important to know what your client is up to. OK, this is one thing, but what else you could do? You could also infiltrate them. What it means? If you have a colleague of yours in another team, and sometimes maybe you chat over coffee, you may also talk a little bit about your projects, the standards that you are proposing. Maybe your colleague will tell you, OK, so these are our projects. And you may find some common points, and maybe your colleague will decide to try this solution that you proposed. And then if they try it and they like it, it may happen that they will promote this solution in their team. So there is no need to go there, rather it happens more naturally. Again, hear them out. So this is about gathering feedback and requirements, but what is important here is to act within finite time frame on the feedback that you got. You can also indoctrinate them. And whenever there is somebody new joining the company, let them first play around with the tools. Let them learn the standards. And when they start with it, they can then proceed to work on the projects already knowing of the standards in place. And now comes the new category, so involvement. So our case was that we quite early on in our team identified that we need to engage more with this internal community. So our natural thought was, let's connect with them and what we could do. Basically, when it comes to connecting, we established sync meetings. So every month we are meeting with each of the teams, learning about what they are up to, what they need. We are telling them about our projects. And we are seeing how we can help them in the future. This is OK, but not very engaging. Another idea was to have knowledge training meetings, which is amazing as an idea. But when it comes to actually having it, it takes a lot of time of the person that's presenting. So you may have your client tell you about their project. But they can do so and people around the department can hear about this project. But first, this person needs to spend few hours preparing for that. So people are not very willing to do so. OK, so connecting worked OK, but not very well. So instead, we decided to empower the people. And we had two ideas here. One was the guilds. So a guild is a group of people focused on a specific topic. So in our department, we decided to have MLOPs Guild and Data Science Guild. I am a part of MLOPs once, so I can tell you more about from this perspective. This group is not very big, only a few people. But those are people who either work in a relevant role, so they are MLOPs. Or they have experience in this area. Or they are just interested in the topic. And what is the result? When you have such an environment, it is best place to talk about those solutions. And I have seen this in real life. Whenever there's something posted in the Slack channel of this group, some idea to be considered, immediately you have responses from at least a couple of people, which is amazing. This is what we want. Another thing that we decided to try is to have our own internal open source solutions. So as a team, we started from this approach that, yeah, we should be the ones who are delivering the toolbox to another team. But as I mentioned, this team may not be willing to use it because, for example, they don't trust it. So yeah, at some point we decided this is not the way to go. Instead, we need to start the project, obviously. So maybe have some version 1.0 of a given internal tool, internal Python library. But what we do later is we give it to our community. And we tell them, OK, so this is something that we started, but now it is yours. You can add pull requests. You can report bugs and fix them if you want to. And you can add feature requests whenever it is needed. And it changed a lot when people learned that, hey, right now I can write a code to this library. And since I will be using it, I know exactly what I want. And in our case, it worked perfectly because people felt that right now they are in control of what they will be using. And they are very happy to contribute. They are very proud with what can add to this tool that will be later used by them, but also by their colleagues. And we have the topic of communication again. And when it comes to the communication, I believe that there are a lot of books on the topic. There are a lot of talks on the topic. So I will not mention that. I will only tell you about one thing that we tried, which we called for ourselves laundry time. What was the laundry time? So basically, we decided that some things are confusing for people in our department. So yeah, they're confusing. But the error is also not very clear. There are some things that maybe are not said well enough. So we wanted to have this hard questions and honest session. And how we did it? First, we asked everybody in the department, give us your toughest questions. Like, whenever you felt like you don't know why the things are the way they are, think about this and ask a question. It may be why we need standards? Why we have this strange infatim that forces us to do something or adopt a solution? Why our priorities are like this? All of those sorts of questions. And once we asked people to do so, they really did ask a lot of hard questions. I think there were over 30 of them being submitted and they were very diverse. And what happened next? Once we had our internal summit in our headquarters, it was the first time that the department met in real life, which was amazing by itself. But also, we had this Q&A session where those questions were answered by people higher in the management. And it was amazing because, first, you could feel the air become cleaner when you attended the session more and more. People were also following up, so they got some answer. But maybe they were not satisfied with it. So they were asking more questions. It was amazing in general. And I think people felt for the first time that they are being heard, that their confusions are addressed, and that people really care about what is happening in the department. So for us, it was only three weeks ago. So I can only tell you about short-term results. But I already see a lot of collaboration emerging from that meeting. And I believe more great things will happen. And this is also something that I believe our department will be having more of in the future. So maybe in a few months, we'll repeat the same thing. OK? So I think that right now, I owe you this slide of what worked best for us. And those are fixed by me. And the first one is the pilot programs. Why? Because we get a lot of insights, a lot of feedback from the team that we collaborated with. Another one, open source solution. Because this is something that empowers the people and gives them the control. And the laundry time, 10 out of 10, I can recommend. So I mentioned quite a few things today. Maybe you remember some of them. Maybe there's one thing that sticked with you that you would like to try in your own company. Maybe even if you are working with external clients, maybe some of those points can work for you. But in the end, I believe that working with clients in general is an intricate art. So it is only up to your creativity what you will try and what results you will get. Thank you very much. OK, so do we have questions? So do you want to use the mic for anyone who's got questions? Hey, thank you very much for the talk. I was wondering, how do you keep the direction of your internal open source software? Because the thing is, if everyone can contribute, at some point someone may contribute to something that might not benefit the other teams as well. How do you keep a general direction, let's say? OK, so I believe we are in this lucky position that we grew, but still there are limited number of people. And as teams, we often collaborate. And when there are tools that are used by 10 people, usually those people are in contact with each other. So they are also discussing what we want to add. And my team is also the one who is responsible for moderation of such efforts. So basically, we are happy to take any input, but we may also consider, OK, is it something that will benefit all of the teams, or maybe only one of those? If one of those, then maybe we can have a dedicated solution just for them and not introduce this change in a library that is used by wider group. OK, thank you. Thank you. Thank you very much for your talk. I'm definitely interested in this being part of a newly developing team at the moment, or a group of teams in my company. You mentioned the idea of guilds. I'm particularly interested in this. Is this something like having cross team teams, or is it like having specific topics that people will join into as they like? Can you explain that a bit more? OK, so the idea of guilds came from the backend domain, so we already tried guilds there. And we decided, OK, let's see how it's going to work for us. And right now in search guilds, for example, the MLOPs guilds, we have people from each of the machine learning teams in our company. And we strongly encourage that to have at least one representative from each of the teams, because every team has their own perspective on the topic. And this is very valuable. For us. So yeah, those skills are focused on a specific topic, or a specific area. For us, we have other areas right now, so data science and MLOPs. But if we want to, we can have guilds that is focused on a very specific tool, even one of our internal tools. OK, thank you very much. Thank you. Hi, thanks for the talk. I think it's quite common to have multiple product teams or ML teams in a company, and a single infra team dedicated to them. How would you manage the overwhelming, growing backlog of requests coming from those teams? OK, yeah. This is something that we also find quite challenging, because the teams work on quite diverse things, and they need tools at many different points in time. That's true. So actually, this is something that we did like a week ago for the upcoming quarter. So we gather all of the possible topics, and we ask the teams, OK, these are the possible areas that you told us you will need. So each of the teams, please rate how much impact on you will have this given solution. And when we have results from all of the teams, we can then have a specific metric that takes in consideration priority of each of the teams, but also in how many teams this priority is somewhere. Yeah, and based on that, we had a spreadsheet, and we sorted it, and the ones on the top are the ones that are being addressed in this quarter. The other ones will be addressed possibly in the future. Thank you. So thanks very much, Paulina. Thanks to all our speakers this morning, and hope you have a nice lunch. And the solution, as always, was a spreadsheet in the end. Yeah, thank you.