 So hello, hello everyone. Is it too loud? So hello everyone and welcome to this talk about Phenops and I hope in the next 40 minutes to shed some light on what Phenops is and what Phenops isn't Full disclosure. I'm an engineer. I'm not the management guy not coming from Phenops And finances so my previous in background is like developer DevOps I currently work as you will see in the company called global data net as an AWS solutions architect and this talk this paper is a Let's call it that Respective talk about the project we had for three years in a large enterprise on Phenops stuff as Engineers and it's been it's been fun. So The large enterprise that Andy and I had worked For is really large. They were spending seven plus million dollars per month on AWS and no wonder they were quite interested in in Doing some Phenops Andy my friend and colleague works for devo team. So my company global data net is an AWS focused company they do everything related to AWS ecosystem and Andy works for devo team their Consultancy company doing all kind of consultancies IT but AWS among other stuff so we started working for This enterprise and it was fun. We started From Phenops foundation. So every time there is something new introduced into IT usually someone comes by and offers Their view of the topic. So Phenops foundation offered their view of Phenops and If you can if you can see this this Well definition, it's really large What is important about this definition is two segments two aspects two things that will be important Throughout the engagement and throughout this paper. The first one is cloud financial management discipline So it is about that so doing Phenops is about managing finances in the cloud the second stuff is cultural practice and Then it comes all the buzzwords that you can that you can think of like enables Organization to get maximum business values and all that stuff You need to have that that goes alongside with the package but two important things like managing money and how do you change the culture in the company and Unfortunately, as you will see until the end of the talk we do only first we only talk about the second But we only do first and what I wanted to offer through this paper is a little bit of insight that Andy and I think Might be valuable if you would like to engage in the second thing. How do you change the culture? but in a way that is You know approachable to engineers and we should know because we are engineers So we were looking at things using our own Perspective now along the way Andy and I came up with our own definition of Phenops and here it is It's beautiful It's like Phenops is about ways to save money in the cloud and ways There is importance in the word ways Because as a DevOps engineer or as an engineer you would like to introduce ways to do stuff like Methodology not just commands not just things that that you do but ways of doing stuff So for us it was ways to save money in the cloud and then you get to know that everyone in that company were always Well before us saving money. So if you were deleting obsolete volumes, you were saving money So that means you were doing Phenops and if you were you know shutting down your instances over the weekends because you don't use them You were doing Phenops, but they were doing a lot of these stuff already What they wanted to do is something more So we came along and we started doing this We started investigating what was happening in the company So before the cloud and again large enterprise really applies to these use cases Before the cloud you would always follow waterfall. You would have everything predictable or at least You know it to some extent predictable, but you would know how much money you were ballpark going to spend You would estimate main days. You would estimate team members. You would estimate Calendar days you need to finish the project and how much money is this going to cost you if you are going to buy some some Machines or networking storage, whatever you would know how much that costs and Then you would be able to assess how much money will it bring back to you What is the return of this investment now when you go to the cloud all that breaks first of all There is no waterfall anymore. At least no one talks about it anymore. It's a different topic whether we use it But we don't talk waterfall. We talk agile. So everything can change. So when you go and do do the project Two weeks later you will get a different requirement that different requirement will require you to change the approach Which would mean that you would pick different resources in the cloud that cost differently and your whole financial construct breaks and That is difficult. It's difficult to lead and this is When we talk before the the talk this is difficult for engineers to understand because it's difficult to run a company Without having this knowledge we have without having this information. It's quite difficult and Every every organization you will meet meets this challenge. This is a dilemma You either and that's the way things operate in people's head. You either give Control to finances and they control the money flow or you give control to tech guys in your company and they control how they build the tech because you know remember the Challenge I need to do stuff the cloud way so cloud native pays you go everything changes If we want to be agile business loves agile business loves changing requirements every week So that's cool. But if we are going to follow that then the finance will break So we have a dilemma and that dilemma has always won always the same answer always the same It's finances Finances win we engineers we lose But you can't really control how engineers do their stuff So what do you do as a finance at least this large enterprise didn't a lot of others that I Talk to or have been in touch with during this time. They're doing the same stuff So everything we see right now is driven by the finance goals and This is because there is only one way finances can limit you So you can be predictable and that's capping your account capping your budget So what you would do if you and this particular enterprise head over? 100 teams working on 400 different accounts and different projects so large large thing So in within these if you would like to start a new project Let's say that you've got an idea and you approach finances and you say I've got this idea And they were managing data and and processing data and and generating all kinds of new information about what they already have And if you have an idea how to monetize something you would approach someone in the business and you would say hey I've got an idea. I think this would work. I've got already clients Interested in that and that's all cool and they would say okay come with some business plan obviously and then you would Come with some ballpark estimate a Couple of slides back waterfall you would come with some bulk up ballpark estimate And they would say cool Don't take it to literally we are agile But that's something we will work with and they would cap your budget with that and that's it You kept you've got your AWS account or multiple accounts. You've got your cap That's not that's the limit you you're not going above that because that's something they can calculate with and Then no one's care. No one cares about what happens after that As long as you stay below the cap and you know what we have found a lot of teams could go much lower But they don't because they don't have to they don't care they do the engineer engineering They do all the fancy stuff. They like to do and there is nothing nothing wrong with that It's just how things operate. So who sets up the rules? Finances who plays by the rules us So if my budget is really high and or poor high enough I can play with all kinds of new tech. I can implement stuff that I don't even need to implement I can left stuff behind. It's all cool. You know what what is worse than that if my budget was said too low I was ambitious I want to I want to show off myself in front of the company and then I say I can do it for less money And then I get into trouble because I underestimate The actual business value of the of the of the project and then I have to deal with that because then I have to you know Whatever I do, they will not change the budget. It stays like that because this kills Predictability that's wrong So again from the very beginning two things one Saving the money and that's cool your colleague said About tools that are generating reports. This is phoenix nowadays only the first stuff So you buy a tool you buy a product you form a team internally and what they will do is they will Present you with the data about what you spend where you spend it How much you know worst offenders top five accounts that are spending money and stuff like that But there is nothing about how you mitigate that, you know what they do They put a couple of these recommendation techniques like switch from Intel to AMD. That's cheaper No one asks you if this viable if does this breaks the Engineering that you have to put in so they don't care It's like I have the product for Phenops and I need I need to I know I need to have both things because that Looks good when I present it when I pitch it to to the management stuff So I say I I've got really great importing it does everything for you It will tell you where do you spend the money how much you spend it? What are the worst offenders? We also do have this recommendation engine You know, we tell engineers everything like when to stop the instances when to kill them went to downgrade and all that and that's all Unusable because while working with this enterprise They wanted to buy a couple of tools and I went very in great amount of details through some of them And it's always the same like switch from Intel to AMD or kill your obsolete snapshots or Don't use volumes that you don't use and stuff like that and it's ridiculous So no cultural change. They don't care. You're just like that then Andy and I started discussing and Moaning about that actually and we were talking about how what you can do to make it better And why is it the way it is and then you know all that stuff? And then at some point and he said this sentence and I put it on the slide purposefully like He thought like just as serverless architecture change the way we look at cloud and this space you go concept And all that stuff that you did you see there? We would need a new way at looking at things so we said Let's let's meet this guy this if you haven't read anything from this guy I strongly encourage you to do so. This is Edward stemming He's most famous about the book out of the crisis from 1986 He's got the book in which he reveals foreign principles on how to make your organization more human oriented more culturally engaged more Oriented towards quality. He's the guy that is behind all these six sigma and and similar Organizational stuff movement. He's got a great biography great stuff. So interesting reading just through his biography You don't have to read the book the biography itself is beautiful But one of the most interesting stuff for me out of these foreign principles is this one craftsmanship pride Now if you think about what motivates engineers to do what they do This is it This is the pride to do something if I'm proud about something. I've done then I'm good I feel good. I feel that I should do something more about that It's like the the cycle in the brain the brain gets happy about what I did and then he wants me to do more It's about pride It's about being proud on what you do and being open about what you didn't do and stuff like that You know being open being being happy and we thought like okay. Could we do something like that? Well, honestly, we didn't do it like that it later came back to us that this is the connection We didn't see at the time, but it's really cool to say it like that So we thought like okay, let's change things and here we have a great enterprise The project that is going on we have a great, you know Place to try out these ideas they already exhausted everything they wanted to exhaust so there was no harm to be done So we said like okay, let's try and look at this as a Distributed environment so the company is distributed stuff So there is no up or down or who owns who but who owns which? Responsibility like a distributed system and you would need a little bit of orchestration Maybe a little bit of choreography to make it work So we split we said okay, we've got finances and they need information about how much something will cost In order to be able to make plans We've got tech guys and they would like to know what they can do to Now watch not save money, but make things better Make things more, you know useful for someone else to do more quality work So how do you come up with that and we started playing with the idea how we can? Pull some data from the system to be able to answer some of these questions now before I forget There is one thing overarching all this more important than anything that that you see currently and this is Company has a strategy This strategy should be the most important imperative for everyone in the company all these Distributed elements in the company should follow the strategy So if the company puts the strategy to save money, that's the task for engineers That's the task for finances if the company puts the strategy to become more innovative on the market Then that's the job for engineers and also for finances and they all think different when you put it in that well Perspective let's call it like that. So if we if we are going to push Towards you know doing the company strategy, then we can do something So first assumption if the company doesn't have the strategy all this doesn't work So I'm just talking a good good talk, but if the company has a strategy then this works and This is why you Will do something like this you want to build on the existing system You want to have all these savings that you can do you know It's it's really cool stuff to delete unused volumes or obsolete snapshots or shut down your instances over the weekends and all that stuff That's pretty regular stuff, and you want to keep it. That's cool But you want to do something more you want to build on that on top of that So this is not the talk that says Phenops is not good or current Phenops is broken. No, it's not It's just not fulfilling on its promise. It's fulfilling part of the promise the other part is missing So we we wanted to build on top of that the second stuff We want to chase the objectivity and this is important objectivity for different parts of the system mean different stuff like for finance Objectivity is how close I am to the financial goal I set or how How certain I am that with a certain amount of investment I will become more innovative company So how can I how can I know that if I'm going to pitch this idea to the GM that we should put some money into Engineers doing something and on the engineering side. This is much more. Well for us nuanced like it means If you if you recommend me something, please don't let it be switch from Intel to AMD please make it a little bit more usable for me and this is what we started working on and we worked and For us it was like decentralizing Phenops We ended up having a model like this. We will stop for a minute here So this this is a model We we've built over time and don't take a close look into it. It's small and it's maybe not readable Really, there will be slides sharing on but this the important stuff is this we started collecting data from AWS So we would collect all the pricing we can get from AWS because you can get it It's really cool and you collect all all the pricing for all the resources You want to you want to oversee you want to control and then we collected all the all the configuration parameters for all resources we are observing like Easy to instances or EBS volumes or ECS clusters or kubernetes cluster, whatever you've got at least in AWS It's the same stuff in every cloud So we would we would pull all the configuration data and then on top of that we would put pull a lot of metric data We wanted to discover what are the trends what are the patterns of use usage of these Resources, so if your easy to instance goes up and down or does something You know different or strange then this is a not not a normal easy to instance something is going on with it But if you have a steady flow of being below 10% of utilization Then you could do something about that so you can learn a lot from that data But you need to spend time looking into it and analyzing it and fighting trends and then since this is again a Decentralized system you need to have a feedback if you don't have a feedback then it's worthless So every time we would come up with an idea We would go and talk with teams and these were the funny talks like we would go and talk with teams That we feel might switch to using spot instances so spot instances are really fun because if you have a cloud native application and If you have an application that is able to survive the outage of a resource Then you really cool to use spot instances because AWS will take them away from you eventually and they will give you two minutes Warning and if you can survive that then your application is really cool now again quality matters So I as engineer and really interested in in stepping into this spot instance stuff not for the money But for the pride You know the price is different So I want to make my application cloud native and this is what I personally like to do So it's not about other engineers. I'm in that position So we build this and you and I have we've built this stuff and it's serverless And it uses spot instances and all that because we thought it's cool And it's good and it's good good stuff and it saves the money So again cultural change now the moment I started being motivated to do something in a different way Is the moment that you initiate the cultural change? If you are pushing someone just to switch from Intel to AMD because the report told them so this will never happen You will generate these reports you will send them to and you know what they say we let's say I Go and talk with the team and I explain to them all the benefits of some approach Whatever you say and they would say to me. Okay, I Don't have the money for that. I don't have the time for that I've got a full backlist of backlog of features to deliver until certain date So I've got SLAs to keep I've got clients to make happy So yes all that that that you bring to me is cool But this is not gonna happen and you know what I know what is my budget cap and I know I'm below So I'm generating more money for this company than I spent so I'm cool I would say I'm cool So you go back and you you know talk with someone else about that because cool idea, but not for us and That was the moment that we realized that we need to introduce a different side into the story finances So we had a connection with tech guys. We can talk with them. We can propose stuff We can discover ways in which they can improve but something was missing finance so we think this is This is a Again, don't look at it. It will come to you when we talk so finance guys So what what we said, okay? We've got a company strategy and let's assume that the company strategy save money Okay, so we generate a list of recommendations Real recommendation based on the the usage patterns We see we talk with teams and we establish what is realistic and what is not we feed back these Let's call them Findings into the system and now we start avoiding these obsolete recommendations and we leave out only the ones that could be used and That are aligned with company policy Let's say that the company policies delete all the snapshots older than 90 days and then you can implement that that's cool It's get it gets trickier when you say Downgrade on your instance type when your utilization level is below 20% for the last 30 days You know, that's trickier stuff, but you can work around that you can come up with all technical Recommendations and solutions you would like and we had real success in it teams were Responding good on certain recommendation types so we started with having three types like red umber and and the green ones and then we ended only with red like Things that are easy to do right now and then once we are we are done with that We will we will improve it but finances that was the the missing key So we went back to finances and we said okay, we've got attraction here We've got some data teams would like to implement these changes, but something is missing You're not following the company strategy. So what is the strategy of this company? Is it saving money cool? You know what we did we generated a list of recommendations We graded them we gave gave them height and weight and then every recommendation would be graded by the By the quality that brings into the game is it saving money? Is it innovation stuff? Is it beating competition? Is it becoming more serverless? It can be anything you think of like these rules you you come up with the rules that follow the company strategy and then we build a grading system and then we put It in the recommendation list and then all the recommendations that are in line with your grading system comes on top So every team would get first on top What is the most important for you and the least important stuff are below and then what you do? You just follow the agile you feed that into the backlog stuff you give it an estimate You you now know how much time you will need to finish those to implement these stuff And then you go back to finances and you say okay This is the company strategy you wanted us to do it like that you wanted you wanted us to save the money This is what we did now We've got tasks ready to be picked up from the backlog But you need to think about how we're gonna do this because you need to allow some time to do it Some money to do it. Maybe some resources to do it for every task has these Lined up, you know for every task, you know, how many days you need to finish it What kind of disturbances it will it could introduce into the system? Is it SLAs is it that you're worried that your system will not be working for some time? How how much you need to test it before you make it real? But you know you know stuff about that and that's in line with strategy and then finances say okay We will postpone several features or the company will say and we will make space for this especially in this case, it was about gaining competitive advantage and we redesigned this system to be really really good about gaining competitive advantage about What it means to be competitive? What it means to be innovative for this company? It was going service cloud native as much as possible because they were large and they wanted to be able to ingest large number of customers in a small amount of time and it was about being cloud native and we started feeding these Info into this the engine we call it cloud fitness framework because it makes you fit in the cloud So cloud fitness framework would do this we would tell it fine Please find all the all the accounts all the projects in which you you've got a lot of easy to instances That cannot be good, you know You've got if you've got if you want to be cloud native and you've got a lot of easy to instances Something is wrong something might be wrong So it wasn't really about at the end of the day if I have to say it wasn't really about us Resolving anything it was about us cleaning up the fog the mist above the enterprise so you can see where you can improve and that was really helpful because this guidance Towards the you know Change was important for the company and then engineers could feel And we are yet to see this in in action for a longer period of time. They just start that so this is really fresh They just started using it but It should allow Engineers to have proper amount of time to sort out something properly because no one is messing up with how they do estimates Again, it's a part of agile process now I'm putting that into the backlog. I am assigning days on to it. I am Assigning complexity into it. Whatever I want to do. I assign it the finance guys and girls have the Responsibility of making this happen from the financial side of things So someone from the company if we are following the innovation path here, obviously Someone in the company needs to make this happen from the management side of things from the operational side of things Because if you don't have that if there is no support from the finance side of things, you can't make these things happen You will forever stay in this save money mode, which is fine, which is completely fine I would just like to go back to the first slide and and eliminate the whole Everything from that definition and stay with the end is in my definition of of phenox And this is ways to save money in the cloud, which is really cool if this is what you were after You know if you want to save the money in the cloud that that's that's something you should do Well, that's something you can do pretty pretty good right now without any fuss But if you want to be competitive and this is why I feel this this fits mostly with with enterprises because it things get much More complicated with enterprises. It's much in my opinion simpler to manage smaller organization because then you Usually have smaller number of people that are making decisions in this case It was like the huge pyramid of people that are involved in making any anything happened And it was it was a challenge and again if I if I if I can think back. I think the the Enabling companies to see what is happening and how they can improve was the real the real deal here Like the report that will give you ten Worst offenders in terms of how much money do you spend this will give you ten worst offenders in terms of How far away from serverless are you and this will tell you that there are projects that are not using any managed services You would be amazed so classic classic monolith running in the cloud using easy to and using MongoDB on top of easy to your own MongoDB and you you're pushing it like that and it brings money it We are not talking about bad engineering here. That's also important for me to say these are fine Engineering practices in play. This is just not serverless. This is just not played cloud native. That's a different topic But it's not about bad stuff happening in the cloud It's the stuff that happens that are not aligned with the company policy So there is nothing wrong with running MongoDB or SQL server on top of your easy to instance That's perfectly fine. It's even cheaper than if you would buy well cheaper initially Then if you would buy a managed SQL server Service it's a different topic. How much does it cost you to run it for five years and to make all these backup stuff And make sure that it's available all the time and all that stuff everything comes with a price So that that's that's a different topic. I'm not talking about that. I'm talking about how Different parts of your engineering stuff are aligned with your company strategy and how that hurts the company that was the the aim here and I I needed to thank to mr. Edwards The last time because this is really great book and again if you go to Vicky and if you just read through what that man did In his life, it's amazing. He was involved. He's an engineer. He's a mathematician. He's a statistician He's a scientist. He was engaged in breaking the Submarine problem in the world war two Engaged in terms of how Americans build ships to break through the barrier in the Atlantic He was engaged in reorganizing the way Toyota works in after the Second World War in Japan He did a lot of stuff, but most of the stuff was about how do you achieve quality in an organization? How do you motivate people to do better stuff? How do you motivate people to be part of the system not part for themselves? but willing to do something good for the benefit of the company and that was That was my talk for today. So I Understand that there's there's been a lot a lot of talk and that's so little of technical stuff There is something running behind this in AWS being serverless and all that stuff and maybe on some some next Conference that you will organize here. Unfortunately, not in January, but sometime I Will maybe write a paper that will be Fully technical about how we did what we did, but it was important for me to to tell the story This is a story how we went as engineers through these phenops For three years three years and it it's been fun now questions if you have any using that ridiculous redhead Mike So Lord have you tried to do some price estimates for the guide guidelines that you created, right? So if you have like one thing it's gonna Reduce the price by half this thing is expensive to implement But we'll have a huge implication on the price things like that Yes, absolutely as part of the process as I said we would gather AWS prices and we would gather configuration data data from the Resources and data trend usage trends as well But collecting AWS prices is important because if you feel that you could be let's say switch Dynamo DB from on demand to provision mode You would have to know the usage pattern of the Dynamo DB you have to know a lot about it So you pick 15 months. That's what cloud watch will give you So you would you would pull 15 months of data and you would go through the data and try to analyze Is it consistent usage at some level then you would go provisioned if it is Isn't consistent like in development environments or staging environments if it goes up and down up and down Then on demand is better even though on demand initially is Price here but being used like that. It's much more, you know, serverless So we would pull the prices for both cases We were first generating the price for a current use case then we would play with the data We would come up with a recommendation as we see like switch Please switch your Dynamo DB to on demand. This is the red stuff because it's easy You just go in and you change the the option the tool never changed anything because a lot of the teams did Infrastructure is code so you wouldn't like to mess up with people's code. You would give give back recommendations They would open up the Terraform or something else change the option apply it and voila They would have a Dynamo DB we don't demand and then month later or two months later You would see the effects and this is how we learned through the system You find a guinea pig you try it out with them You see the effects and then you say, okay, this works. Let's make it You know the the full blown rule for everyone and then sometimes you see it doesn't work So you say okay like with Dynamo DB is especially interesting because sometimes you feel that switching to provision mode will save money But then something else would hamper you like There is a case in Dynamo DB where you you're really consistent But then sometime during the month you have a high peak and your provision even though you would put auto scaling in Cannot code with the spike and you break the system so you cannot do that the system needs to work So in these cases you need to stay on demand because on demand will give you this Scalability and then this is how you learn you put that back into the system and now from now on if we see these Spikes we don't recommend switching to on demand. Maybe we could recommend something else later on But at this point no so yes, we would pick up the prices We would generate two types of prices one for the current use case and the other one for something We feel is good to recommend and then we would be able to generate the savings And this is what finances like because then you can present all the teams running in and you can say okay This team if they would do something they could be saving this amount of money and this team not so much and You know then you pick your your favorites Then you don't bother with managing teams that that can save small amount of money and then Obviously the the the price of of the change comes into play It's a lot easier to switch the item or DB from one billing mode to another then push some someone to use spot instances That that's horrible. That's really horrible or push someone from using I don't know lambdas to containers You know lambdas are great until certain point and in this enterprise They were starting using serverless lambdas are great and then as the number of customers grew Exponentially the bill on AWS was even larger than that. It was horrible And it was like the the ultimate time to stop everything and to switch everything into containers And then the price came down, but then the amount of lambda calls that they made was so Unbelievably high that it was you know the price was was horrible So these kind of stuff you can you can see that and you can you you know You can pick your places for improvement. That was important. So but sorry the answer is yes We picked prices. We calculated the the savings We can make what kind of recommendations we can give to finance guys and this was important for them for that Everything that we did also was only CSVs. We didn't want to deal with any Fancy graphs or stuff like that finance guys already had those teams So they were able to print all these pretty graphs and pies and charts and all that stuff We would just give data and that was cool for us because you know as an engineer You like CSVs or something like that because I don't I don't care how that looks like That was Yes If you want to be innovative again with in line with company strategy if your strategy is to save money It's not really about architecture You know, it's about operations You're in a in a mode that runs and that that that gathers money and that's what all all you care about Architecture is important the moment you start thinking about being competitive being innovative. That's your that's your game if you're just in it for you know running the stuff and and being Moderately interested in making it into a better stuff then you don't care about architecture You just care that you've got a lot of engineers in line that everything works that operations are happy and you know You run the stuff you you run the the business that that's That's what it is and you as an engineer need We all need to find our peace with that because you know, it's our job If the company strategies make money, then we do that save money. We do that. That's our job part of our education at least mine was to Come up with optimal solutions as an engineer an optimal solution means that it needs to be Financially viable not something that is fancier that the greatest the latest and all this stuff But something that the late's good with company policy with the money The company is willing to invest the the amount of people we've got at disposal The the tech level of people that are able to pull the stuff off. So all that comes into play So that's our job us as engineers So if the company is not willing to innovate or you know Then you align your own principles with that and then you say I I'm okay with that or I'm not if I'm not I'm looking for another job if I am then I'm staying here and giving my best to fulfill the you know the job to To do the job Yeah Yeah Hi, thanks for the presentation. I had a question like so who's like as I understand you did some Consulting for external company right like my organization also has this Thing where we already are cloud native innovative and everything and the cost is just because it's AWS is too high So whose responsibility should it be to? To manage this cost because like I was looking at the teams are trying to over provision because they are they are wary of stuff They don't know stuff finance doesn't know how it works at all So should it be like a FinOps team now in every enterprise or what's your take on this? Thank you I have my own opinion on that. I think it's a matter of trust. I don't think it's anyone Responsibility to be responsible for the the overall situation in everyone pulls their own stuff in so us Talking about distributing the the the organization is exactly that Become accountable for what you can become accountable for so it doesn't make any sense for engineers to become Accountable for money-spending with accountable for you know tech Implementation if that tech implementation can be cheaper then we can be accountable So we are pushing something that is price here just because we can or just because we don't care Then it doesn't it doesn't make any sense But having these any organization actually that has these two silos will have this conflict The moment you split them up at least in this case So I'm talking from a use case perspective So the moment you split finances and tech guys and the moment you introduce someone as a you know Body in between or communication body. Yes, that's phenops. That was us in this case So we were able to pull the data about the usage and then we were able to say these Engineers might do better because this is not efficient enough. You could do better About something regardless of what there is let's say and we would be faced with the engineers and we would have meetings with them We call them SWOT engagements like SWOT teams, you know specialty So we would go and meet with them and we would say to these teams Hey, this is what we found about your project. So you could be better You could change this into this this and into this and most of the time these teams would get back to us saying we would We don't have the time we don't have the money We have the pressure of releasing a new Release and stuff like that. So it's look it's either option one or option two So you pick but it's not like that then you go and talk with finances and you say, okay We've talked with these guys. These are the outcomes. This is what we can give you We can show you and then if you know what they need to do to make something happen Then you can estimate the cost of the change and that's important for finances You say if they are going to become better Then they will need two months time to implement all these changes What this will bring to the table is this this and that or Everything can stay as it is but you need to find your piece with the current price tag. It's just like that. Okay It's off. So that you having someone like a mediator between these two parties helps someone who understands tech and business and this is in in this case, this was the nature of Phenops having engineers in disguise in Phenops role so being able to understand to explain to everyone. What is their role in the process? Otherwise, it doesn't make sense Yeah, you need Phenops Okay, the time is up if you want to ask me some other questions you can do it later on. Thank you