 Thanks for coming. I appreciate it. I know we're standing in between what you and launch so we'll keep this compelling and interesting Hopefully my name is Jim Hasselmeyer. I'm with HP and with HP's cloud business unit my cohort and crime is Rajiv Pandey Who's also from the HP cloud business unit? We are going to be very deliberate about kind of the organizations that we're in because We're going to be talking about a variety of internal HP organizations But we're changing the relationship or we're going to be migrating basically between a provider Part of HP like an IT provider versus a line of business or consumer people that are using our Technology and using the open-stack technology to deliver value to HP customers HP and customers We all have multiple day jobs One of the day jobs that each that Rajiv and I have has been working with internal customers who have had workloads working in Amazon web services and There have been Activities going on to take those workloads that have been delivering value to HP customers out of Amazon and into our HP public cloud environment, which is an open-stack environment So that's what we're going to talk about today is sort of our experiences and And learnings that we've had as a result of doing that So these are sort of the topics that we're going to go through today We'll set some context around kind of HP cloud and open-stack We'll set the context or to give you a quick tour of what these migrated projects actually do the value that they're delivering Talk a little bit about the factors that impact the migration whether what makes them challenging or what makes them easy What are specific learnings are and we're and then we're going to hear from a customer and we put that in quotes because it's An HP customer Evan Chesley is here. He's from the HP printing organization and he was responsible. He and his Considerably large team were responsible for a lot of what you're going to see the results of so he'll give some perspectives as well And then we'll we'll finish up with some conclusions and such So I'll turn it over to Rajiv Thanks, Jim So if we look at cloud computing timeline just to set some context and there's a lot of history that of course is elated from this slide but Currently, I guess working for a company that was formed in 1939 and having been there nearly 20 years. I like to take historical perspectives currently the Analysis indicates that cloud computing the term was was originated by compacts George Favaloro and or Sean Sullivan. They're still debating which one of them or both of them came up with it What's less disputed is the fact that Ben black and Chris Pinkham at Amazon wrote this paper on standardizing Amazon infrastructure and In passing they mentioned the possibility of selling virtual servers as a service And that led to in 2006 Amazon web services being offered a few other developments in 2011 HP joined OpenStack and the reason I wanted to highlight those two Items and in fact there's a little bit more about the OpenStack story in HP I'll come to in a moment is that there was this gap in time when HP did not have a public cloud offering and yet Public clouds existed and there were businesses in HP that were interested in this technology and A lot of the examples we will give you in a moment happened to do with the printing business There's a lot of good reasons for that One is that the printing business at HP is still a tens of billions of dollar sort of business Shipping tens of millions of printers in a year and so it's not you know a small business But it's also you know ink and paper or laser toner and paper so it's it's media and content and content services lend themselves to this because there's content ingestion content delivery and content services are also something that you know you drive consumption By providing content services some of these content services may succeed some of them won't so you're bringing up infrastructure and tearing down infrastructure depending on the whims and vagaries of the service that you tried or expanded or Interacted and so there was a lot of attraction to that Also other reasons are just things like you know shadow IT We had a great keynote comment about it's like dental rins that exposes where you're not brushing From the Disney gentlemen. I thought that was great. I'm still pondering that but there was this gap and so that's how we ended up with HP having services at a different public cloud vendor and then HP of course joined OpenStack in 2011 and started contributing as well as consuming and the contributions are Manyfold hopefully you've swung by our our booth and seen you know our four ptls and the fact we're we're contributing to a lot of the projects and In about 2012 when we had the two data centers and we had launched a sufficient set of products there was considerable interest in the migration and Again, there were some very good reasons for this HP is very into open source that we have a long history in that space The fact that there was this community now of cloud Computing folks inside the company that were you know pushing this now I don't want to do disservice to you know utility computing and HP's done a lot of research in that gap I showed you but there's this product made a lot of sense from that point of view and then also the fact that You know Fred P. Brooks once said plan to throw one away so the the migration also gave folks a chance to rearchitect and and so they were able to take the lessons learned and and and move them forward and Hopefully you've been seeing the current you know HP Helion launch last week And there's plenty of talks and and information available on that. So I invite you to look into that what I'm going to do next is give you a little bit of of Flavor of some of the applications that we migrated and tell you a little bit about them and that will set us up for Describing some of our experiences. So again the the notion of content being delivered Is part of the system so HP wall art has to do with wallpaper and the wallpaper, you know There's ink involved. There's these latex inks that are green and they don't out gas and you can You know consume them immediately. You don't have to wait for chemical processes to settle down We have these large format printers that can print in two and four and whatever Foot width swaths and then they have adhesive backing that's water soluble So you're not using any toxic chemicals to put this stuff up or be able to take it down easily So that all combines into we were able to go off and license content from folks as diverse as NASA and National Geographic and Nickelodeon and then we created a web application that you would take You would measure the dimensions of the room you would figure out You know where the light switch was and where the windows were and where the doors were and whatnot And you could come up with your own creation or Spongebob's leaning against the door or whatever, right? And then this could be reskinned by the mom-and-pop shops that had bought this large format printer and were trying to drive consumption and so content ingestion from all these content providers content delivery to all these fulfillment points and so this is one example of a web service that was or a Workload that was migrated from Amazon Another has to do with image-based fraud detection. So HP makes as I pointed out makes a lot of money in selling Consumables and these consumables, of course, there's a refill and remand sort of refilling and remanufacturing of cartridges Which is a fine thing to do You know if you're into that but it also might void your warranty But you know, that's a buyer beware sort of thing What's not so cool is when those refilled and remand cartridges appear inside of boxes that look suspiciously like their HP and they have that burnt markup and so there's There's a system that we've put in place in some of the regions where there's a lot of fraud going on where these Refill remand cartridges find themselves in these these boxes that are Counterfeit HP and it has to do with steganography So in coatings inside of images and stuff so you have a mobile app You can take a picture of the cartridge and decide whether it's really HP or not or Investigators can do this as well. This is a very simple application If you well my good buddy and colleague at HP labs with the shell sim ski would beat me up for saying It's simple because the steganography in the back end of that is not so simple But from a from a implementation point of view, you know You've got this application that captures images and sends it up to the cloud and computes some variations and figures out Whether it's real or not and so, you know simple database images But again, we want to show you a sort of a spectrum of of workloads and we'll talk about that in a moment as well Jim's going to tell us about some of the other workloads. We migrated So a non printing app That was migrate migrated is focused on helping people learn Windows 8 environment for Windows users out there Especially you've gone through the migration to Windows 8. I have not so I haven't experienced this There's some learning curve associated with that from what I understand And we HP included in our Windows 8 System an application a tile an app that helps people along that learning curve And one of the things that's kind of cool about it is that the contents of the tile Actually rotates and so the user sees different content available you can learn about File system where you can learn about a new app or whatever and so this app ships on every single Windows 8 PC that HP HP ships and the app goes back to HP and gets that tile update Comes back to our cloud to get that tile update and keep that content fresh Let's talk about the complicated one and and then we're going to get right into Some of the specific experiences that we've had Rajiv talked about the scope of printing and the importance of printing to HP and And one dimension and one might be saying gee I got to print something I go print it. What's the big deal? the printing business is really driving some massive transformation in in how they do printing and how people look at printing and Suffice it to say that the environment that we talk about for printing in the HP connected environment allows people to Pick content that they want to have and when they want to have it so that it can get stuff gets delivered to them automatically at their Desire and when they want that to happen and there's just a whole host of library of stuff that people can pick from in order to have You know to have fresh new interesting things that are helping them in enriching their lives could be coupons It could be all sorts of different things The structure for how this is all put together and kind of using the big block kind of thing is that The main thing that we're going to talk about today is the subscription management box in the middle that is the system that has the Content vault the subscriptions of what people actually want to have delivered when and It makes the decisions as to what needs to be delivered when and then hands it off to what we call the webprint platform to actually get it delivered There are also Solutions upstream that are generating that content because it's fairly complicated content And so a main point I want to make here. It's just gonna work Doesn't work on the screen the two boxes on the left represent 12 roughly solutions and those solutions on their own each Individually have multiple stacks dev stacks test stacks production stacks and so those two boxes on the left represent a very considerable amount of consumption within the cloud environment This is the dashboard that they use to monitor their actual delivery Suffice it to say what I want to get out two things I want to point out on this is one is we're talking millions of print jobs a week and then the other thing We're talking about is that notice the each little bar is an hour Notice the cyclical nature of it. It's lends itself to the burstiness nature of a cloud type of environment So let's get into what we've learned Rajeev and I got asked a lot how hard is it to migrate and we Decided that there's a very solid answer of it depends And it depends on basically three different things how complex is your solution How tightly integrated was that particular solution to Amazon technologies themselves and What are various aspects of the team that you've got that are doing the the migration on the team dimension? It could be we've got a big team and they're really cloud knowledgeable And therefore that might positively impact the speed with which that migration could be done on the other side of the coin It might be a small team. It might be a team. That's even though they're running in Amazon They may be relatively new to some of the new technologies You know, there are just a variety of things that impact that One of the things we're not going to get into I meant to say this earlier And I forgot one of these we're not going to get into is the details of let's say API differences between the two solutions One of the things we found I thought it was kind of interesting One of the things we found is we work with team after team people who are familiar with the API sets in Amazon They'd look at the open stack API sets and they go we get it. We see that they're different. It's just some different You know, it's not that big of an evolution shift We got to make some changes, but you know, we understand what's going on And the things that we're highlighting here are sort of some I'll call them some higher-level learnings There's some other a ha's that we got it We didn't expect as we help the teams kind of go through these migrations If I expand a little bit on that AWS integration access We tried to come up with some analogies as to how to describe this and and we thought this was a pretty good way to try to to drive this point home Conceptually when you look at the offering at Amazon You can either if you need to have some sort of service a database service or a load balancer Whatever the case may be you could build it up out of components yourself You could create a VM you could put HA proxy on it You could do all sorts of things like that or you could use their load balancer service or their database service and depending on your perspective Those pre-packaged services, you know, what's on the right might be what be would be the most attractive to you It gets you up and running quickly and you know all of those sorts of things but you get what you get and If you have a specialized need that may not be the best solution for you So when we talk about AWS integration in that one dimension here To the extent that there are services being consumed that were pre-packaged and bundled that created Sometimes a little bit more work and getting it into the open-stack environment We had solutions that definitely fell into that left-hand side of the slide where people had sort of built up some things They'd use some fairly generic things out of Amazon and then getting those over into the open-stack environment was I don't want to say it was easy, but it was not it wasn't a huge issue Basically, they knew what needed to be done. They just needed to do the work another area that has positively impacted a number of these Migrations is is the user model that not only at a foundational level exists within the open-stack environment, but also The way it's implemented in the HP cloud environment happens to be implemented in the HP cloud environment itself If you I talked about multiple solutions in this complicated printing solution environment Generally speaking each solution has two projects or two tenants associated with it a dev Development tenant and a production tenant and then from a user perspective there are developers testers and ops people and One of the things that's worked really well in the new environment, which I think I've been can go into Either on stage or afterwards if you have questions is the ability to have a user model That allows fine-grained control over who can talk to what tenant Their user popular their total population over here on the left is on the order of 70 to 80 people working across Low tens 20s types of tenants they have no shared logins They have an understanding if something happens in a particular tenant they know who did it and so this level of control Was an advantage and as part of the target migration environment they're going to that they that they very much liked the other interesting or another interesting thing that I think we experienced as we worked with with different organizations is Realizing where Lines of business were in their adoption of cloud concepts Technically this slide goes a bit beyond the scope of what we set up to talk here because People that we talked to or that we're talking about Primarily were at least here they weren't all here But they were at least here because they were running in an Amazon environment But another one of our day jobs was not only working with businesses HP businesses that were coming out of Amazon There were some that were coming from servers under their desk But in both cases Some in this case and some in this case It would take multiple conversations with them to realize that they weren't yet thinking about or addressing The architecture migration of going to an architecture that could really stand up or withstand a simian army getting getting let loose on it and To a certain degree I think they thought they were adopting cloud Methodologies by being in a cloud environment, but when we'd look at the architecture We'd have conversations with them if we realized that they hadn't really internalized that yet And so I guess I bring this out or we're bringing this out in this session today that you know Depending on where you are if you're a provider working with a line of business or you are a line of business Those who work with cloud day in and day out We get this and so that's how we think that's our perspective and It's very easy to assume the people that we're talking to have the same perspective And it took a little while sometimes to figure out that they didn't and and that was part of they migrate the logical Migration process as well. I guess this is the point. We're trying to make Okay the other benefit of being a Public cloud hybrid cloud private cloud provider and Having customers is the ability to look over their shoulder both at what they would do to fill gaps or Make the tools work for them and of course from that you can derive inputs into the various projects And that's the beauty of open source and open stack as well is the community process You know all the design summits going on today is you can take your learnings take your inputs and Make them actionable in those contexts So one of the things that the HP connected team found was with the amount of resources They were consuming across data centers across Availability zones across regions was that the console view afforded them by the tool Didn't give them that at a glance micro macro view that they needed And so the other beauty of this is they built their own and so this is an example of a tool that they built that let them at a glance discover more information about across AZs across tenants across data centers and Get an understanding of what their inventory looked like when you have hundreds and hundreds of virtual machine instances Running this is an interesting question if you've got a handful You have a handle on it when it gets larger than that you need to be able to do that micro macro sort of analysis This is another example of what you're consuming and again when your projects get so large you want to know what thresholds you're Getting close to whether it's on your memory limits or IP Addresses or images or whatnot and be able to see how you're doing vis-a-vis those thresholds those limits that you've set So another example of a tool that wasn't present in the console and again this is some work that Goes back to last year and so the good news is the if you look at the new horizon console There's more and more of these concepts that are being implemented right there out of the box And so but it was very insightful to see some of this occurring Another was the relentless automation. We talked about the architectural migration that occurred along with the the migration of the workload and The automation that was afforded by heat was something that was incredibly attractive to the HP connected team And so they embraced it they embraced it early and they became a source of a lot of feedback so this is a 71 line heat template they're actually in cloud formation format and then this is a tool It's actually an automatically generated visualization gentleman named Ben Ben Butler Cole Produced this so it uses dot and graph vis if for those of you in that space And so you can see that this takes if you haven't been to one of the heat talks there was a couple very good ones here and so you'll have to look at the recordings But hopefully you all managed to catch one and so the parameters the three parameters result in this one resource with a few outputs And so you can see the what this template produces by the time the HP connected team was done This you can see the the graph the image the visualization is at the very top of that And then I blew up a little bit of it so you could see What scale they were pushing it at and so this was something where again in the migration Having someone like Monty Taylor and his team available to take your calls. It was it was a useful thing To give you a better idea the number of objects here, you know It included a database as a service several load balancer as many virtual machines over 140 block storage volumes and all sorts of other virtual objects all Driven by these heat scripts and so it really proved the viability of of this Automation approach and it was very impressive to watch this these stacks just come up and so this was a Very interesting part of the project and and very fulfilling and then the leverage value of this Once you do this the the ability to to roll that out to other projects is really valuable as well now let's bring our customer up Customer in air quotes Evan Chesley from the printing group to share with us some of his Experiences with HP connected from that customer viewpoint. Thank you Evan. Thanks for Jeev So as you know these two great, you know shepherds of onboarding projects into open stack and HP cloud have Been reviewing some of the different stories. We thought you know my particular Workload would be a good kind of deeper case study. So I'm just going to talk for a few minutes about What our project went through kind of its scope what we learned as we moved from Amazon to HP cloud Again, I'm in the printing business. I'm not part of the HP cloud army that showed up here this week I'm here as a customer I support a publishing solutions business in HP printing and for all extents and purposes We just want to live in user land Yesterday during Bryce's Jonathan's keynote. He talked about the super users and I like adamantly insist I'm not a super user of open stack I want to know enough about open stack to be a really effective and agile business user But I don't want to like operate an open stack cloud I'm going to talk a lot about open stack technology But I want to use it as aggressively as I can in user land and just run a business as fast and as Agile as I can So in the publishing world, you know, this is a quick look at one of our production stacks This is the subscription license content Web application that we talked about that's used for a lot of web connected consumer printers Basically 450 objects that have to be wired up and running in flight for production And this is just one stack out of up to 20 that we have in flight at any one time from staging and load testing And you just dev reference type stacks In Amazon, we weren't using cloud computing in the truly rich deep API driven Automated way that cloud computing can be compelling But we still had to bring up hundreds you know about 800 different Virtual machines and you know thousands of resources as we migrated so well We had to kind of ask ourselves do we want to do this the hard way or do we want to change the model? as a customer we Basically, you know took grasp of the opportunity to use open stack in depth Still as a customer though, so we partnered with HP cloud to understand how to use open stack as best we can But we didn't ask for any favors or any special private APIs or anything. We just Went deep using what was available. You know made public by the public cloud Our model then was to start at the bottom layer. Well, we need to provision infrastructure We want everything to be automated. This is this model called infrastructure as code So we could have done heavy orchestration tools, but we're talking open stack here We want to really play deep with open stack. So back in the grizzly time frame. We went deep with heat It was still pretty rudimentary, but we just placed a big bet on heat. We love the model of one json Manifest which can deploy every object on that page in one shot one API called my heat server Which you know as the user I you know heat was pretty new I had to build up my own heat server But a heat server in user land worked completely well on top of a real public open stack deployment on HP cloud So we use heat infrastructures code it builds up all those resources now We need to install middleware application to all the configuration my particular project We use puppet in great depth to do that Once puppets finished running across from my virtual machines I've got a running business application and it was one self-service call at the very bottom You know at the orchestration level to make all that happen This allowed us to deploy a lot of resources on HP cloud in Only three or four weeks where before in the old model of using the cloud in a less agile way it took it Many many months to bring up the same number of stacks Where we are today. This is roughly a snapshot of my one publishing business Kind of footprint on open stack Something of around 16,000 open stack, you know objects or resources they're organized in stacks You know that's the beautiful beauty of heat. It's very stack oriented. It's our primary object of you know operational reference But if you look at it and that was Rajiv's a snapshot of the visualization of a heat template You know we have this many objects in flight, you know many virtual machines many more supporting resources including neutron in-depth and A very small team can support it so again as the customer I don't want to have to learn open stack and really have to hire Open stack wizards to run my business I'm using the tools to abstract that and a very small more business devops oriented application operations team Can run, you know this many resources on top of the cloud and remain very agile iterate stacks rollcode as fast as we want We feel it's like it's been very successful again the the model which we Love adopting a top of open stack and I urge any migration You know if you're looking at moving something from one cloud to the other It's the perfect opportunity to adopt much deeper You know business application as code model end-to-end orchestration you want to Leverage the chance because once you wire in your you know the chance gets a little harder to capture and That's our story Thank You sir Before I go to conclusions. I'm gonna tell a story that Evans may be sick of hearing me tell But um, you know in this world, it's we all deal with technology and Details and does it work or does it not work or whatever and we don't get into the emotion of it too much And there was one time it's probably one of the most memorable moments on this project We had to get Evan and I had to get on a call. We got a lot of calls But there was one call in particular where we were doing the heat engineer on our side Had been working with you to get a bug worked out To be able to stand up these stacks and when we got on the phone to ostensibly cover some other topic This engineer had just provided a patch to Evan, but it hadn't been tried So while we're on the phone you launch the script and it was quiet on the phone And then you go there it goes and he goes and both the engineer and you were going this is so cool Because all this stuff just started to get created it got wired up and it was it was a really well and go back to slides I mean that moment was going from this is a white blank piece of paper to Heat in six minutes bringing up, you know everything on there And that moment was something we'd been betting on and I think only open stack You know in great depth allowed us to get there and the heat part of it with you know Great engineers helping move heat forward in HP. It was it was an exciting day. It was neat. It was very neat So the takeaways Is Rajiv and I pondered what we wanted to convey in terms of the takeaways from these projects that we've had I think it boils down to these four things Automate automate automate. I mean we've heard Evan say and he's told us before Cattle not pets treat your resources like cattle not like pets. It was interesting to hear the cattle not pets phrase used this morning I took that as the the pets are at the solution level not at the infrastructure level But when you're going to be doing large scale deployments You need that automation to ensure that you're you're deploying and creating things consistently When you're looking at true enterprise and enterprise scale types of deployments You need to have and we need to be sure that they has that has the tooling in place To enable that to happen to provide the higher level views the summary level views like what Rajiv talked about With the tools that have been created We found ourselves on a number of phone calls where people were in essence looking for the AWS solution in the open-stack environment And we found ourselves saying it's not Amazon. It's a different environment We got to change you know change the thinking a little bit So we just have to kind of if you're helping people go through this process I guess I would just say it's a process of helping them understand that the mindset's a little bit different sometimes and It's not the same And then lastly be aware of that solution migration versus architecture migration, which I talked about We very much found ourselves in the mode where we thought people were either either had or were going through an architecture Migration and they really hadn't gotten to that point yet and then that caused a different type of conversation In order to effectively move that migration forward So with that we have a few minutes left if you've got any questions I'd ask that we use the mic so we can get everybody to hear and Rajiv why don't you come up to and we'll kind of treat this as a panel Hi, thanks. This is a nice talk. I had a question about a comment that you made almost in passing That with some of the groups that you talked to the fact that the APIs are different between you know The cloud providers, you know, whether it's Helian or or AWS wasn't really an issue. I'm wondering if I Guess I'm questioning a little bit how much Easier or better the experience of migrating would be if you actually had more parity in terms of APIs or Similarity, I know that in the printing case, you know, they weren't really using API So maybe it didn't make a big deal, but I'm wondering about some of the other Product teams you talked to you know, it I guess I'm saying it didn't come up as an issue When we looked at the issues that were required or that we had to go through in order to In order to get the migrations done The work that needed it to be done that didn't it didn't seem to come up as in the top tier of issues. Would it be easier? Possibly, but I don't know if it how much it would actually move the needle You talk about the solution migration versus architecture migration So in the process we're able to do a lot more consolidation like the instance counts and things That we able to kind of reduce that and also, you know, you have You know since you're going in fresh you could be putting up bigger instances and you know, what kind of efficiencies were you able to achieve? Yeah, and my particular project migration We definitely didn't see it as just pick up a workload and move it, you know whole we Revisited a lot of those types of decisions. It's the perfect time, right? You're gonna have to open up your whole model deployment model operations model just to make sure everything's the right fit So it would you'd be remiss not to ask those questions in our case though You know because cost is an underlying driver to all of this some of those decisions are pretty well made in advance about the Resource sizing it's much more about the tooling to allow you to turn those over, you know to provision De-provision things faster that we learned That was the big transformation for us, you know the application architecture didn't change a whole lot But the operational architecture for provisioning the application changed entirely So on the operational end as Evan spoke to, you know, they were able to change the whole provisioning approach in architecture Some of the other services indeed They had been running for a while and some of the design decisions They made to your point about the sizing of the different components or where they found the bottlenecks, you know Once you have this system running you learn a lot about how you would have really built it if you had the chance And so we did see some Changes Architecturally as well as sizing with these services saying well, here's what we've experienced or here's what we've learned and Especially with the content flows, you know and and how the the trends and they could also project forward So not only looking back, but also looking forward and saying given that we're going to ship this many printers with this capability And and so they could also look forward and plan for the ramp Second question. What what experiences did you have in performance area from when you went from AWS to open stack HP cloud? Just throughput quality of service and those those areas. So there's two levels at least to that There's the underlying infrastructure performance and each cloud is going to invest in different capital, you know utility available Our application we had to tweak a few things because HP's hardware is different than what Amazon's running and there's different families But it was just a natural, you know, check your RAM check your performance and then dial things and it wasn't a strategic difference HP versus Amazon At the the next level though where the agility really matters for us is the ability to provision stacks put them in the customer's hand You know the developers the testers get an application process done and then deep provision the resource There because we implemented all of this infrastructure as code and you know iterative automated Processes the performance of us in our business life cycle is you know two orders of magnitude or I think I started saying three I two and a half orders of magnitude improvement and that matters more to us than the raw compute performance matters It's the business life on the line yeah, a lot of these content plays involve third parties and providing them with a a Facsimile of the environment in which they're going to deploy so that they can make sure their piece to that pipeline that content pipeline works Well, you know, they don't have any guesswork left because they can bring up exactly the same environment So that's really valuable as well My question is for you guys mostly from the service providing point of view not not so much the the end customer point of view Sorry But I was wondering because in this you're sort of mentioning the migration of shadow IT user groups to you know your your platform and At the same time being open-stack based you're just bringing the the orchestration Functionality on board and you know making it available. I'm wondering if if this you know has caused you to Be ready to do more on-boarding hand-holding then you might have expected You know as opposed to look here's here's the here's the here's the implementation guide. Just go do it Yeah, the real beauty to this project was the fact that there were Projects of different sizes Jim showed that diagram of the different size teams different projects embracing different degrees of the functionality and features provided at this You know at Amazon and then migrating to open-stack They provided us with a great template and a great spectrum of you know what it takes to you know engage and Jim And I got better and better at how we structured that conversation and the questions We would ask and we actually even did some tooling in that space so that we could assess better What is this going to take we could do expectation setting and it was it's a very interesting process And we had enough of a of a user pool Customer base to be able to try out different things and different ideas And then there were folks that we could get too early enough and we could apply these lessons learned so yeah, it did definitely provide us with a template and and a process and You know these are some of the key lessons we learned But there was a lot of blood sweat and tears involved in deriving them You covered it well any other questions. Thank you all. Okay. Thanks for your time. Thanks for sure You