 Well, hi there Good afternoon I'd like to start with the question someone of you ever set up a cloud infrastructure. Have you ever set up a cloud infrastructure? No, not so far. Okay. Have you ever set up a cloud infrastructure? Can you tell us one or two sentences about this? Yeah, okay, those are exactly the key features. Well Welcome to my talk. My name is Neitz-Marclos I'm here with some kind of a split brain. I work for Inovacs in Germany I live in Munich and we actually do something like you told us we plan we design consult and actually from time to time implement scalable data center infrastructures clouds kind of I'm also a member of the board and the program chair of Linux dark the conference and takes place annually in Berlin in Germany as well and Well, the obvious disclaimers if you are aware of virtualization configuration management and all this fancy stuff we hire and if you also aware of those stuff and Like to talk about this. Please take place. I take part in our core for papers next time of Linux tag will be most probably in June 2013 well as my title already suggested I'm here to Tell you about ten ways how to ruin your your cloud computing experience. I Did a couple of cloud projects so far and this is my way to share my experience Have a view and I'd really love To share also the way the way back So if you have some comments or if you have questions don't hesitate to ask Don't hesitate to comment what I talked about So we can well have a better experience and maybe enhance our experience to avoid Well disappointment about cloud computing speak about cloud computing. Anyway, what is cloud computing? I have a big problem with this term Because those are just examples Who of you knows about OpenStack the OpenStack project? Okay, quite a few who knows about own cloud Also sugar CRM Yeah All of these three projects. They are obviously a couple of others Which are the like this is just an example like Apache stack we've seen there we've seen open nebula You've seen several other storage components. You've seen obviously a couple of other so there's a service project SRS frameworks which of these is Cloud computing all of them or just one of them. It's for me. It's very confusing especially if you talk to other people Especially if you talk to the with management management usually has heard about cloud computing, but Very often they have no precise idea about What they're talking what they're actually looking for are they're looking for some Infrastructure as a service deck are they looking for some Storage solution are they're looking for a software as a service solution and this becomes difficult if you talk to engineers because engineers tend to Talk with very precise terms and therefore I prefer At least what I am doing most of my time is infrastructure as a service Which is the most to my opinion a technical part or at least it's in the stack the lowest Virtualizing Yeah, the infrastructure like We knew in the past to install servers and now install Virtual machines on top on some virtualization holes so actually there's no need to confuse your engineers and I try to avoid the term cloud computing and During the rest of my talk, please let me know if I violated this rule Well, if you are planning to set up a new Cloud computing infrastructure as a service System if you plan a new data center and you set up a new private cloud you name it If you have Five engineers in the room. They have usually about six to seven opinions about Which operating system to use which? established solution to use which hypervisor to use and so on and You often hear something I heard Ubuntu server does not work with virtualization the sister of my brother-in-law's Uncle taught me it did not work for him. So this must be broken and to my opinion, it's a very Shaky ground to make decision on based on feelings. So It's very important to define a specification or something. What do you actually want to accomplish and Based on this specification, which is hard work actually try to evaluate the potential options and evaluate them I Talked about just today earlier today. I talked with some storage guys What to actually use for for the shared storage somewhere you have to store your VM images somewhere and Wow I've I got a couple of new ideas and Suggestions about how to implement this and If you talk to the people to the marketing people or to the the developers, they often very excited Obviously about their solutions, but take your time do some testing do some test that up and evaluate the options idea number three environments and how to Set them up At the moment I work for for a client whose business is Actually to run a surface a service Yeah product He has implemented the software system based on Java running in a VM and at the moment they have a 100 about 100 clients and each of the the in each instance of the software runs in a separate VM and Each of these 100 Instances have been handicrafted by hand by the system administrators themselves. So They edited the hypervisor configuration files edit copy and paste a new virtual machine Edited and decided about wow should we give this four CPUs or should we give this six CPUs and They decided on on memory and so on and so on after this they Inserted the CDs. They exported the CD images to the VMs They booted them. They configured the operating system. This does simply not scale This is the first problem. The second problem is it is very error-prone because They showed me actually that they're wiki they had for each of their clients Vicky pages and each page consisted of printed on on paper something like three or four pieces of paper and It's very easy to make one mistake in all those steps. So Don't even think about Doing all these steps by hand if you are planning to set up a scalable IS platform and often it's Most people know this and those people use actually something like a configuration Management system or whatever but there are The small pitfalls like Okay We forgot in the script to set up or configure an additional IP address on this Specific interface for example. It's a very simple task to do on a command line But if you do this Manually you have to remember this very step for the next time and the next time and the next time and it's Very likely that you forget it sooner or later or at least once in the future So take this extra time try to make this right But don't overdo the whole thing down over engineer I also seen situations when colleagues try to Deal with every potential situation that might come up For them for other people or in ten years in the future whatever It's not an easy task to to to find the balance between the two of them but nevertheless if you If you're sure that a specific task needs to be repeated twice or at least three times It's always a good idea to to use your configuration system to implement this within your puppet chef Salt stack whatever you name it next Subject Stunt me at the first time and I experienced it. I was working for another client and the client insisted on Buying Some wow some cloud computers some infrastructure as a service Services from third-party running an enterprise level operating system on Enterprise level contract and they paid quite a Lot of money for that. We said well, it's always a good idea to have commercial support as a backup You have to decide if you actually need this But if you decide that you need this commercial support, that's fine But after After that we set up the cloud infrastructure and some of my colleagues took over for the actual day-to-day operating of this cloud infrastructure we earlier set up and It appeared they had some problems with I Don't remember something that the storage was very slow in Certain situations and they complain and they complain and I said, okay You have this very expensive support contract with your service provider. Please call them and open open the bug and Insist on on have this fixed or at least have this investigated But yeah, well, we could we could try to give it more memory or we could try to set up another machine or We could try Rebooting several times without saving all or data or something like that But they Were not willing actually to do to call and insist on the SLAs well so my opinion to High money Service level agreements is a little bit split because It's often a decision to well trade off fear with money or something like that. So We pay you this amount of money and if something goes wrong, we May blame some someone else and I'm not sure if this is a very good Basis for infrastructure as a service or in all a technical a technical project so Think about what you actually want and what your personal service level agreements are This I think it's number five Yeah the eternal battle between developers and operations operations teams It's Someone of you are working as a developer Who is actually writing code? Oh, yeah, quite a few and Who of you is is responsible of actually running Running some infrastructure some servers, whatever Okay, I think the admins are slightly in the majority and Who of you have experienced either as an as an admin as a system engineer as a Ops guy that the developer sent you some broken code or some poorly documented stuff Something that was difficult to run or was error-prone or something like that. So Yeah, okay and who of you developers complained about the the nosy and arrogant Admins they just don't do what they're supposed to and Have their own way. Please raise your hand. Okay Very polite very polite audience today. Yeah, wow Obviously there are two There are two groups and there are two sets of duties that still have to be done, but Working together and talking together is probably most of the time a very good idea and we tried several Several approaches to overcome a little bit this this this gap of the two groups so a colleague recently told me that they Invited there. He was also an administrator. They invited one of the development Guys to join the team and actually do one of their software deployments or something like that for for a week or for for a new release I think and that was very helpful because at that point the developer learned the problem that comes with running and administrating a Server system and the the problems with Undocumented log files and and stuff on like that and on the other hand they sent one of their administrators to actually do some pair programming together with the Development team and that was also very helpful how it looks like to do coding without proper permissions without a route account for example and They learned a lot from each other there are a couple of new ideas like scrum and Kanban and Agile Agile methods Which are very or are quite common at least in the in the developer community these days who is actually Using some of these methods like scrum or Kanban Please raise your hand and who has never heard of any of these methods Okay, well, that's less than I expected so my impression it is that the whole idea of scrum and Kanban Is not that widespread widespread in the Operating community compared to the Development community, but it's it's still rising the number is rising and There might be some good ideas in there to my opinion, there's also Some kind of voodoo stuff involved with this as well, but if you seriously go and try these methods to to work together and work as a team and Sometimes they call it a deaf ops I made some good experience from that And it helps in your day-to-day job to understand What the others are doing? Yeah, if you set up a Cloud infrastructure There are a couple of components involved usually there is the virtualization engine. There is the Configuration management, there's the storage layer There are deployment tools and so on and so on and For example a friend of mine Was not satisfied with the existing deployment tools for example, we had a look into puppet for example and And Puppet is written underneath and I think in ruby is that true? Yeah But he did not like Ruby He was a per guy and he said well, okay, I can do better. I write my own a configuration management system in Per And well, this is actually quite a task and to be honest he wrote a rather decent system, but it's very hard to maintain something like that like that Don't underestimate The work that is necessary to support and Enhance and to debug and to maintain such such a system the same applies to writing small shell scripts here and there and writing a share script that automates the Initialization of the virtual machines and another script that pushes the configuration on there and another share script that Roles out the software and so on It's very difficult to maintain these small scripts and the key point is Standardization here Standardization is a very strong weapon if it's used right because there's so many other people You can talk about this you can Google Problems you can most of the time even Google the answers which is even More useful and if you join forces with with other developers with other admins It's much easier To set up something Which is easier to maintain Yeah, well We invent we and it's also Sometimes it's a little bit about Not invented here. You probably know this because we wrote is This small tour we actually Want to use it even if we know there's something Which does more or less the same job But has some other features. Well, however, you may your mileage may vary in this situation Okay, I I'm interested in your opinion about this experience when I set up a load infrastructure And not again for for client We we finished that project and after that we we set up an extensive set of Monitoring hooks and did a lot of monitoring and Spend some time doing or lock TV or watching and look for the fancy fancy graphs and so on and Pointed out. Wow. Look at this machine It's utilized More than 60 percent. I think we have to throw in more CPU on this or was it memory? I don't remember But it's my opinion that especially in the virtualized cloud infrastructure this is kind of Waste of resources Because if you run a proper cloud infrastructure everything or at least the key Parameters should be elastic should be Adaptable so if your application actually needs more more memory it should get more memory and that's What is get you less for? if you see you reach a certain Freshholes, you should just and most probably automatically Raise the assigned amount of memory same applies to To displace and to CPU utilization, but having a virtual machine idling 40% of its time. It's just waste of resources It's obviously not that easy In reality because you always have peak times and depending on your On your business and depending on what you what you actually do You won't have a flat line of resources over a day over 24 hours period or during During week or whatever But if you look closely to your graphs, and I hope you have a monitoring system in place you can use this very good for that You should look for Ways to adapt the necessary resources to your need For just to give you an example One of my clients was in a was a big TV station and Their web servers has been very To a big degree utilized during the the evening when the people Looked the TV shows and looked up something they had on the screen or whatever and So they needed more CPU and especially memory during that period on the other hand The developers themselves coding this website always Demanded more memory because of their tests and their optimizations and stuff so We decided and set up a scheduling task that assigned during the morning and the early afternoon To shift the memory more to the development and staging clusters and in the evening when the Developers left the office at five o'clock or something we shifted back the memory and the CPU To the web servers again So if you go for cloud you should also use the feature that Comes with it. I already talked about environments I Think it's pretty common practice nowadays to have one production environment and at least one Testing or staging environment Who of you uses something like that with a multi-level okay, most of them and I am interested in how many different environments do you actually run do you run with two environments, please Free five are free, okay Yeah, okay interesting I Seen all of those Setups if you have Just two environments, it's sometimes difficult to test for well different groups of developers But on the other hand if you have five Testing environments or or five environments in total Are they actually all looking like the production environments? No, it's the answer Okay, the plan is having a plan is always a good idea. Yeah, okay The key point about majority environments is to have them as close as The production environment is because that's the whole point of it if You check for something that is simply not Mapping to the reality It's going to run after the the rear the live deployment Why did you Make all the effort in the first place so it's important to have a configuration system that Has its sources in a single place I think this is a good best practice so We tried to Refactor our configuration system for example To have as much as possible from from a single source we actually put all our configurations into Software control system depending on the client a subversion or get a repository and We don't use the configuration files directly, but we pull it from the repository and We try to have as much common configuration code as possible and have a very lean a layer of Things like the actual names and the actual numbers of virtual machines That get applied to this set of configurations This is really hard work to do but It saved us a lot of work in the long run It's always a battle especially if you talk to management To assign the proper resources especially in terms of Developers of admins of end of time on and for fair especially on that time But if you convince them to give you these resources to my opinion it usually pays off rather quickly after a couple of weeks or Or at least months To enhance your configuration and and other components as well scalability Scalability is a nice feature if you have it, but it's also expensive But it's even more expensive if you don't plan it and Forget to implement so if you have ever hard-coded in your code that there is just one database and I don't know several The the database is reference from the code in 20 different locations. It's very hard to set up That changes that so you can scale that up for example At more Breed slaves for a database Well, well, you name it it depends obviously on the on the architecture you set up but Scalability planned in the beginning is always something that pays off in a long run This is number 10 the 10th way to ruin your cloud experience. It's a non-technical Experience I made There are so many Problems that actually don't need a technical solution I bait a lot of people come up with very Fancy and enhanced ideas To work around and to model this apparent problem I said it's a little bit Exaggerated, but I experienced something like that. How would you how should we implement the SSD bagged hot fail over high ability enhanced LW driven role-based user management For just you and me for example Isn't it Enough to in this case share the root password which might be easier to implement Not that I'm a big fan of shared passwords to the lease but it is Well, we just failed at this situation to think about is this necessary for business case or the use case or whatever and There are so many situations I hope that the people who actually implemented that Had Thought about this in the first place. So less is more some times because if you have a simpler Architecture, it's easier to maintain. It's easier for you to enhance the important parts of it and To have more stable overall installation. Well these have been my 10 ways how to ruin your cloud experience and I hoped that I actually gave you some ideas where to Where to start in your day-to-day work This is my contract information if you like to Ask me something or have a suggestion if you're free to contact me or just ask right now and Give some suggestions some some opinions However, thank you very much for coming here and for attending the session are there questions or Remarks I Yeah, wow Usually That's necessary most of them think about scalability anyway, but Not all the time in all situations and I think that's that's fine and actually that's what we are for So If you have a special mindset and if you have set up An application stack before you obviously think about the problems that have been Arisen in the past and it's our job doing some consulting and find out which might Become a bottleneck in the future Well, if it comes to to the conclusions, well, it depends obviously because Someone who is actually running one server with Free directly attached Terabyte drivers is hardly convinced to pay for a high scalable Geo Delocated storage solution, whatever. So It's just necessary to do to find the right balance. I think But talking about takes place. Yeah Yeah over there Pardon Yeah Yeah, I do some I do see some patterns One of the the patterns I see is that The whole idea of cloud computing the list is this DT and stuff It's often underestimated most of my clients are Still stuck with virtualization and they don't think About the idea that goes beyond virtualization, but scaling and Automatization and and stuff like that and that is at the moment Most of our work has to do with something like that and And We have to try to explain the advantages of the good cloud computing Without sticking just to the marketing blah blah, you know and Trying to show them the Actual advantages they have so your system administrator at the moment needs About one day to set up a new client after You set up the proposed changes It should be possible to set up a new client for him within one hour. So It's an advantage So he gets eight times fast or something like that And Wow But it's hard to generalize have you Have you made different experiences? Yeah, well, they are obvious advantages money-wise for example if you Look for the numbers to run Say 50 machines 50 servers which can be depending on the setup the five six seven eight virtualization holes and Setting up some configuration management on top of this This makes sense Not only for the very big organizations, but today and both are most of our clients Medium-sized Companies who just want to save money save save power save workforce Yeah, well the local governments. I have no clients Within public administrations a couple of friends on actually on the on the other side of the table and they consider this Cloud computing is at least to my opinion for public administration still something they Carefully well evaluate Maybe it takes another year or two. I don't know I Follow rather closely as far as I understand it what the what the Guys from current development actually do It comes to facilities monitoring of Data from the Linux kernel and so on but this is usually not Not to the least the problem We are we had a problem so I Talked about the example the service provider that hosted the the private cloud for our client and To set up a new VM for example took them Under the SLA two days two days and Actually, it took them usually about three days because they Edited the configuration files. They did copy and paste they had a complex process involved and so on and I said, whoa, this is cloud computing. You should be able To point your browser to a website and click on add a new virtual machine but They did not do it this way and these kinds of problems are much more common Non-working c-groups in a in a corner subsystem or something like that because well It's comes to two real technical problems He usually Easily come to technical solutions if you talk to the right people There is Any last question I I've been Showing that we have only a few minutes left Yes Things answered. Thank you again one