 Hello everybody. Welcome to the Cloud BoF. Here are some of the members of the Cloud Team as exists at the moment. There's a bunch more people who couldn't make it to Taiwan this year and due to the sheer number of time zones between us and them unfortunately they're probably not going to be in ISC either but you know they're with us in spirit at least. So we'll get on to other things in a moment but we'll ask a couple of quick questions first. Who here has actually used any of the Debian Cloud images? Woo! We have users! Awesome! And did they work okay for you? Yeah? Okay. So what we want to go through today is a quick discussion of what we've done in the last year or so and what our plans are for the coming year. We still have a lot of work to do. It's like a lot of things in Debian. The work is basically open-ended. We are still continuing to try and work together to improve things. Feedback on what people are finding with the images we're making and obviously offers of more help are always welcome. This is very much above. This is not a presentation. Although we have a whole bunch of us here it's just because there's a bunch of people doing work in this area. Please don't hesitate to ask questions or pass your thoughts on as and when. I said this is not a presentation. This is a conversation. So Thomas, you have the agenda. Yes. So let's start a bit with status updates what we did. We got print in Seattle last October or November which got even mentioned by DPL. Yeah. There we are working on moving our build systems to FAI mostly which is still working progress and on testing of images. We have some scripts but they are not yet integrated fully into our build process and are not working yet on all cloud providers. So this is also something to do. In the meantime we also moved to Salsa. All of our repositories. Yeah. I became DD and I got access to Casualana so I can now run those tests that we work on in Seattle. Helen is starting opening an end process. So we will probably got more and more members who have access to the machines and we can work on more basically more automated, more bulletproof test because we also moved to Salsa. So we should probably start thinking about more CI, the process of building those images, more automation, less human interaction. So one of the things that we've been doing is getting together these sprints. So far they've been hosted by the big cloud platforms. We've had two sprints in Seattle so far. The first hosted by Google and then we were hosted by Microsoft Azure Team last year. We're hoping to have another sprint this autumn. The folks at Digital Ocean offered to host us last year and we're just starting to organise logistics for that. One of the really warming things, one of the really great things about working in this team is we can see that a lot of companies are not just consuming what we do but they're actively getting involved. So we have people inside Amazon, inside Microsoft, inside Google who are directly working with us on producing and testing official Debian cloud images and that's a really, really nice thing to see. I just want to give a little bit of background for the people who doesn't really know what we are doing so what we want to achieve. So we want to build official Debian image for the cloud providers so that we trust as a community and assign it by Debian going through the QA from Debian with a lot of testing so users can trust this image. So to achieve this we need an infrastructure to build all these images and this infrastructure is going to be built in a machine called Casulana is what he mentioned. And we also need the testing infrastructure so the current planning for testing is analysing the image aesthetically, mounting it, deploying it in a local machine, then deploying it in the providers to check if everything works well. So we have already planned how to implement those infrastructure but it's working progress yet. One other area where we made progress in the past year is that there is now a cloud flavour in the kernel so there is an additional kernel image Debian package that has a lot of for cloud providers not needed parts removed from the Kconfig file. So for example PCMCMAs I think is removed some video camera stuff and so on which makes booting the machine faster on the cloud provider side and makes the image just smaller though this is something that also got accepted by the kernel team and by FTP master to have an additional package. Ok so what images are the cloud team responsible for like do you also are maintaining the Docker images and the vagrants images? No it's something that so far I'll admit we've been not great at talking to the folks who are building those. I keep on mentioning it and I must admit I actually need to talk to people, sit down with them and work out how much overlap there is with what they're doing, what we're doing and does it make sense to actually collaborate more. Yes but are these images being maintained by Debian people right now? They are not in the team is that right? So it used to be that the Docker image was made by Poltag. I'm not sure if it's still the case. And for the vagrant images Emmanuel is building them and we also have the file configuration for vagrant images. Ok thank you. And it's precisely because you have to ask that question that we need to make official images right? So one of the reasons that the cloud team even started having these discussions and trying to work together was that very reason. Over the last few years obviously as use of the cloud, use of containers has taken off a huge amount. We've had people come to us asking exactly those questions so inside Amazon or inside Google or whatever I can see these images that claim to be Debian. How do I know it's Debian? How do I know it doesn't have malware included? That's one of the reasons why we're trying to do this. We want to make sure that people can easily find the official images that are blessed by the Debian cloud team and also blessed by the providers where they're running to make sure that the images are safe, clean. You don't want to have lots and lots of random non-free stuff included even if it's not malware. But equally you want it to behave well, you want it to interact well with the platform it's running on so you get good performance, you get all the features that you need. The cloud providers acknowledged and understood that it is crucial to the project that are also to their customers that they are providing images that the community can trust is real Debian. On top of obviously the three big commercial cloud platforms we've already mentioned, we've been making open stack images for a very long time using a lot of the work that Zego has done. We have those for AMD64 and also for ARM64 which is something that I'm quite happy with speaking as an ARM employee. I'd very much like to add PPC EL64 on issues that I don't have access to such a cloud to be able to test it. So this is a bottle in the sea. If you know someone at IBM or another company that has a cloud on PPC EL64, please get in touch. Obviously we want to provide images that are useful on every platform where they're useful. So if there are any other platforms, any other providers that we don't provide stuff for, please talk to us. We know that there are many, many more than just the three or four that we've mentioned so far. But we need to know about them, we need to work with people and we need to make things happen. OK, do we move to now FAI and status of images? So where we started from was that each different cloud image was built using its own special snowflake tooling. It's a matter of record in Debian we have far too many different image building tools. I remember Wiku told us was it 11, 12 that you found for your talk in Heidelberg? Sorry? And we keep getting new ones. One of the things that we've agreed within the cloud team is we don't want to carry on using different tools for different images. Fundamentally the similarities between our images are much more than the differences. We want to be able to provide something that is common that will give you the same feature set as closely as possible across any platform that you run it on. There will be differences, for example, there's no point running the Microsoft WA agent in OpenStack. There's no point running the Google tools if you're running on Azure. We want to make sure that also other people can take what we have and either derive their own images from them or use the configuration that we've come up with as a good base and either build their own images, extend, share their own images. Obviously Debian is free software. We want people to go away and do their own work on top of it. That's the whole point. So after a lot of discussion at the first cloud sprint we had 18 months ago, we basically agreed we would be working together on trying to build all our images using FAI, which is an increasingly, I guess, different area from where FAI started. So it's not just fully automated installation, we're now generating images. The nice features of FAI are that you have your own configuration, which is separate from the scripts that build things. So we can then easily publish the configurations that we have. They're all free, obviously. They're easy for the people to then derive from. I think we're basically happy with it. We haven't yet quite got to the stage where all of our images are built that way, but it's work in progress. Basically, an advantage of FAI is that there are classes. So we have some base classes for Debian, then base classes for cloud image, and then we are adding class for Amazon, for Azure, for whatever else, and it just adds or removes packages that are needed or not needed. So this is also good for people who want to build their own images. They can take our set of classes and just add new variants or something. I want to correct a little bit. We are on the stage that we can build the images with FAI. The only thing that is missing is automatic continuous integration that the images are built on the Debian machine. So if you want to give it a try and want to build an image, you can just get the tools, the configuration and do this on your own. Also, like a lot of volunteer projects, the other thing we're missing is good documentation about how all of this works. If anybody does want to help right now or right today, that would be an awesome thing would be to dive in with docs. I didn't have a question. I was just raising my hand to say, let's hang out together later today and I like writing documentation. Be careful of admitting that. You might get lots of friends. So that's the state of play of where we are today. As I said, we're hoping to organise another cloud sprint for Hogleys in November we're looking at. So October, November this year, I said digital ocean offered to host at their offices in New York. We've not really started organising it fully yet. More details will be coming out on the mailing list and whatever shortly. And we have the beginnings of an agenda being worked out. We got together last night. There will be more and more things that we need to talk about and work on when we get together for a few days. We do tend to work very well in the sprints and I must admit, not quite so well or not quite so publicly well outside of the sprints. One of the things I'd like to have us do is be a bit more active publicly so that everyone else can see what work is going on. I liked in Debian, not just to see things being done, but to have things being very visibly done. It's a great way of building a team. It's a great way of getting more people interested and getting more people involved. So if we were to wrap up this first section with some comments, the first one is that we moved everything to Salsa. We don't have CI turned on the way we want, but when we do it can be watching Git repos and be pushing to Kazulana for build. Kazulana is a monster machine that Debian bought 18 months ago or so and we got fully up and running at the last sprint. It's where we build all the images. Although the providers are offering us VMs and are encouraging us to build on their infrastructure, we maintain the principle that we build our images on infrastructure that we control. I guess the last piece was we're interested in the ability to push images to a variety of providers. Right now we only push to three providers. We do it in a manual way and we can tie that in at the end of the CI process after test and validation. So if I got correctly basically you would want to not only have continuous integration for the images but actually also continuous deployment to all those providers whenever something passes QA and we decide that yes this is the new thing we want to release. That's really good to hear actually. Exactly that. So of course we will have the stable releases that go out each time we do a new stable release but we'll also have regular updates with security things happening. We'll also have regular whether that's daily, weekly, whatever builds going out. Once we have a proper CI pipeline going where builds are triggered when changes happen what we will want to do is of course is run our own QA and then we can push to the various platforms. Each of the platforms have their own extra QA infrastructure some of which is huge in terms of what things do. It can take a couple of days for an image to go through and be tested fully. So obviously we will be providing our images we'll be pushing into their test pipeline and then things should come out fully tested, fully approved and ready to go. Currently I think we provide at least for OpenStack and Azure we are providing daily images, weekly of testing and each time there are updates on either the main archive or the security archive the images get rebuilt and we publish them again. I think we agreed to have weekly builds for the image for testing and daily builds but not official for Debian only for CI to check if everything is okay so if we detect something is wrong we can notify it quickly and fix the bugs as fast as possible. Is there some kind of roadmap or dashboard where I can see the current status of this image? Not an overview for all the images at least. I know for Azure for example that there is azurebuild.debian.net which has Jenkins running which shows the latest logs and downloadable daily images. I think for OpenStack just download from Casilana. I would like to contribute to the team but I was thinking where to start like is there some place where I can get a... So for the big four providers Amazon, Google and whatnot so we will push to them but for people doing OpenStack things I would very much like to see somewhere where public cloud providers and whatnot could register themselves and get either a trigger or an image push to their cloud so we would need to do some programming with a web interface where people can register these kind of things and there if you want to volunteer doing that it would be awesome. Right, and the other place where we've had a volunteer whose name I can't remember because Zoe and Rubish has already started working on an image finder so that would be something where you can go and if you just know you want a Debian image which includes the following features that you would be able to then go and find out say the AWS ID for it or find where to grab it for whichever region you are in Google Cloud Engine or whatever so it's significantly harder than it should be apparently to find the official Debian images and make sure that you're using the thing you think you are so an image finder like that is also a good way to help I know a bunch of you have one and we have people looking at what they do we could do something similar but we don't have to follow them it's a good way to dive in and get started I reached out to them and they are saying that they will not publish their make their code public for the image finder because for reasons one thing that it would be really nice to have volunteers to work is testing infrastructure because right now we have some code but it's a code that we did in two days, three days so it's really basic we are not near what we want to achieve all the staging of tests we want to achieve and also we need that all of this needs to be integrated in a way that is easily integrated with all the infrastructure so we can trigger the CI test so easily and I think this is like first phase before moving to other features and this would be really helpful basically the testing will be non-trivial because we need to build images then we need to check it statically then we need to run it on some basic tests on Debian machines so we can trust it then if it passes we need to upload it to particular cloud providers and test that it runs on Amazon that it's run on Google and for example on Amazon that we can enable this enhanced networking on Google that we can attach some high performance whatever so it will need to be multi-staging and it will need to be integrated into our building process and our CI process so we cannot use basic Salsa CI because it's just one step and quite limited so we have some plans but we need to first document them a bit discuss and probably it will be multi-staging stuff but for those who are looking on what we agreed what the test site should look like there is a GOBI document GOBI documents from the sprint in 2016 and 2017 the test suite that Thomas and Helm worked on basically the set on tests that we agreed that are nowadays documented in the GOBI collaboration editing we also have some pictures and diagrams explained so if you want to know more I can send it to you and explain it to you and also another thing is that we use a PHY tool to build images but we also want to isolate building so we have to build a virtual machine execute PHY inside it, generate the built images and also manage all those virtual machines so I don't think we have it yet that was my question is anybody already doing the building of the images in a clean virtual machine or not yet? so currently we all use FAI just on our own computer so there's a script in the Git repository called launchkvm.sh this was done by Ben from digital and I think we should have a look if it's okay and try to use it but what you can do is you can get our repository and run FAI on your own machine so this is still in progress basically previous scripts like booster visit and so on we were running this on clean AWS machines but because we decided to move to FAI we need to polish some corners since you mentioned that I can run FAI on my own machine how much resources do I need to be able to run those builds and the answer to that is not a huge amount it's essentially doing the bootstrap and tweaks around it so you need to have the space for your image you need to have sufficient memory and kernel to do it obviously the bigger the faster the machine the faster it will go but don't worry too much about resources so let's discuss maybe these vendor agents and stuff or do we have something? so basically we are now building our own images but to fully utilize cloud providers there is need to have some agents some demos running on virtual machine in cloud providers their functionality differ on different cloud providers so for example on Amazon there is no particular need to have it it will work without any agent but then we won't have detailed monitoring we won't have some ability to run easy to run command or something like this we won't have this SNM something I don't remember what's the meaning behind it but basically this is advanced management of machines on Google on the other hand we need to have some agent to be able to inject SSH key to login into it I'm not sure what's the status on Azure but I guess this is similar so voila is currently already in Debian stable and in the images it's also used for getting the SSH keys into the machines getting the monitoring or getting information back into the Azure monitoring suites and so on so basically the status differs between agents for different environments it looks like for Microsoft there is the best situation Google has packages but in their own repository because they are some problems with the FSG and quality of packages so one of the things that we've been telling all of the platform providers and most people are understanding this is that obviously if their agents are going to be in official Debian images they need to meet Debian standards so they need to be DFSG free obviously they also need to be in the main Debian archive we are not going to build images and put Debian's seal on them if they are pulling in things that are not already in the archive because it's not Debian that way is it so that does then lead to some tension that some of the providers want to have quite frequent updates to the tools inside their images and that can be a bit fought if it means that they want potentially major updates inside Debian stable but that's a discussion that's ongoing the very first thing that must happen is they've got to be in the archive and they've got to be maintained and we also managed together with a release team a stable release team to get at least a Vala agent for Azure updated in a stable point release mainly to cover the Azure stack which is an on-premise Azure to support that kind of new hardware and we also had some discussion with a stable release team regarding other agents as they are leaf packages they are probably not going to break most of end users installations as agreed with the release team so long as the updates to the agents are reasonable then there shouldn't be a problem with these things going in but who defines reasonable needs some trust of what we're doing and of what the cloud providers are doing to make sure that the newest agent doesn't do bad things it's not going to be very buggy so one of the things that will help on that of course is as we boost our own CI if we're doing a lot more testing on all of our images we will be able to verify that a new version of an agent isn't going to break things how is Ubuntu solving this problem? Are they shipping non-free software? and the frequent updates problem also? That's a very good question, I honestly don't know Ubuntu in my experience tend to be a bit more open to doing less free thing as when it makes commercial sense I haven't looked so I know but if you look and tell us that will be wonderful Thomas what was next on the list? Next we are almost done Let me check As you probably noticed there is much work in progress we need a stiff point that we need to have it more public so we intend to to have some status updates on our mailing list basically we have mailing list so you can subscribe to it So yes, we've got the Debian Cloud mailing list we've got the Debian Cloud IOC channel where at least most of us are idling we do need to be more public though and as I said we need to have better docs A to help people to see what we've done and how they can get more involved I promise faithfully that we're going to be doing more of that in the next year if nothing else I'm going to make it my task at the clouds wind to get a lot more of that organised If you want a hands on I've seen at least one cloud provider being here in the room persons from one cloud provider in the room I'm willing to give you a hands on on Azure platform probably SIGO can also help people to show the images on OpenStack I guess One question, is there anybody interested in building their own cloud image and want to use the tools Just if you start and if you have any problems with Fi I would suggest you could also join the Debian Cloud IOC I just did I did about a minute ago and then if there are some I would guess in the beginning there may be some question related to FAI and then I can help you Currently I'm basically building my own VM images automatically when I provision new VMs but it's a very janky process that basically involves building any trimFS blob that has a precede file baked into it and yeah At least it's automated and there is a possibility to start creating our own image of OpenStack So one of the things that we started last sprint was a conversation about formalising our relationship Debian probably through SPI with the cloud providers so that our images can be found in the marketplace for example so instead of finding in the public area you can find it in the AWS marketplace so we hope to not conclude but continue that conversation this fall and understand the pros and cons of such an agreement with the cloud provider and the availability of the images to people who want to use them and how they can be found so one thing is the image finder the other one is actually being in the marketplace for Azure we are listed in their marketplace depends heavily on the platform we still have in flux our management of accounts and access to this cloud because basically we need to have our own official Debian account on different providers and we need to be able to manage users so I can do something publish images but so basically we need accounts for users for real people and we need accounts for roles to some machines can automatically publish something so this is also still something that we are working on but I think we discussed that last year with Steve extensively and he agreed to hand out accounts for publishing for I think that was everything we had on our agenda yeah so man we just seen ten minutes on so are there any more questions people do we have any more volunteers who want to come in and spend the rest of their lives making Debian cloud images great or damn I was hoping we would get somebody so what will happen in the next months that I know of is that the kernel cloud image will be further tweaked and we will hopefully see the feedback from the various cloud providers that those kernel images are working properly on all cloud platforms and maybe there are even more things that we can disable in key contract to get faster booting process first I'm really sorry for monopolising Mike but since you mentioned the kernel package for cloud being tweaked further I was kind of curious so much difference did you actually notice like lightning the kernels that we shipped in VM images the boot time is like quite much like half of the boot time only that's actually a lot we removed a lot of things that are not present on cloud platforms also network stacks that we do not need and realize I have a quick question is it for instance installing security updates on the first boot of a VM something that is done currently and Cloudidit does that for you? This is an outstanding question though some of the cloud providers want all the images to do on attended upgrades or similar to make sure that images that are running are as secure as they possibly can be obviously if you're hosting say 100,000 Debian installations you will really care if there's a root security hole that is working its way across your entire campus corpus sorry of course there are also potential downsides to doing the unattended upgrades of course your database server for your website suddenly in the middle of your trading day decides no no I need to do a postgres update and your site goes away there is no good single answer it's an ongoing discussion. That's why I've asked about on first boot because that's an easier decision there whether Steve brought up the discussion during last sprint and also after the past sprint on the Debian development we should enable unattended boot per default on our images not even for cloud images but also more generally. The Debian security team don't like unattended upgrades there are apparently they see quite substantial problems with the design and how it works I haven't spent enough time yet looking into it to agree or disagree. I do think that we need to come up distro wide with a better security policy I'm sure all of us as responsible admins of our own systems are making sure we're always patched we should make it easier for our users to get that too I think the problem is that unattended upgrades happens at random times and you really want to run it on control times so some kind of say staged updates that you can say that now is the time to update a handful of hosts and find out if it works and then put it on to the others it's a trouble we don't really know who should be the one who controls these staged updates. So there's all kinds of conflict that can be done actually until F is coming Do we know what the other OS providers do and so on? I know that CoreOS does automatic updates and reboot, but I have no idea about other Ubuntu does definitely I'm told that Fedora have something similar I don't know how well it works or any details I'm afraid Amazon, they have their own distribution they run updates on restart it's great but of course this varies massively by use cases so for people who are running their database server for their e-commerce site they'll have very long running cloud images which they will want to manage totally equally we also have a lot of people 30 seconds, 5 minutes or so they don't care what actually we've had feedback for Google folks in particular were pushing for this one of the reasons why we care so much is they care if a system takes 10 seconds to boot instead of 5 seconds to boot for exactly those people so they don't want cloud in it taking time at start up that basically means your boot time has tripled now again for those of us running a laptop or a server at home who cares about 5 seconds but if you're running 10,000 images to do a very simple thing and then you want them to go away again 5 seconds x 10,000 ads up another thing is that when you have a very large deployment just imagine 10,000 VMs doing a test upgrade at the same time on your local mirror that can be a lot of resources too I think we need to support all these different schemes to do it at reboot and so on but I would guess it's not a special cloud problem but how Debian could support all these different kinds of when to do the updates so I have a question besides the big cloud providers Google, Amazon, Azure, are there any other cloud providers that you would like to see in the beginning that Debian supports and where we should upload official images, just shout the names of the cloud providers so I would actually like that to be supported Gandhi.net currently ships extremely janky images and it would be actually quite nice to have properly supported ones I just say that there are a lot of Chinese colleagues here I know that Alibaba also have very, Alibaba and Tencent in China they have very popular platform being used by millions of people, if the colleagues here can connect us to their people actually I'm using Alibaba which is called Aliyun and maybe tomorrow I already know that Alibaba is very special because they cut down something in the device and install something called Alidun which is called program yet last year, maybe somebody heard of that so I don't think they will accept some image from Debian official reports but they do have such a thing. Do you have contacts in Hondro with people from Aliyun that we could get in touch with? No but I'm keeping in touch with them so I can contact with them if you want to talk to them okay I can help but the problem is Aliyun has one option which is default on that is enhanced by this option they will install something into your system which I believe it will help for the whole system that cause problem last year. This is for security officially or literally this is for security but I trust it has something to do with other stuff which I could not say. But I guess even if they install something we cannot do anything against it but we still can provide an official Debian image that the users can use. It's said to go back on the topic of automatic upgrades. I'm slightly worried that what we'll run into is something not entirely different to what we do by installing recommends automatically where lots of people then just end up turning it off and I don't think that would be a very helpful situation. So maybe what we could actually do is provide either some flag you can set through CloudInnet which basically sets a policy or even provide different images so we provide a short lived image which I mean we can do something really crude like it will do power off after 24 hours or like some like that or we could do some other mechanism to say you shouldn't be running this on long term and then it's up to users to do whatever they want. Hi, so I'm Kay, I'm the person from one of the Cloud providers. I work at Microsoft so just a quick note on this so one thing that we see a lot is that customers do we for our internal Microsoft services we have an image that we base off of and automatic updates or unattended updates are turned on in that and most of the people who use that image turn off unattended updates because it breaks them during production and they need to be able to test and deploy on their own type frame and so I think that making it easy to create custom images is really the best answer to this so people can start with whatever the default is and then create their own and they have focus all over it. Absolutely, it's what we need to do, it's what we want to do. Anyway, I'm told by the video team we are out of time so thank you everybody for coming along I hope that was useful and interesting. Please if you want to know more, if you want to join in, if you really want to help catch us either here this week or join the Debian Cloud IRC and mailing list and we will we will be keeping you updated there. Thank you very much.