 So to follow up with a demo from what Jimmy and Timmy showed us earlier from Google Is Mandy from Google. So please give her a round of applause Hi, everyone. How you doing? Great to be here. It's really wonderful and whether it's gorgeous bunch of smart people in the room, so I'm gonna really follow up on what Jimmy and David presented earlier. We're gonna talk about Google Compute Engine and one of the reasons we're here Obviously is because we now support Debian. We have we see and squeeze so Compute Engine now with Weezy and Squeeze My name is Mandy Waite. I'm a developer advocate for the Google Cloud platform And if you want to find out more about me about dot me slash Mandy Waite So we're gonna spend about 10 15 minutes talking about Compute Engine a bit deeper dived and Jimmy and David Gave you but we're also going to talk about customizing virtual machines Then we're gonna have an interactive demo. It should be a bit of fun hopefully We'll get to that very soon about my feelings about doing interactive demos Right, okay introduction to Compute Engine So Compute Engine is part of the Google Cloud platform And the Google Cloud platform is basically broken up into three main areas. We have compute. We have storage and we have services In the compute side we have Compute Engine, which is an infrastructure as a service offering and We're gonna talk about that. That's the entire subject of this talk Then we have App Engine. App Engine is a platform as a service offering and This slide who was at Jimmy and David's talk? So we've already seen something similar to this already, okay So I'm just a little bit deeper. There's coming some of the same things that they did But I want to provide context for those who weren't here so App Engine is a platform as a service offering and Basically, what that does is provides you with an entire software stack. So you build your applications on top of that software stack and The software stacks are what we call run times and we have run times for Java We have run times for Python, run times for the go programming language and one for Python as well PHP and I was having a conversation today about PHP and Probably not a good idea to make PHP available to more people in the world So my PHP must die t-shirt is in the post so Where computes engine provides you or removes the need for you to worry about actually building and maintaining service App Engine actually provides you with the software stacks. They didn't have to worry about building software stacks buying and licensing software And what that does that frees you up to actually develop your application code develop your business logic I'm actually worried about installing software stacks on the storage side. We have Google Cloud Google Cloud storage, which is an object store This is used to store in files Unstructured data any type of file data files executables images blobs that kind of thing Pretty much any type of file and pretty much any size Google Cloud SQL is our relational offering. So think my SQL in the cloud and cloud data store is our No SQL our massively scalable no SQL service that we use internally, which is built on top of a Megastorm big table and Which used to be tied exclusively to App Engine, but recently we freed up from its reliance upon App Engine And it's now available from pretty much anywhere via an API on the services side We have Google BigQuery and there's probably going to be some bits missing because the boxes aren't showing up. So Hopefully that's going to be reasonably clear Google BigQuery allows you to run interactive analysis across massive datasets And we're talking about interactive here. So you can type in a query get results back very quickly The data sets we're talking about are the size and petabytes terabytes billions of rows of data Uses an SQL likes intact. So it's very simple to use very familiar to most people who've used SQL before It allows you to get results back in seconds and not minutes or hours like you would do with say a map reduce We have Google Cloud endpoints, which is our entry point into the cloud This is implemented on App Engine But it pretty much of using App Engine. You can use cloud endpoints to access any any part of our cloud This allows you to write your application logic your business logic and expose it as an API and the API is a very very similar to the API that we offer The discoverable and you can build client libraries from those API's Android for iOS and for the web Caching we have caching services pretty much everywhere. So we cash at the edge. We have mem caching implementations and cues we have task cues that allow you to Join App Engine and Compute Engine together. So you can actually move your workloads to where they're most appropriate and We have a lot more and of course the whole thing about this is it runs on Google infrastructure This is the same software and hardware infrastructure that we use to service your search queries and also to run YouTube and Gmail those kind of services So Google Compute Engine very quick look at this We're going to be talking about virtual machines about networks and firewalls about storage resources and disks and about the tooling that you can use to actually manage all of those compute resources and The API that you can use to actually build your own applications around those compute resources compute resources a quick overview of the computer engine architecture Basically make four main areas virtual machines networking storage and the API and tools This doesn't have cloud cloud load balancing You may have seen if you saw this slide earlier from Jimmy's talk He actually had load balancing as a component But we were kind of we released it last week and it was kind of touch and go as we released and I kind of changed my slides I removed that end up with no load balancing and then really couldn't be bothered to put it back again But we do have load balancing So let's go a little bit deeper into each one of those sections so virtual machines We have Linux virtual machines. So we obviously have the Debian. That's why we're here. We have squeeze and we Squeeze and wheezy and we also support sent us, but we're not here to talk about sent us. So we have many virtual machine hardware configurations basically combinations of Memory CPU and disk so you can build pretty much for whatever you want to in terms of the hardware configuration We also have fractional virtual machines as Jimmy mentioned a small and micro virtual machines and these are shared core instances So therefore workloads that don't really require huge amounts of process and power We have root we have root access effective root access for all of your virtual machines and we recently introduced sub-hour billing So you pay for the first ten minutes you always pay for ten minutes of your instance uptime But then we build by the minute from that point onwards So if you use it for five minutes, you'll pay for ten minutes if you use it for two hours, you'll pay for two hours In terms of network resources well virtual machines, they don't live in isolation. They need to talk to each other They need to talk to the internet so All of the virtual machines within a project are linked by a private network and you can build your own private networks You can use network resources and firewall resources to create rules that Determine how the virtual machines will interact with each other and with the internet Advanced routing allows you to build advanced networking configurations. That's Allows you to implement things like virtual virtual private networking recently we introduced a layer-free low balancing as I said that was announced last week and Remove from the slides. Unfortunately That allows you to low balance all of your HTTP traffic or other traffic across multiple virtual machines or virtual machine endpoints It also supports health check-ins to make sure that traffic is only routed to routed or routed dependent where you come from to instances that are up and healthy and It's effectively a cloud native implementation And but what I mean by that is that it's implemented as part of our network infrastructure It's not kind of bolted on it doesn't need an instance to run on Please do stop me if you have any questions as well So virtual machines need disks. They need to be out of boot They need to have places where they can store data temporarily or permanently So the two offerings we have in terms of discs for the virtual machines are persistent disk and scratch disk Persistent discs live in the cloud. They're virtual virtual machine independent And some people are concerned about how you would use a that type of this to actually boot a virtual machine But our experience so far and feedback from people we we've been working with people companies like map are Have fed back that the persistent disk is as quick to boot as other offerings of a scratch offering some other Cloud vendors So you don't have to worry about persistent disk being slow to boot Also, these persistent disks are shareable. They can be attached and detached from virtual machines and they can be shared between virtual machines You can actually share them between a group of virtual machines in read-only mode So if you have a bunch of data on the disk You want to share between multiple machines. You can do that in read-only mode We also support snapshots So you can make snapshots of a disk state a particular time and then you can use that snapshot to actually create new persistent disks And again snapshots are project level resources They're not instance level resources that they live in the cloud and you can use them with anywhere within a project a Scratch disk what we used to call ephemeral disk is effectively a local disk local to the virtual machine It's tied to the life cycle the virtual machine lives and dies with it So it's basically for temporary data. You boot the machine from it store data on it temporarily once the virtual machine goes away That this will go away, but you can use it. You can add scratch disks to your machines You need more this space for a temporary Temporary results and during the demo. We're actually going to look at how we can create a scratch disk I've already mentioned the other cloud platform storage options cloud storage cloud SQL and cloud data store So we pretty much have everything covered in that respect The API and the tools so all of our tools all of our tools that we provide the tools that Jimmy and David demoed to you things like gcu till and the user interface. They all work via our API Now API is Jason over HTTP for the standard arrestful API the main resources the nouns are the Compute resources that we mentioned so far projects instances networks firewalls disks snapshots and so on the actions the verbs are standard HTTP verbs get post Post is used to create delete and we have custom verbs for updates and Really a combination of put and post for different different functions All of the authentication for the API is done by a tool, but we also have service accounts, which I'll mention later right, so Next thing how many people before this talk had actually heard of computing June Okay, so David had so like a Few of you how many people have actually used had hands-on and computing June and you one person David again You should work with Google you do what the Google so if you want to have computer engine It's in open preview now. It's that was announced at Google. I owe back in May So you can sign up for it now paid support is available for levels of support We have an SLA for all of the customers that use it and you can get started that particularly URL And the important things you need to remember from that URL are cloud and compute But if you go to cloud.google.com, you're probably going to get there anyway and The kitten the kitten wants computer engine before when it was in limited preview the kitten couldn't have computer engine But now I can You guys don't care about kittens. I do really wrong audience for kittens So custom customization options So a virtual machine the the image of a virtual machine getting a Debbie an image and it's still not on your instance It probably isn't really going to be enough for you You're not going to want to run a pure vanilla OS image So you're going to want to customize the image you're going to want to install software on it You're going to want to run services on it and provide configuration Install packages and so custom software from various sources and we're going to show an example of that later So one way the first way to pray to customize your image is to take a Google provided OS image and to customize it and Put that back into the cloud Put it back into computer engine so it's accessible for you another your provision of instances and we'll go through that process in a second Also, you can use startup scripts to dynamically configure your instances. So it may well be that you have a Base build and a base setup But you have start up scripts for various different use cases for their various different purposes And I start up scripts are kind of similar to rc.local that they install software boot time You can combine the first two options so you can have a broad base image that you use or multiple base images that you use for multiple machines and multiple bit VMs And you can then customize those individually for groups of VMs or for specific specific VMs using startup scripts and The final way is to use to build your own images using build debbie in cloud Which used to be called ec2 debbie and build amy. Am I like to say amy or do I say am I? Okay, I Like spelling but I never know where to spell things out or not like SQL SQL with me. I can't say sequel Well, when I used to work at some we were when I used to work at some we were slapped around for saying sequel because it was Just not done So ec2 debbie and build amy and that now supports both Amazon ec2 and computer engine images and Jimmy mentioned it during his talk So currently the versions we have of debbie and the stock images that we have when you install computer engine when you want to provision instances Squeeze 6.07 and we see 7.1 I did have sent us on this slide, but I thought it might offend you or so So creating custom images. I was going to walk through this, but I feel I'd like to spend more time Doing the interactive demo. So custom image Basically, what you have to do is create a new instance and you have to provide it with a service account Or you have to tell it to use a service account now the service account basically does authentication for you Normally the authentication flow we use is our off so if from a computer engine instance You wanted to access another service or another API you would have to authenticate against it and you would go for an all flow But in this case you can provide a service accounts Which actually stores a refresh token in its metadata and it will handle all of the authentication for you So in this case when we create the new instance, we're saying create a service account Give it the access storage for which means it has access to a Google cloud storage can do reads and writes and we're going to use cloud storage for storing our image Then you SSH into the instance and customize the image setup. So in this case, you'll be all installed custom software add packages do configuration start services that kind of thing and Then you're going to create an image table using image bundle.py a Python script that we supply and That will create a large image for you Will take some time to run depending on the level of configuration you've you've made on the image Then once you've done that upload the image file ball to Google cloud storage Where it's accessible for you to use with GCU till one of the tools that Jimmy showed you earlier To add the image back into compute engine It now becomes a project level resource for you to use within all of your project within your particular project projects are basically the Hours in a wall of the resources that you create So when you create virtual machines and networks and such like you do them at a project level and they're kind of isolated You can't share these resources between projects And then once you've done that the image should now be added To the list of images are available for you to build instances with and you can do gcu To list images to see the images that you have and it should be there and we'll have a look at that in the demo instance metadata So virtual machines need to know stuff about their environment. They need to have some context in which they're running So the metadata server provides that to the instance. It's basically a Dictionary of key value pairs. It has some basic information pushed into it at provision time Things like the hostname the image the zone that kind of thing But you can also use custom metadata and that's set by the API But all this stuff is read by the instance specifically so it gives the instance context about its environment It's accessible via the instance metadata server. So you just run HTTP colon metadata using curl or something like that to get information from it and It's useful for very small amounts of config data. You can push startup scripts into The metadata server but less than only less than 32k There's also project level metadata, which is accessible to all instances So you can actually set metadata at the project level and that will be propagated to all of the instances that you create a Very very brief example. So we set an environment variable MDS and just use curl to access the metadata role config Get those values back and also Because you may your software may be running on the summer unknown cloud And you might want to know what cloud it's running on in order to find out of your running on computer engine You can just ping metadata google.internal and that will if that comes back with a positive response and you know you're running on computer engine So you can obviously code that you don't need to use ping you can use some kind of IP ping or something like that And startup scripts so the whole point of talking about metadata was because startup scripts exist within metadata They're provided provisioned by the metadata server at provision time and they're very similar to RC.local. They run at boot time If you reboot the virtual machine, which it is possible to do Although you probably wouldn't do in more like it to delete the image and delete the instance and create a new one But if you reboot it, it will run again Example use it is to install packages and to start services But you can also bootstrap other compute resources. So in the two examples here a Startup script is used is created and then it's pushed into the metadata at the instance creation time So that starts start up script start up script Startup.sh will live in the metadata for that particular instance Sorry Okay Right, so now we're going to go into an interactive demo so Now interactive demos are normally places where you have a room for the corner cases So corner cases when your demos go wrong and normally everybody in the room is a corner case It always breaks. It's always going to break and hopefully it won't break so much So, let's go through this What we're going to do if you want to follow along if you have a laptop and so few of you have laptops But if you want to follow along, I'm going to do this myself, but you're perfectly welcome to follow along We're going to look for the most unusual objects in the universe Okay, we're going to look for weird galaxies and loavers and that kind of thing And so we're going to be using software That's going to be used with the large synoptic survey telescope Which is about to be built in Chile on a mountain somewhere in Chile And the plan is that the large synoptic survey telescope will image half of the sky the southern hemisphere every three days or so Over the course of 10 years, it's going to gather 60 petabytes worth of data And it's going to image 10 billion discrete stars and 10 billion discrete galaxies And the software we're going to use is part of a concerted effort to actually build highly scalable astronomical analysis tools The camera used with the LSST is a 3.2 gigapixel camera And I want one of those on my phone because Would I can match the size of the files? It's like 20 terabytes a day in this project will generate, which is like crazy So what we're going to do we have a thousands of images we can pull from These ones were taken by other cameras. Obviously the telescopes have not been built yet But what we can do is we can take images from a particular area And in this case they focus on galaxies where particular galaxies where we have images for And we can take individual images and we can overlay them and Then we can align them to make sure they're aligned for the correct coordinates And then we can build a very deep image of that particular area of the sky So it may be focused on a particular galaxy or some other interesting object But we'll get a very deep view of the sky and all of the all of the Objects that were in that particular frame or in that particular area will stand out very clearly as you can see in the second image Which is basically a composite of the first three or four there So they're prerequisites and make sure you had to follow it installed you need a browser and You'll need to have local storage enabled and JavaScript enabled and you'll also need a terminal and SSH so you guys being Debian hackers you're gonna have access to SSH and Ice weasel Is a great browser and it should work perfectly Jimmy's tried it and it does work Perfectly one thing you may get you may actually download the slides because you're gonna get a copy of the slides when you run this They may actually download them rather than actually view them in line Quick word about gcu till Jimmy and David Demonstrated the gcu till tool earlier. This is basically the computing gym power tool Allows you to access all aspects of computing gym apart from maybe some of the metadata Like who has access to the project that kind of thing project level resources It allows you to create virtual machines networks instance it discs pretty much everything you can do within computing gym and I've been trying to find out figure out where to put this slide because I wanted to introduce you to you till first But it might be more helpful for you to have a look at it later when you actually logged into a virtual machine So provision and managing virtual machines for this demo. We're gonna need some virtual machines that you can work with There's lots of third-party tools that are available to provision instances things like right scales scalar puppet chef They can all be used to revision instances dynamically provision them based on workloads But you can also use a script and this would just invoke gcu till to add and delete virtual machines And what we're using here is a script we created for this particular demo called cl.sh And I was going to run it live, but it probably take too long so I Decided not to But that's what you'll see we'll run the script and it will create 50 to 60 instances for us It takes four or five minutes to create that many instances Okay, so how many of you want to follow along? Okay, so the first thing you need to do is go to gce-demo.appspot.com in your browser and that spot.com is the Domain for app app engine applications if you don't have an a Google apps domain. That's where your App will live. The only time you create an app engine application. It will have a dot app spot.com domain So From the drop-down you see when you go to that website select debconf 2013 gce interactive demo and Then enter your name and click the start button Yeah No, you don't need to see the fgc. So basically the idea of this demo is to give you access to an instance that you can use So we wanted to give you a quick simple SSH SSH access to the virtual machine Local storage is required The script pushes the token into the local storage So Sure Anybody following along had any luck? Yeah, you get a PDF. So that's the rest of the slides So you can reference them locally you can once you have the slides you can follow along at your own pace Or you can just wait for me and follow along with me. How's it going Jimmy? Okay, so I'm going to move on to the next slide. So Once you're there, you'll find you have a copy of the slides. You'll either get them viewed in Displayed in line or you get them downloaded as a PDF file and At the top of the bar on that URL there's a link to participant info if you click on that It will give you some information about the actual virtual machine. So That will give you information. I'll do it myself That gives you information about the virtual machine. So the username Very very small username and password External IP address and an SSH connection that you can use SSH into the instance look okay So the participant info is there really just so you can SSH into the machine really easily And I was gonna do that here Yeah, you can follow that Yeah, I'll get back to it. So that's what everybody wanted to see so If you want to SSH into the machine just basically copy and paste the last line there GC code lab whatever version you have All right, it's the one you click on participant info. You need to click on participant info to get your individual They're up here participant info. Yeah, you got it Anybody struggling to find a participant info. Did you find it? Was that sorry? Okay So how many of you logged into the instance? Okay, all right, brilliant so next thing we're gonna do is The network seems to be struggling Hopefully your connectivity to your virtual machine will stay up on my name But the next thing to do is to actually add a scratch disk to your assigned virtual machine In the home directory to use that you logged in as you'll find there's two scripts One called ad scratch dot sh and one called get tile dot sh now They're basically there just for simplicity just simply by the process You should be able to cut and paste the information in the slides the copy the slides that you have to actually Create the disk and just set the permissions on the disk But if not just run the ad script Twiddle slash ad underscore scratch dot sh and that will Create a new scratch disk for you what we talked about earlier. We mentioned we have persistent disk and we have scratch disk This will create a new scratch disk for you Now they say if you get stuck just run the script Anybody looking at those slides because you have a copy of this information is on the slides that you have to Do you download it or do you? displayed in line The shortcut the shortcut is the easiest way to do it Yeah, it's like cutting and pasting from the PDF files is really difficult. It tends to remove hyphens and such like so I'm just basically running that now So if you type in DF minus H You'll see that you have quite a large about matters scratch disk available to you It's mounted on the session mentee slash scratch zero and it's about 1.8 terabytes in size And again, there's type the size of scratch disk available to you depends on the size of the instance that you're using so the next step is to actually assign yourself part of the universe and To do this, you're going to pull down something from my health service from from actually from Google cloud storage By connecting to a URL on another app engine application Can call compute codelab dot app spot dot com and what that's going to do is connect with Google cloud storage and pull down a Bunch of image URLs for you and some metadata And once you've done that you need to copy some files using GSU till again something At all that's a Jimmy mentioned in his talk GSU till allows you to interact with Google cloud storage So both of those steps are actually using Google cloud storage ones pull down this this URL of files and one to get the Files you need necessarily to actually run the next step And again, there's a shortcut at the bottom there Twiddle slash get tile.sh Twiddle get underscore tile.sh And you can run that and that will give you the tile assignment I'm going to do the same thing here and obviously We don't care about the Wi-Fi connectivity speed at this point. We're in the cloud. So that's pulling it down from The cloud onto your instance Okay, so the next thing you have to make sure you're in slash mnt slash scratch for the next step Scratch your sorry So make sure you see these are slash mnt slash scratch zero For the next step I'm going back to the slides. Hopefully it will connect quickly Another time. Yeah, sure Your IP address that's Right. Yeah, so I did that in the startup script so the startup script Actually pulls down metadata from this from what we talked about earlier. So the metadata has a external IP address Yeah, I'll show you I'm kind of worried about we'll run out of time Okay Basically, I wanted to make it simple Otherwise you would have had to actually make a call to give you a metadata server or to GCU tool get instance Actually get your IP address So I tried to make it simpler by doing some magic behind the scenes and putting that into the script So that makes life a lot simpler But then it's used for wrong here to actually Do that with a curl script the curl call And that's what a startup script looks like in this case, but there's various stuff. We've done we've simply by the login We've actually Pushed or set up the users. We've actually set up a user called gce codelab xx whatever the version you had We've also enabled password access and restarted SSH underscore config To make that work, but we've done various other stuff to set up the environment Going back to the sides very quickly so we can wrap this up Yeah, so you can use you can use Jason underscore pp to actually pretty print that Jason file So Jason underscore pp redirects the file into that and you will pretty print the file for you It gives you a list of URLs. They're all grouped by color They have a relationship with each other. It's quite a complicated process to actually generate the image and at the bottom there It's got some metadata which includes the name of the The name of the file that will be generated. I'll go very quickly show you that So you can see it So Jason underscore pp We were redirect Tyler Simon Jason and again make sure you're in mnt slash scratch zero and that's the list of files that it's going to look at and Also some metadata at the bottom The actual coordinates we're looking at and they name of the file that you will produce 41.png Some people are obviously gone further ahead because 41 is not the first image. So The next step is to run one of the scripts we downloaded earlier called make co add cloud PY and that's the command line string You had that in the sides again. You downloaded so Python make co add cloud PY Tile assignment json, which is a file. Do you download earlier and it's very important to include minus minus max images equals 15 Otherwise, you'll be here until this evening when the bar opens The bars open already probably So make sure you specify minus minus max images equals 15 otherwise it could take quite a while It's going to take three or four minutes anyway But I'm quite prepared to put up with that So if you can run that and see how you go and I'll do that myself Well, that's one and I'll put the slides back So basically what's that what I'm doing is actually analyzing a bunch of files and it's kind of limiting the number of files It's going to look at but it's looking for them to find regions that match with each other It needs a key file for this to work correctly. So it needs to find a key image that will Actually be used by the rest of the system to actually align all the other images If you limit to number 15 images, sometimes you don't always get that key image. So it breaks So if you do run, I've actually tried to make sure there's at least 37 or so of these Assignments that will work without going beyond 15 images but really you should be running without the minus minus max images and that will always work but We're 15 is a little bit more dubious, but you should be okay But it just takes a little bit wilder a little wilder run and it's downloading all of these 29 megabytes fits files And fits files if you want to look at the fits file itself. It's in TMP slash MNT slash scratch zero slash TMP and it's got some Ascii headers that provide information about the coordinates and such like for what the image came from and it has some binary data at the end of it How's everyone getting on? You're giving up yet. This is normally part of a larger code lab That we do we ran it for the first time at Google AIO and we normally run it in two hours So that kind of gives you some idea of how long we would normally spend on this. I was hoping it would work pretty quickly Yeah, you'll have your own virtual machine. We have 60 available. There were 60 people in there We could accommodate them. I could have created some more very easily, but we have 60 available So in this case, I decided to bump it up to four cores So you have a four core one just to make sure that it didn't take too long to do the processing The processing is reasonably intensive in terms of CPU usage But obviously the big part of the actual process is actually downloading the files themselves. So That's what's taking so long to actually run Originally, we had one core instances and I'll just say if you want to carry on Now you have the sides if you want to carry on and try this afterwards finish off if you don't get finished During this session, then you can do it afterwards Yep Okay, so you got questions anybody More questions and answers should be just thing So one thing I will say about a computer engineers that we have credit packs for startups So anybody who run the startup and wants to get started with computer engine Hopefully by next week, we'll have credit packs that you can use so we can give you to get up on money It's not free to use currently at the moment. You have to sign up and provide your credit card to get started, but As I say next week, we should have credit offerings for startups to access computer engine We have them already for App Engine Taking too long That really depends what you want to do if you want to build a cluster of a thousand instances then Obviously that's going to cost you quite a bit Again, like Amazon, you only pay for instances that are running and for resources are actually available and running So it really depends on what your use case is Just play around I can't give you any exact examples. Jimmy, do you think of any of your head? I would say just pull up the pricing web page. It's actually reasonably cheap I don't know the current comparison to the recently changed Amazon prices of a couple of weeks ago But it's it's very affordable to just play around certainly on that price scale. Yeah So Okay, so all thanks guys, thanks for Trying to follow along. Hopefully it would have taken less time, but we kind of got stuck. I expected this to you, but Now you can carry on using it you can carry on finish if you want to I'm not going to shut down the virtual machines for Until later on this afternoon. So those virtual machines are still available. You can just SSH into them and you'll be fine Again, what's wrong? Oh We're not working again, there we go. So now we got a 15 minute break and then lunch So everybody give it up for Mary and her great excuse me Mandy. I don't know why I got you married Mandy and her great demonstration here