 As we've been said, welcome to Amazon. This is the Amazon Web Service office here in Singapore. Who here is already using Amazon Web Services? Wow, 50% or more. All right, so has anybody given Code Comet a spin yet? All right, so good first intro, right? Something useful for you guys. So this talk was actually a Code Comet, Code Deploy, and Code Pipeline. I have 15 minutes. So what I'm going to do is spend most of my time on Code Comet, but I'll try to explain how these services fit together, all right? OK, this turns on. All right, so let's talk a little bit about continuous delivery and what role various services play in continuous delivery. Then we'll look at the services, and then I'll spend some time on Code Comet. So this is what a continuous delivery pipeline might look like. So this is a very busy slide. And a lot is going on there because typically, yes, in a continuous delivery pipeline, a lot does go on there, right? You are quite used to writing up your application code and putting it into version control, right? And then you may be also responsible for setting up a build server and making sure your builds run. Unit tests are executed, maybe a static code analysis, maybe an integration test. And then you or somebody makes a deployment happen on production, right? The way we think about systems at AWS is we think that infrastructure is also code these days. With the cloud, infrastructure is just a command line away. You just say, I want to start a new machine and deploy my code. Those are actually just pieces of code. Start machine, this type in this region with these properties, install Apache or Nginx or whatever, deploy my code, pull from here, version this. Everything is code. So why not treat it like code? So you can basically check in everything, your code, your infrastructure stacks, supporting services, third-party APIs that you are using, their descriptions, their access control. Put it all in version control, which is here. And then a build server kicks in. Does your build, if the build is successful, it probably uploads and packages file like a zip file or a JAR file or something else to where you keep your artifacts, which are the things that you'll install. The artifacts will be used for QA slash testing, maybe load testing. And when everything is okay, then they will go into production. That's what's going on. So stuff coming in of operations happening and stuff going out. And you know what? Stuff goes out not to just a production. Stuff goes out to a QA environment, to a development environment, right? So they're like mini-production. You are going to do testing on that. This is what most of us do. It's also interesting to get some data out of all this process, monitor the process. Once you measure something, you can try and improve it. So you could, for example, have code metrics and see over time how the complexity of the code has been evolving. You could have code coverage metrics and you could see over time, over various releases, how have you been improving your code coverage? Is it increasing, decreasing? Those are indicators of code quality, right? Some numbers. You could use the number of check-ins per day to measure how agile you are, right? That could be an input to your process. Things like that. Which means yet another tool somewhere there, right? So that's the continuous delivery pipeline. Let's focus on a few services that we could use to get up and running quicker. The code commit is a service which is just basically managed git. Code pipeline is a service where you can describe workflows. The previous multiple steps that I described going to various environments, various steps happening. It helps you manage that. And code deploy is a tool that will make it easier for you to deploy a final, shiny artifact onto your production or development or load test environments, right? So since most of the time it's going to be spent on code commit, I'm going to give you a quick overview of code pipeline and code deploy first, so let's... But this is how these services fit into the big picture. So all the way from code to deployment and after deployment, deployment and provisioning means managing the infrastructure, deployment is deploying your application, monitoring is how well is your application doing. There are various services. So obviously code commit is related to, you know, where you code, you check in. Code pipeline helps you take that code and run builds and tests and tick off deployments. Now the deployments need infrastructure. So either you are standing up a machine yourself or you're deploying, you created an environment and you just keep deploying to that again and again. All your environment is dynamic. Machines keep coming and going. So there are services to help automate that, right? If you have a deploy environment which can grow according to scale, you might like to use Elastic Beanstalk. If you are comfortable with tools like Puppet or Chef, you might like to use OpsWorks to do the installation, automation of configuration and stuff. And on either of these environments or your own custom environment, you can use code deployers whose one single function is to install your shiny new software reliably. All right. And if you log into the AWS Management Console, you will see this cluster of services right there, these three services, and the Beanstalk of service they're also there. Right, so this is your typical workflow. You have some code version running in build. Currently, you know, you've just checked it in and it's failing the build. You have some code which passes the build and you call it a beta and you deployed somewhere and people are hammering on it. Maybe QA are testing it. In AWS, there's yet another state, gamma, but you may or may not have it, think of it as pre-production. Finally goes to production. You may have just one production environment. If you have, you are very lucky. If you are not so lucky, you will have several and therefore you will be thinking all the time which version is running where. Right. The thing is to move code from source to all the way here, there's a workflow, there's a flowchart which is unique to your organization. You may have more or less stages than this. You may have more or less environments than this. Some of the, you may have if say all the unit test pass you may want to automatically promote that to say QA. As soon as my unit test pass I will push that deployment to QA and that should be automatic. But you may not want things from QA to go to production automatically. You may want a human switch there, right? So your flowchart could look different depending on your circumstances. So what pipeline is a service where you can do this graphically and you can have automated promotion of code from one stage to another depending on conditions and you can have gated ones where it won't automatically go ahead and just show a green button ready to go unless somebody clicks on it. It doesn't do the build, it doesn't do the various steps by itself, what it does is it plugs into other systems. It will integrate with things like Jenkins or cruise control or what have you to plug into any most version control systems out there. So you just define the workflow. You say pull from here, push to Jenkins, wait for this output. If this output, it's good to go. If the output is something else, it's the build has failed. So trigger off another workflow, maybe send an email saying something happened, right? And the reason you might want to use this service and should of say a shell script is that when you try to script all of this up, where's that script gonna run? That itself becomes a reliability problem. It's only on your laptop, you're off on holiday, that doesn't run. If it's on some server forgotten somewhere, that server goes bonkers and your build system has failed. The second thing is it has concept of retries and it can handle transient failure. This workflow is just a bit more robust. It protects you from having to, well, I've built pipelines before where nothing was wrong except the script which was trying to kick off Jenkins for an integration test had a bug. My code didn't have a bug. Jenkins didn't have a bug. The script was just pulling. Unit test pass, let's do integration test. That thing had a bug, right? So it will just show up here instead of hunting around in logs and stuff. You'll know exactly which stage is failing and what are the errors. All right, so once your artifact is ready, you can use code deploy. We say it applies to a fleet of EC2 instances, but it can actually deploy to any server, on or outside AWS. How it works is you basically have an agent, a software agent that we provide. You install it on whatever machines. On AWS, you have images with this agent already there so you don't have to install it. You just bring up a machine. You just tell it which pipeline in code deploy to connect to and it will then, when after that code deploy takes over. In code deploy, you send a command like deploy this release to this bunch of servers. That bunch of servers is your fleet. So that's fairly uninteresting, right? You can do it with the whole bunch of tools already, with Capistrano, with just SSH, with Fabric, with Ansible, with Chef, Puppet, whatever. So the interesting thing about code deploy is is when you reach scale, there are some things to worry about. Imagine you have more, let's say four or more web servers running version one of the software. And you wanna deploy version 1.1. So you don't wanna bring all of them down at the same time. You want to do a rolling deployment. So nothing could be easier, right? Just write a for loop with a sleep deploy to server for I on servers deploy, wait, deploy, wait. What happens is for some reason, when you were installing on server one, something went wrong, there was a bug, that server never came back up. You want to roll back, right? So now there is a if, then else, somewhere there, right? What happens if the first time you deploy version 1.1, there are some database upgrades to be done and it takes longer than usual. And once the database upgrade is done, then two, three, and four can be deployed quickly. If you wanna try and build that logic, script is becoming more complicated. Things like this are common when you have large deployments, when you are doing things at scale. Code deploy is a service which is designed to help you do those deployments at scale. So if you have a number of servers on which you're deploying, it's getting out of hand. One-shot deploy on everything is not cutting it. That's when you consider code cutting. So this thing came out of the deployments done in AWS. There's an internal tool called Apollo, which Amazon.com built for its own use. That thing has more than a million servers under its control. An average deployment of new release goes, the average size of the fleet is about 10,000. The peak is 30,000. So at one point of time, it could be deploying to 10,000 machines. Now, imagine all those machines are trying to pull and then sending feedback over this stage pass, that stage pass, that broke, this happened. It will retry. It has the logic to retry. It has the logic to do rolling deployments. It has the logic to say, install my software on 10% of my fleet at a time. It can take, if you have multiple web servers, you have a load balancer. It has the built-in capability of taking the web server away from the load balancer, then doing the deployment, and waiting for it to become healthy, then attaching it back. Makes those kind of things easier for you. If you're deploying to two servers, maybe overkill, right? But very cheap, so you can always try. All right. Not gonna go into that. I'm gonna jump into code coming, right? In AWS, all right, I should say a few things about code coming. So code commit is managed git, but you already have many managed git services, right? Most popular being GitHub. So why would you ever want to use code commit? The code commit is addressing an audience that is slightly different from GitHub. It's targeted at organizational code repositories or enterprise code repositories. So if you're working for a large company and you are using a Git repo, there are a few requirements. Those repos are never public. They're private. Secondly, you need to have some kind of access control who can do what, all right? Usually you don't have a free for all unless you are a very self-contained small team. There may be most engine authentication requirements. Most likely the organization, if you are running this company, if you are the IT manager of this company, you already have an authentication system in the company. You may have Active Directory, you may have some kind of SAML-based or OpenID-based authentication system, which you're already using within the organization. It would make sense if your version control was also integrated with that. So people use the same login that they were using to login to their laptops or to the email system. If they use the same login to login to the version control or do check-ins, then everything would be connected, right? Right, so that's the problem code commit is trying to solve. It's not trying to be the best browser of code browser out there. It's not meant for a single person collaboration. You send a pull request or something like that. It's, or clone a repo, start your own. That's not what it's trying to do. It's trying to be a GitHub, but for an enterprise for the internal use. Or if you're a five-person developer, five developer company, working on your next big idea, don't want your repo to be public right now. You are collaborating with external people. You're outsourcing bits and pieces. You need tight control. Who can do exactly what? Then this might be for you, right? Also make some assumptions. One, you probably already know of a perfectly good code browser. So it doesn't supply you with one. So just use a GitHub. It doesn't give you a GitHub desktop or some shiny client like that. It assumes that you already have a Git client that you're perfectly activate, that you probably have standardized on. So you're not gonna just, you don't really, you didn't come here to get a new Git client. You came here to just have a secure repo. All right, so how does it do that? So authentication system is based on AWS identity and access management framework. So when you create an AWS account, you can also create users and groups and give them permissions. So all authentication is that. With us, as an example, you may have a repository admin and a repository user. Repository user can do Git pull. They cannot do a creator repository. They can't create a new repo. Maybe they don't have permission. You may have a power user and a non-power user. The power user could create and merge branches or delete branches, but a normal user can't really create a branch, for example. All right, so those kind of permissions. The content is stored in Amazon, simple storage service. So this is a service which is, which comes with very high durability and availability. It's, so your content is actually redundant across three data centers. You're never ever gonna lose your files. The durability is something crazy like 11 nines. So very highly durable. On the back end, the content is all encrypted. So why is that important? Maybe if you're working on an open source project, it doesn't matter. But if you are developing code for a financed customer, if you're developing code for some enterprise customer, they want tight control over intellectual property. One of the things they ask you is, how are you ensuring that this code is protected, that this code is secure? All right, so one easy way to say is, you know, it's here on Amazon S3. This thing has so many security certifications. It's one of the most secure public clouds. And by the way, all the code is always encrypted. All the metadata is stored in a fast, no-SQL managed database service called DynamoDB. The way it's designed is that search through metadata is always fast regardless of how large your repository gets. And yeah, the key that managed in us by a service called KMS, which is also a supported, sorry, AWS managed service, so you don't have to do anything in there. Yeah, so very highly durable. The code is kept securely encrypted in multiple data centers all the time. Okay, permissions, so you have lots of control. Code coming comes with three template, policies, one is full access, which means create repository, delete a repository, do anything with the repository. One is power user, so you can create a repository, you can branch, you can do whatever you can delete it. You can do just about everything except delete the repository. Pretty good, right? So you don't accidentally end up deleting. And read-only, so you may even give read-only access. And you can have your own custom policies, which, you know, mix and match whatever permissions you like. It supports, because it integrates with OpenID or SAML or Active Directory, you can have multi-factor authentication. I don't know who would want to use that, but in some environments, finance and security, those kind of environments, you need tight control to modify code, right? So you do have this dongle, logging, and then you can check in. Right, you can also generate temporary access credentials and give somebody access to your repo, even maybe read-only access to your repo for a short amount of time. And then after that, the credentials will automatically expire. Nobody can read. Encryption in transit uses the normal Git protocol, right? HTTPS and SSH. It doesn't use the baseline Git, it doesn't use just HTTP, uses HTTPS only or SSH. So the data is always encrypted when it's being transferred. And I already mentioned when it's stored, it's encrypted. Authentication for HTTPS, the AWS IAM credentials are used. So no username password, there's something IAM gives you. Code commit provides you with a Git credential helper which helps Git use the IAM credentials to authenticate. You can also use SSH if that's what you like. You can create your own SSH key pair or use your existing SSH key pair and upload to the public, just like you do in GitHub, right? Pretty much. All right, let's look at this. I'm using the command line tools, but you can do all of this using the web-based GUI, but that doesn't look geeky enough. No, actually I was trying to take screenshots, all right, and then becoming too big, too small, can't see, this is much clearer. All right, step one, I'm a super user. I own this AWS account, so I create an account called repo admins, I say. I am create user repo admins, I get a repo admin. Then I said, we give you built-in policies or we can create our own, so I'm just gonna use built-in policies. So I'm saying, I am list policies with this name AWS code commitful access. I get a policy, I just look at this long string here, this ARN, that's Amazon resource name, just a opaque string, it doesn't matter. I'm gonna use this here, so I'm saying, attach to repo admins this policy, this thing. So suddenly this repo admin user has those permissions, right? Cool, so I got a repo admin. Oh yeah, I need to get some credentials, how's this guy gonna log in, right? So I'm just going to say, add up this ARN, create access key, and it creates an access key, the access key ID is this. This is shared, this is kind of a username. This is your secret access key, and this is never ever shared. It shouldn't leave your laptop, right? But the great, good news is that should you accidentally leak this key, you can go back to AWS IN and quickly disable that. Just disable it, so nobody can do anything with that key. Can't tell you how often this happens that people are checking in code on GitHub and they accidentally check in code with their access key and secret key embedded in it. Property files or in the headers or somewhere, right? Maybe just demo code. And people are scanning GitHub and Bitpocket and looking for this pattern, access key ID, in various formats. And if you accidentally check in your keys, they're boom. The next thing is somebody has access to your AWS account, even just through the CLI, and they have launched 10 dozens of servers all doing Bitcoin mining or sending spam. It happens so much that AWS has also written a scanning tool and it sends you an email saying, we have detected this. All right, we had to do that, because so what happens, right? You wake up and boom, you get a huge bill and then why, last week my bill was, last month my bill was 10 bucks, now it's 9,000 bucks. How did that happen? And then you log into AWS account that you normally don't use very much and you see all these machines where did they come from? And then you call AWS support. So, you know, all right, so yeah, this thing is crucial, so you never leak it, right? If you do leak it, you can quickly go to the AWS console, disable it, generate a new one and figure out how that got leaked and never do it again. All right, okay, so I got the keys, right? Credit access key and secret key. By the way, this thing is not stored on AWS. There's only one time you can download it and this is now, after that you can't get that. Is it okay to be wrapped up soon? Yeah, let me wrap up. So, you put the key in your local AWS config and credentials. You, and let me skip that. You create a repository. You can get the details of the repository. This is a critical bit, Git config, credential helper. So you put this command in Git config, this thing. This is what Git uses to use your IAM credentials to authenticate. So once you have that, now you can just go ahead and then you can just clone that repo and then start Git push, Git push. It will just work, right? Right, some things are missing, kind of the biggest thing is probably no Git hooks. So once you check in, you can't trigger a hook just yet. So you have to depend on polling, that's it. And the other thing to take note of, this code commit is available in the AWS US East region, but it's Git, right? So it's not a big problem. You are anyway pushing too far away with those. All right, I'm done. That's pretty much what I had to say about code commit. You can visit the application management blog for interesting tidbits about application management, deployment, those kind of issues. And I have an announcement to make. We are, the reason I'm speaking here is because we are missing a technology evangelist who normally speaks at user group events and they're looking for one. So if you're interested, drop us a line, right? You know where to find this. All right, thank you. Thank you for your time. So our next will be Jason. Jason around.