 Setting up Google Cloud Platform is a breeze. You have a variety of options that we'll go through in here. Before we get started, let's talk about signing up for Google Cloud Platform. Google Cloud Platform is a public cloud, so nearly anyone with a credit card can sign up and begin using services pretty easily. You should note that a credit card isn't actually required to initially sign up. However, continued use does require one. But that raises an important point. Google Cloud Platform usage may incur billing. From time to time, Google Cloud Platform offers a free tier of usage and or credits for new users. These offers change, so you should check on the latest availability available on Google Cloud Platform's website. These credits or free tiers do come with limits. Sometimes a certain amount of use per month is allowed of a certain type, or sometimes a certain dollar amount is just credited to your account when you sign up and you're free to use it on any variety of services until it's exhausted. Regardless, outside of these limits, whatever they may be, or if these free tiers are not available, usage will most certainly result in charges. We'll go over how to look at what charges are available and maybe even set up some alerts to ensure costs don't get out of hand. Make sure you know what you're using and have the authorization to bill a credit card before going any further. Let's walk through signing up for Google Cloud Platform. Let's start our Google Cloud Platform adventure on, ironically, Google. Let's search for Google Cloud Signup, since the URL might change after we publish this course. I think it's a safe bet to say that on Google's own search, you're probably going to get either an ad or one of the first results will be the right one. Let's choose the Google Cloud Platform free tier. A free extended trial, $300 free credit and always free, seems to be the offer right now. Your mileage may vary based on the time and click the try it free or sign up button. If you have an existing Gmail account, feel free to use that. You can also click create account if you don't have a Gmail or you'd like to create something separate. Whatever email that you use, make sure that you have access to it. It's going to be important because you'll have to verify it later and you'll receive important notifications at that email address. If you chose to sign in with an existing Gmail account, no worries, you won't have to go through this process if you've already verified a phone number. You may be presented to verify a phone number or your email address if your existing Google account doesn't have those erifications in place. Whether you created a new account or using an existing one, once you've completed these verifications, check your email and your text messages and you'll be able to access the Google Cloud dashboard with the username and password you just established. Now that we've signed up for Google Cloud Platform, we have a variety of options to access the tools available to us. First and most ubiquitous, the GCP dashboard is available in any web browser. This includes an innovative cloud shell that we mentioned earlier, an interactive command line interface that handles the installation of software and access to the command line software all through your web browser on a remote machine that Google runs and hosts for you transparently at no charge, or at least no additional charge to the resources you already have running. This is one of my favorite ways of accessing my resources in Google Cloud Platform as any web browser and any machine in the world with simply accessing and logging in to my account, I have access to a powerful command line interface. iOS and Android native applications are also available on their respective platforms as app stores. The features and functions available in these native applications may vary. They're typically a subset of what's available on the web browser and using the cloud shell interactive CLI. However, notifications and other native first application features are available in these that you may find useful in addition to using the web browser and command line shells. Finally, Google offers the cloud shell command line toolkit very similar and exactly the same code as what is provided in the cloud shell interactive CLI for installation on your machine, your server, or other local hardware. This is actually the exact same code as provided in the cloud shell simply delivered to a different mechanism. Let's return to the web browser that we use to sign up for Google Cloud. If you don't have it, don't worry. Open a new web browser and go to console.cloud.google.com and login with the credentials that you created. Once you've finished your sign up, you should be greeted by a screen similar to this. If you've lost or closed the web browser, you can get back to this point by going to console.cloud.google.com and signing in with the username and password that you used. Let's take a tour of the dashboard. You can click tour console and Google will walk you through the important parts of the console. Let's do that. A quick five minute walkthrough will guide you through the high level concepts of resources, projects, and activity logs, searching, and how to get support. Feel free to walk through the console tour and resume this tutorial when it's complete. For now, we'll click cancel tutorial. We'll see you back here if you choose to go through the console tour on your own. One of the important central features of Google Cloud Platform is the project. On the top bar, you'll notice my first project is already created for you. By clicking on this, you are able to see or create additional projects. Projects are logically separate entities which group together cloud platform resources. They also allow you to consolidate billing for one set of resources against a certain account. It's recommended that you use projects to separate things which are, well, separate projects. This allows you to do accounting, billing, and access control in a consolidated way for things that should stay together and things that should stay separate. Throughout this course, we'll just use the My First project that was created automatically by Google. You'll notice it has a name, which is a relatively human-friendly name, and it has an ID. This ID is used across Google services when you're referencing the project to create resources or search resources or connect to resources in. It's this ID that is used, not the human-friendly name. By accessing the product and services menu in the upper left-hand corner, you're able to look at specific items within Google Cloud. Let's click Home. This is typically where you'll end up when you log in from now on. It gives you an overview of the project information, and it'll give you an overview of the estimated charges for the month. This is a convenient way to make sure that you don't have anything running that's accruing charges that you may not be aware of or that you don't actually want to use. It also gives you some information about usage. This is customizable. We'll go into that a little bit later. In the left-hand menu, you can see access to a variety of categories. Google's compute services, storage services, network services, Stackdriver, which is their monitoring, logging, and tracing offering, and other ancillary tools, and their big data and machine learning offerings. We'll go through each one of these in a lecture later in the course. For now, just know that this left-hand bar is your primary means of accessing these services. You can also use the search function to search for a given tool, such as compute. Searching from compute allows us to add a VM instance, a shortcut to an actual action, a view of looking at our current instances, or compute engine, which is the family of actions for VMs. One important thing that we should go over is identity and access management. We've created our account here. Our account allows us full access to everything within our Google Platform account and our projects that we create. We can allow other people, members of our team, etc., to access our project as well. We also may want to create accounts for eventual use for service machines, for automation, if some process needs to access certain resources in our account. All of that is done through what's called IAM and admin. IAM stands for Identity Access Management. We'll go through this in depth later in the course, but for now, just know that your login that you're logged in with right now are the keys to your kingdom. Keep them secure, and sharing them with anyone you don't fully trust is not a good idea. Instead, using IAM and admin, you're able to create and invite other users to your account to collaborate with you and to share on certain terms that you can define. This allows you to create them an account that defines what they can and can't do. It also allows you to create a service account. A service account is an account that may be used by a process or an application which allows it only certain access. So even if those security keys are compromised, only limited information or billing information is compromised and the amount of damage that could be done with that leak is minimized based on the permissions that you've assigned that account. Let's look at IAM. Using IAM, we're able to invite people to collaborate on our project. Remember, we're inviting them to our project, not to our whole account. By clicking Add, we're able to add new members. In this case, I'm going to add myself at a different email address. I can choose a role. I can make them a project administrator, editor, owner, or any other number of services and I can only let them access those services. In this case, I'll make it so that I can access this project's Compute Engine resources only. Click Save and email has been sent to that email address and logging in with that account will now only allow me administrative access to compute resources within my project. This is a very useful tool for small and large teams alike and removes the risk in having to share, login, or billing information. Another important area to look at before we get started is the billing area. You noticed on the home page, we were overall spending. At this point, it was zero. By clicking on the billing tab in the navigation menu, we're able to access a detailed breakdown of what our bill is when we're spending our money. In this case, we can also go to budgets and alerts and set up a variety of spending alerts. Let's create a budget to help us avoid a surprise. Let's call our budget Test Budget. It'll be connected to and anytime it goes over $10 including a credit we'll make sure that it sends us an alert. Let's click Save and now anytime that our budget is exceeded and we hit 50, 90, and 100% of our budget we'll be notified. This is a great way to make sure that you're not spending too much on a given project or you're not spending too much on a given billing account. We'll be able to set up more rules and we'll get into that and you can consult the documentation as you need. Let's examine a unique feature to Google Cloud, the Cloud Shell. In lieu of having to install software on your local computer, Google provides you with the ability to access a robust command line interface directly through the web browser. The way this works is a cloud container is created on a system that Google manages at no additional cost to you. This cloud container allows you to access the Cloud Google Cloud Shell. The Google Cloud Shell is a set of utilities you can download and run on any machine, but in this case they're downloaded and run on a remote server and you're giving access to them through a web browser. It's a real Linux environment and you're able to do a variety of tasks in it. It has support for a variety of languages and it's configured automatically for using Google Cloud with popular tools such as text editors and other tasks and other tools you'll need to complete certain tasks. Let's go ahead and click Start Cloud Shell. While it's starting up, we should note that one of the important features of this is that it maintains its state. Even if you close this web browser, come back a month later on the other side of the world logged into your account. That state will have been preserved for you. As you can see, we're logged into our Cloud Shell now. We can type help about commands that we can use and as we go through the series of lectures, we will continue to use the Cloud Shell for items which may not be as easy to do in the user interface using the web browser. Some things you can only accomplish by using the command line and we'll leverage the Cloud Shell instead of asking you to install software on your local machine. If you'd like to install software in your local machine, the exact code that runs in this Google Cloud Shell, Google Cloud Platform provides documentation that is specific to Windows, Linux, and Mac platforms on how to do this. They also describe how to authenticate and connect the software that you install on your computer to your Google Cloud account. Since Google offers a pre-maintained Cloud Shell that is exactly the same as the software that you would download on your machine, we won't go over that and we'll encourage you to use the Cloud Shell instead. Not only is it maintained and more secure, but it's a whole lot more convenient and consistent. Now that you have accessed the Google Cloud Dashboard and are hopefully familiar with it and have accessed the Google Cloud Shell, let's actually dive in to the Google Cloud Platform itself. In the following lectures, we'll look at the exciting features of each major category of Google Cloud Platform services using the account that we just created. A very long way since computers were just beige boxes sitting on your desk. Instead, computing in the cloud offers you the ability not only to set up your own machine as a bare virtual machine over which you have complete control and configuration, but a variety of services that run at a higher level. A variety of services that give you ease and scalability, maintainability, and perhaps even a different economic model than simply renting time on a full machine. In the next few lectures, we're going to go in-depth on each of Google Cloud Platform's cloud compute offerings. Compute services, as we define them, are focused on executing some type of workload. In the context of Google's cloud, compute is focused on actual processing of data as opposed to storing, transferring it, etc. That doesn't mean that the compute services don't do some of this. It's just that their primary goal is to process data in a way that is most effective for the user. Whether it's a fully-managed function-level execution in Google Cloud Functions, where the end user need not worry about what server it runs on and where, just that the code is written and it executes quickly, to single-instance VMs, where the end user is responsible for everything from operating system configuration to booting to network configuration, Google has a variety of options that should fit nearly anyone's needs. Google's current compute offerings include Compute Engine, Kubernetes Engine, App Engine, and Cloud Functions. We'll go over each in its own lecture, so don't worry about grasping every last detail of each one of these right now. Instead, just to understand that there are four distinct offerings that have four distinct advantages and disadvantages. And remember, they're not mutually exclusive. You can use one with the other. It's about picking the right tool for the job. Compute Engine offers virtual machines. That means you have full control over the specific configuration and, with that control, comes a large responsibility for managing nearly every aspect of how that virtual computer operates. Next is Kubernetes Engine. Kubernetes Engine uses Compute Engine instances and manages them for you to run a Docker container workload, or a variety of Docker container workloads. It is a hosted Kubernetes offering which runs on top of Compute Engine instances. Kubernetes simply provides you a framework for running complicated Docker applications. Google App Engine is a fully managed serverless application platform. It handles configuration for you and has very opinionated patterns as to how to accomplish certain things, as to how the application is developed, packaged, and deployed. In exchange for this level of specificity, Google App Engine removes the need to set up your own server, manage it, and scale it. As long as the application is packaged in a format that App Engine understands and written in a language it supports, App Engine is able to provide services that don't require a server to be provisioned to run a given application. And finally, Cloud Functions are single-purpose, standalone modules of code that run in response to a given event. This event can be an HTTP request, a message put on a messaging queue, or the placement of a file in a cloud storage bucket. These functions are designed and required to run relatively quickly, usually less than about five minutes, so no long-running processes here, certainly not an appropriate offering for something that has to run for hours and hours, but on the other hand, you get down to the microsecond billing, you get no maintenance, just code, and small units of execution that are easily understood and maintained and deployed. As you can see here, these four offerings, while they interact and all run on top of a given computer, your need to manage that computer and how you're billed for it varies. Again, don't worry if you don't understand every single aspect of each one of these four. They'll have an independent lecture, and which will go into the details of each. Instead, just understand that picking an offering really relies on a robust and well-designed application architecture that leverages the right tool for the job. The capabilities of each product offering vary, and so does the pricing model, what they cost and what you're billed for varies, so the various needs of an application, both on the technical side and on the economics side, will likely drive your choice. We'll go deeper on to each one of these in the coming lectures. I'll also note we'll take a slight detour to discuss something called Cloud PubSub. It's not a compute offering per se, but it relates to how messages are sent and received by other systems. So knowledge of this is important to understand specifically the app engines and cloud functions products because those two offerings leverage Cloud PubSub or can leverage Cloud PubSub in certain situations and configurations in response to events that come in on those message queues to drive a computing event. Don't worry if that doesn't make sense. We'll go over some real-world examples when we discuss app engine and cloud functions. For now, just understand that the four current services in Google's cloud computing quiver have a variety of different applications and capabilities, and in some situations they work together to create a robust application architecture. But our running on larger systems stored in Google's data centers around the world. These machines run on very large shared hardware systems running something called a hypervisor. A hypervisor allows Google to isolate resources that function just like a single machine and sell them to you on a per second basis. These machines can access networks, storage, other services on Google Cloud, including items like GPUs for machine learning applications, solid state disks for storage of data, and even private networks to allow you to create applications in hybrid cloud environments. Much like any other virtual machine offering on Amazon, Microsoft, or other clouds, virtual machines provide you the most flexibility in defining how your workload is deployed, but they also require the most intervention and administration. Similar with industry standards, Compute Engine is built based on how much CPU and RAM you use on a per second basis. The unit of measure for CPU, or the central processing unit, is the VCPU. It's equivalent to a single hardware hyper thread on the underlying hardware. What does that mean? Well, processors are built to be able to process information in threads. When you're buying a VCPU, you are buying the right to use one single hardware thread on one processor, on a machine, hosted by Google, in the geography of your choice. You can purchase multiple VCPUs to connect to the same virtual machine. Similarly, RAM is measured in gigabytes, just like your machine on your desk, or that you would physically purchase for your server room. Regardless of how much CPU or RAM that you choose for a virtual machine, you can use committed use or sustained use discounts. These discounts are automatically applied. Sustained use discounts provide you up to a 30% discount if you use a virtual machine for the entirety of a month. There are lower tier discounts available for less utilization throughout the period of a month. Committed use discounts are a little different. Committed use discounts is a commitment upfront for a given period of time that you would use a virtual machine for that period of time. For example, if I knew that I wouldn't have to run a machine, for example, running a database, for three years, I could tell Google that I know I'm going to use at least four VCPUs and 32 gigabytes of memory for three years, and they would offer me a deeply discounted rate. This is billed per month, but the commitment is valid for the full year, whether I use the machine or not, and I will still be billed for that. Consult the pricing page on Google Cloud Platform's website for more information as to the terms and conditions for committed use discounts and sustained use discounts. Regardless, if you've purchased a commitment and you start a virtual machine, if it's applicable under the terms and conditions, Google will automatically apply that discount. Similar, sustained use discounts will automatically be calculated once you hit the right threshold, and you can view them on your statement at the end of the month. In addition to the VCPU and RAM costs, you will also pay for any persistent or local storage that is attached to your machine. I mentioned this separately because VCPU and RAM is tied to a virtual machine. On the other hand, persistent disks, at least, exist separate from a machine. You can take one persistent disk, attach it to a given virtual machine, remove it, and attach it to another. On the other hand, if you were to delete a virtual machine, the VCPU and RAM associated with that go with it as well. So from a billing perspective, they're slightly different. Beyond the physical specifications of how much CPU and RAM a given machine has, one of the most important characteristics is the software that operates it, both the operating system and the software stack running on top of that operating system. Your choices for virtual machines on Google Compute Engine are diverse and open to extension. At the most basic level, Google Cloud Platform provides public images that offer hundreds of options and combinations of compatible Linux and Windows offerings and versions, both in free and premium versions. Everything from Red Hat Linux, Susie Linux, Debian, and Ubuntu are supported, along with many other flavors of Linux. Windows and its variety of versions and distributions are also supported, as well as third-party vendors who offer free and paid images that include a base operating system, as well as their software stack, such as the popular corporate software system SAP. You also have the option to build private images that allow you to build your own operating system and software stack to deployment to your VMs. This provides you the ability to bring software that others or Google may not be offering, or provides you the ability to take software provided by Google and others at a later of customization and configuration so you can launch virtual machines without having to reconfigure them each time. A facility for building these images based off of virtual box images or other virtualization technologies also exists in a form of third-party tools as well. In addition to the traditional pattern of bringing up a virtual machine from an image and making configuration changes directly onto the system, Google Cloud Platform provides you with the option to use a Docker container as a form of base image for your virtual machine. The way this works is Compute Engine will take your specification as to what container image that you'd like to pull from either the public Docker repository or a private Docker repository that you've hosted and run it on that virtual machine. That virtual machine runs a Google-maintained image of Container OS. Container OS is a minimal, highly supported version of Linux that allows you and is optimized to run containers. It tries to get out of the way to allow the container to have maximal use of the underlying resources. The VM boots under core OS and then starts the Docker image as you've configured it. Discs can be attached just like any other virtual machine and mapped to Docker volumes in the specified configuration. Just like with actual hardware machines, there's no one-size-fits-all solution for a variety of workloads. Some workloads require a large amount of CPU and are relatively small in memory footprint. Some have a very high memory footprint, but the compute needs aren't nearly as pressing. Some workloads might require a variety of high CPU and high memory availability. Other workloads may require a very fast local disk, but not very much CPU or memory. Whatever your needs and whatever permutations exist, a variety of predefined machine types exist in Google Cloud. The first is the standard instance. The second is high memory. It's aptly named. These machines have more memory as it relates to CPUs. On the other hand, the high CPU machine type provides you more CPU and less memory. The ultra-memory machine type has an extremely large amount of memory. Currently, you can get up to four terabytes of memory on an ultra-memory machine. And finally, there's the shared machine type. These are very small machines, which do not have a dedicated vCPU. However, share a CPU with other virtual machines running on the host, and based on availability, you may have full or fractional use of the vCPU. These are very economical and low-priced images, largely for testing or small workloads. If none of the predefined machine types fit your needs, you can also create your own custom machine type in the interface. Using a convenient slider or text entry interface, Google allows you to ask for how many CPUs you think you'll need and how much memory you think you'll need. There's a basic ratio that's enforced between cores and memory. Currently, it ranges between 3 gigabytes and 9 gigabytes of memory per CPU. However, for an additional cost, you can up this ratio and extend the memory. This varies and consult the pricing information on Google's website for further information and the most current information, but the key thing to note is that you can choose your mix if one of the predefined machine types feels too small or too large. It's important to note that the predefined machine types typically are more economical than a similarly sized custom machine type. However, if your desired and optimal resource allocation exists in the middle of machine types, it's very possible that a custom machine type would be able to save you money. I recommend the use of the cloud pricing calculator on Google's website for not only up-to-date information, but to allow you in real time to scroll to the various options and compare predefined machine types and how many resources they have with your own custom machine types. Further, the pricing calculator will give you the most up-to-date list of predefined machine types which does tend to vary. Beyond CPU and RAM, the next important part of any server, of course, is persistent storage. CPU and RAM are also the items by which you're billed and the units by which you're billed, and persistent storage is the third. Much like in a hardware machine you're able to insert and remove disks, you can do this on Google Cloud's Compute Engine Virtual Machines. Disks can be defined through the web interface, the REST API, or the command line. These disks are billed by gigabytes used and can be backed either by hard drives, which are spinning magnetic media, or solid-state disks, SSDs. As you'd expect, the hard disk drive offering is not nearly as performant as the SSD, however, is significantly cheaper currently. Again, consult the Google website for update pricing information. Regardless of whether you choose a hard drive or a solid-state disk, these storage mechanisms are attached to your virtual machine via a storage area network. This is an important thing to note. While highly performant, it's not nearly as fast as a disk which may be stored directly in your computer or directly in your server and attached via local storage interfaces. This provides you with some flexibility if you were to stop a virtual machine and needed to reattach that disk to a virtual machine in a different region or zone or perhaps even just a new machine you created. However, it does have certain performance implications as even the fastest network simply can't compete with the speed of locally attached interfaces. To help provide predictability around this, whether you use a hard drive or a solid-state drive, the performance of the underlying storage system that you attach is measured in two ways. First, by something called IOOps, short for input-output operations and throughput. They're a related metric but slightly different. IOOps measures how many operations that disk can accommodate in a given second. Throughput defines how much data can be read or written from that disk in any given second. Typically, the larger the volume that you define, the more IOOps and higher throughput you can get. Both are measured as read and write and both are measured as sustained or burst. Basically, all of that means you're able to access a certain level of performance from the disk that you create and it is defined by two major variables. Number one, whether you chose solid-state or hard drive and the size of the volume that you define. Other factors go into performance but they're not nearly as important as those two. Factors such as machine size, how the disk might be formatted, especially for SSDs, how much CPU is allocated to the given host machine can impact the amount of IOOps and throughput available on a given volume. Regardless of how large your volume is and whether it's a hard drive or a solid-state disk, you can use something called snapshots to back up, replicate, and move your disk across the Google Cloud Platform. Snapshots are also billed by the Gigabyte but at a reduced rate. Snapshots also provide you with the ability to ensure that your data is safe and if you need to recover the state of a given virtual machine, you can simply create a new virtual machine based on the state of that snapshot at any given moment. It should be noted that snapshot storage is almost always significantly smaller than the actual disk size. A variety of reasons for this exist but mostly because snapshot storage leverages not just compression but only saving the actual data changed from the previous time a snapshot was taken instead of making a wholesale copy of the disk. For backup and replication purposes, this can provide significant savings and speed gains when taking new snapshots. While the Google Cloud Platform Storage Area Network is amongst the best, it's simply no comparison when a disk is attached locally to a machine. For this reason, Google Cloud Platform offers local storage. This isn't exactly the same as persistent storage that we just discussed, though. Local storage has some unique advantages and some unique limitations. Local storage on Google Cloud Platform is an SSD or a number of SSDs. Currently, you can attach up to four to most VM types that is physically attached to the system that's running the VM. There's no network in the middle. As you can imagine, this provides ultra-high performance. However, local SSDs, because they're locally attached to a given piece of hardware, lose their contents when the VM is restarted or stopped for any reason, and this includes maintenance. They're sold at 375 gigabyte increments with a limit on how many you can attach. This varies by machine type and it varies by time as Google continues to change their hardware. So consult the Google website, compute engine page for more information onto the current limits. They're really good for temporary workloads, caches, etc. But since they are subject to lose their data, they're ephemeral and they're physically attached to the hardware under which your virtual machine is running, they can't be moved between virtual machines, they can't be snapshotted, and if your virtual machine for some reason has to be shut down for maintenance, a power outage, or any other reason, you will lose the data stored on that SSD. For that reason, it really should only be discussed or used in applications where loss of that data is perfectly okay. So far, we've discussed virtual machines and disks. These are two examples of resources in Google Cloud Platform. Resources are simply a logical object and typically the smallest unit of organization in Google Cloud. Resources exist inside a project. A project is a logical grouping of resources that are all bound to largely the same work. Resources within a project can typically utilize each other and connect to each other through certain auto-discovery mechanisms, but also for billing purposes are consolidated. While you can use things called IAM roles, which we'll discuss later in the lecture series to limit access between resources within a given project, a project is designed to offer you the convenience of grouping items together that likely and should work together, even if they don't directly access their resources all the time. A Google project in Google Cloud can represent a single environment, a single application, or a single organizational unit. It's up to you as to how you use that unit to factor your projects. Many examples include having a separate project for development, staging, and production. In another situation, a separate project for each different project you're working on for a different client, for a different employer, or just a different task that you've taken is another way of using that logical separation to keep resources separate both on a technical and a billing perspective. One of the conveniences of virtual machines is that you don't have to worry about where that hardware exists, how it's getting power and network. However, it is a practical consideration when considering where the users of a given application are and where it's hosted, how physically far they are, but also from a redundancy perspective. If we would like to host multiple copies of our application or our workload, having virtual machines in different parts of the world, our country, etc., would perhaps provide us with an ability to be resilient from failures due to natural disasters, etc. It's for this reason that Google offers the idea of VMs being launched into a given zone. A zone is the smallest geographic representation in Google Cloud. Zones live inside regions. Regions can be thought of as cities or metropolitan areas and zones can be thought of as different buildings or different rooms within those. Zones are in a reasonable amount guaranteed to be independent of one another. For example, if power were to go out in a given zone, ideally, the power systems for all the other zones within that region would be completely independent. That way, you're able to have a high availability system in one city or one geographic area in one region without having to traverse large network links or slower network links across large distances and still be able to have separate systems which are resilient from failure. An outage in a region would be something much more major that would take out a few city blocks or an entire city, for example, a hurricane, a major fire event, an act of war. These kind of items would be a cross zone or regional disruption, which are typically a whole lot less common. New regions and zones are coming up all the time. You should consult the Google Cloud Platform website for an update list of regions and the zones within them. When you do this, it's important to note that different regions in different zones might have subtly different capabilities. The type of CPUs offered in each zone varies as does the type of GPUs offered in each zone. The network capabilities tend to vary by zone, and you should consult Google's website for an update listing of these, as well as meet potential pricing differences between zones or regions. Regardless of what zone or region that you choose for your virtual machine, you have the option of running a standard or preemptible virtual machine. Standard instances are just what they sound like. They run until you stop them, and they live in a stopped state until you choose to restart them or delete them. Preemptible instances, on the other hand, are technically the same. They run the same hardware. They're configured exactly the same, except you check the preemptible box. But they're much cheaper because they're sold based on the excess capacity that Google Cloud Platform has at any given moment. They may be stopped at any time once that excess capacity goes away, and they'll always only live 24 hours. Preemptible instances are wonderful for short-term workloads, for batch processing, or for dealing with spikes in traffic, which you simply couldn't predict, or would be too expensive to handle through a standard virtual machine. When preemptible instances are deleted, all information and disassociated with them is also deleted. That's incredibly important to note. For standard instances, you have the option to choose between sole and shared tenancy. Remember, virtual machines are simply a portion of a larger hardware machine which you're purchasing the right to use for a given time. If another user is running a virtual machine alongside your virtual machine on the same hardware, there might be certain compliance issues in certain industries or for certain applications, or for optimal performance, you may elect that you only want to be running on a machine that's running your work and no one else's, so you can truly control what is happening in terms of workload on the underlying hardware. By choosing sole tenancy, Google Cloud Platform charges a small premium in exchange for assuring that your virtual machines run on hardware that no one else's virtual machines are running on. This feature is largely there for compliance needs for large organizations or any size organization with an application or a workload with an isolation requirement based on compliance needs. From a performance perspective, Google Cloud Platform has consistently demonstrated that shared tenancy tends not to affect performance too greatly. However, your own experiments and metrics are going to be important in helping you guide this decision if performance is the reason you're seeking sole tenancy over a need for compliance. Now that we know a bit more about virtual machines in general, let's see how the VM lives throughout its lifecycle. All virtual machines, whether they're standard, preemptible, where they live, etc., are defined in the console, the command line, or by the API by an external process. We've already looked at the console and in this course we'll focus on the console. You select the name, the type, the image, or the container, the location, network, disks, and any other items that you would like to specify in your virtual machine. The virtual machine gets created and you can view it. Once it's created, it will boot and start just like a regular system automatically based on the options that you previously chose when you defined the virtual machine. They may start on a specific processor or microarchitecture type or with certain graphics processing units attached. For example, you may choose when starting a virtual machine that you specify that virtual machine, run in a system with an Intel Skylake or newer processor. Google will charge you a premium for making this specification. Whereas if you were not to specify it, you'd be charged standard rates and have no predictability over what machine your given virtual machine will run on. It may run on a very modern processor or it may run on an older one. You can consult the Google Cloud Platform documentation and website for a list of currently available microprocessors as hardware gets released and Google upgrades its hardware. This continues to change. Consult the price list for information as to what price premium a given platform might pull with it. While your virtual machine is running, it's quite possible that something could go wrong or maintenance simply needs to happen. There may be an underlying hardware failure, hypervisor security fixes or maintenance may be required, or there might be scheduled facilities maintenance that requires the hardware running your virtual machine to be brought down or somehow altered. If this happens, Google Compute Engine offers seamless migration for most resources. If you have a GPU equipped to your VM or if it's a preemptible VM or a few other cases, this may not be possible. However, most virtual machines can be moved without any impact of processes inside it to new hardware. You can set an option to turn this functionality off if you'd like. However, it's largely seamless and we discuss it mostly for informational purposes as it relates to resiliency for running your virtual machine for a long period of time. Whether you chose a standard or preemptible virtual machine has an impact on the next step. Remember, a preemptible virtual machine may be stopped at any time by Google for their needs or at the 24-hour mark. You can also terminate a virtual machine that is set as preemptible. However, you can't restart it. Once you've stopped it, it's gone. It'll automatically be deleted. On the other hand, standard virtual machines can be stopped at which point the billing for the VCPU and the RAM ceases. Any disks, IP addresses that you're using will continue to be billed. However, the VCPU and RAM billing will not continue until you start the instance again. You may either elect to start the instance again or delete it if you no longer need it. The resources that are associated with that instance if you choose to delete it can be retained unless the boot disk for that image is specifically marked to be deleted when the instance is deleted. But remember, disks are separate resources in Google Compute Engine and you can easily retain them and move them to another virtual machine or simply let them live on as disks sitting on their own for some future use. It is, however, most economical that if you're no longer using a disk to archive it using the snapshotting function. Currently, snapshot rates are almost 50% cheaper than the standard hard drive persistent storage rates and even more savings versus the SSD rate. Now that you know a little bit more about virtual machines, let's move on to a practical example. We'll get into the advantages and disadvantages of VMs as it relates to the other three compute products once we've gone over those three lectures. For now, let's just look at a practical example that brings up a Linux Debian-based virtual machine. We'll start the virtual machine with nearly the default options and then we'll edit it to include an additional persistent disk beyond just the boot disk and we'll add more VCPU and RAM. Now, there are several ways you could accomplish this. You could use the command line or G Cloud command tool from your local machine. You could perhaps even use the REST API if you're so inclined or talented but the most overwhelmingly common and user-friendly tool to do this is using the Google Cloud Console on the web. We'll use the Google Cloud Console not just now but throughout the course that we already looked at in the first few lectures in this lecture series. We'll also examine the REST and command line equivalents in the Cloud Console interface just so you have familiarity with those concepts in case you deep them. We'll get started by going to the Google Cloud Platform Console that we explored and set up an account in earlier in the lecture series. You should be greeted by this page. By going to the upper left-hand corner navigation menu we can access the listing of all Google Cloud Platform tools. Let's click here. You can see frequently used items are at the top. Yours will be empty the browser remembers I was exploring in Kubernetes Engine and a few of the other services earlier today. You can also elect to pin certain things to the top by clicking the pin as you hover over a given service. This will save it up at the top of the menu for convenient access. Another method is to go to the search bar. We're looking for Compute Engine. Let's type Compute Engine. We have a variety of hits because Compute Engine has a number of related items like disks and instances, etc. Let's just go to Compute Engine. This would have been the same if we'd chosen it from the navigation menu on the left. If this is your first time accessing Compute Engine or their first time accessing Compute Engine in your given project it may take a moment or two for the API to get ready. You will see I'm greeted by a status message since this is a new project that Compute Engine is getting ready that this may take a minute or more. This will only happen the first time you access Compute Engine in a given project. Once the indicator that it's ready has gone away or if you were never forced to wait let's click on Create a VM instance the Create button. There is a standard dialog which will provide us the options both simple and advanced in order to create a virtual machine instance. I'll keep the name instance-1. This is a user-friendly string that you can set to nearly anything and is largely for your reference. You can choose which region and zone that this is being created in. I'm fine with US East 1 and the B zone is fine with me. The region continues to grow and you can see that Google Cloud Platform has a variety of regions throughout the world to find one that's closest to you. Next, you can choose the machine type. Obviously, the smaller machine that you use the less it'll be in terms of pricing. You can also click Customize to be able to get a full listing of all of the options. These are simply the predefined machine types that we discussed earlier. If one meets your needs well I would encourage you to choose that over using the customizable machine type as they tend to be slightly cheaper. However, if there isn't a good fit a custom machine type will be more economical. You can use the sliders to change your variety of options and if you notice it'll also update the price on the right hand side. If you click Details it will break down as to what is driving that cost. Notice that the sustained use discount is automatically applied and that's assuming that I'm running the virtual machine for the entirety of the month. If I stop it overnight, etc. and reduce my usage I may qualify for a lower or no discount at all. Remember, those pricing terms change and I would encourage you to look at and consult the latest version of the Google Cloud price sheet as anything I mentioned here will likely be out of date as soon as the course is published. You could use the Compute Engine pricing link to have a convenient window open with the most up-to-date pricing. You can choose your CPU platform. Remember, as we mentioned this will incur a slightly higher charge. If you noticed we chose Intel Broadwell or newer and we can choose Skylake or newer. In this situation it appears that the pricing isn't simply not being increased. Based on your zone choice that pricing may go up or down. We can add GPUs for machine learning. We can also choose to deploy a container. Remember, by default Google Compute Engine will create a boot disk with an operating system installed on it. In this case a 10 gigabyte disk with Debian Linux 9. If we chose a container it would ask us to specify a container image which is hosted either on Public Docker Hub or our own repository and also give us the option to define environment variables and other Docker specific items. For more information on how this works consult Docker documentation and fill in the fields as appropriate based on your Docker images documentation as well. I'm perfectly happy with running a Debian Linux 9 instance. However, let's go explore what our other options are. Remember, we have a variety of information and images available to us. We don't have to just run the stock image. We can choose from application images such as SQL Server. We can choose from custom images which we may have created. Snapshots of previous VMs we've taken that we wanted to save or even existing disks. In this situation, we'll stick with the default. We can leave the next set of options completely on their own and not touch them for now. We'll discuss what service accounts and identities are later in the lecture series. As a prelude, this offers us to limit the virtual machine's access to other resources in the project based on assigning it an identity which we will then restrict. But let's not worry about that or the firewall for now. Let's explore a few more of the options. We can add a description and labels which allow us to add exactly that to our virtual machine. A human-friendly description for other coworkers, individuals, customers who might want to know more about what this virtual machine is about. We can also specify disks while we're creating the virtual machine. This is something for the purpose of the demonstration later so we can walk through stopping a virtual machine. However, we could attach a new disk or we could attach an existing disk when we create the VM all in one step. Again, for the purposes of discussion and for the purposes of demonstration, we'll demonstrate how that works and we stop the VM and edit it. We can also set our sole tenancy options that will allow us to be able to say this virtual machine should only be ours. I'm fine with all the defaults and we'll go over details of items like firewalls, networks, and management in further lecture series and in further lectures in this series when we have gone over those specific topics. Since we've created a rather large machine type to examine its effect on pricing, before we actually kick the start button, let's go ahead and take it back to something a little bit more reasonable instead of the $1,200 a month. One vCPU at sustained use is about $25 a month. Sounds much better to me. Let's examine the command line tool and options to create this exact instance if we were to prefer to use the command line. At the bottom, by clicking equivalent rest or command line, we can see if we had elected to use the command line interface, the G Cloud tool, this is what all of those options specified would have to be to get the exact same outcome. You can cut and paste this. You can run it in the cloud shell or you can simply use it for reference or you could ignore it if you're confident that you'd simply only ever use the web interface. This is simply here for your convenience in case you'd like to learn and see the equivalent in either the command line or in the API form if you're a developer. This allows you to use this URL and this data structure to programmatically call Google Cloud Platform to create this exact instance. Again, go ahead and click close as these are simply references to understand that there are other methods for creating instances in certain cases either if you're a developer or limited to the command line. Go ahead and click create to create our instance. It may take a moment or two for our instance to come up. Let's wait for it and as soon as this turns to a green check mark we'll know that our instance is created. It's ready. Let's click on instance one. There are a variety of ways that we can access our instance. We can click the SSH button most conveniently. Let's go ahead and SSH into our instance. This brings up a web-based SSH client, handles all transfers of SSH keys to the virtual machine and will give us a login prompt once that process is completed. Once your virtual machine has come up you'll notice a familiar, hopefully, Linux command prompt. Let's have a look at some system statistics. Using the top command we can see that there are really no processes running, that there's approximately 3.7 gigs of RAM and there's nothing else running on the system beyond the normal system processes. This is to be expected and it's been up for about a minute. Let's go back to the console by closing this window and let's take the system down. Let's do two things. First, we'll add a disk and second, we'll upgrade the machine type. It should be noted that you do not need to take a machine offline to add a disk. You create a new disk and attach an existing disk while a machine is running. However, how to change the machine type requires taking the machine down by clicking the stop button. Let's give it a moment to stop and then we'll be able to edit it. Once the machine has stopped we'll click the edit button. We got a message that stopping the VM instance succeeded. Let's click the edit button. Remember, you can edit certain things while the machine is running. Let's change the machine type so we're going to wait until the stop was successful. In this situation, let's say that we discovered that our workload simply needed a little bit more memory but not too much more CPU. In that situation, let's go look at the high mem predefined machine types. One more CPU and 13 gigabytes of memory. That's opposed to the one CPU and 3.75 gigabytes of memory. One more CPU. Sounds good to me. Let's scroll down to where it says additional disks. Let's click add item. Here, we can choose to keep the disk when deleting the instance or delete the disk when deleting the instance. Let's choose delete disk. That way, when I delete the instance, I don't have to go and delete the disk on my own. But remember, if this is going to store critical information after the instance, it may be important to choose the keep disk option to ensure that that data is not lost. In the name of the disk, we have no disks created in Compute Engine yet. So we'll click create disks. Any existing disks that were not in use would have shown up there. In this situation, we can choose to create a disk from an existing image. The images, for example, like operating system images or application images that we were able to choose to delete the instance are available to us. A previous snapshot we might have taken of another disk or a blank disk. In this case, let's create another 10 gigabyte blank disk. You'll notice the warning and the estimated performance summary. Remember, we discussed IO ops and throughput. In this situation, because we've chosen a relatively small disk, Google is warning us that we have IO ops both on read and on write and a relatively anemic performance in terms of megabits per second of data that can become in and out of this disk. For good performance, Google recommends at least a 200 gigabyte disk. Let's make that change to see how those numbers will change. You see they went up in some cases by almost tenfold. Consider this when you're building a disk, how much space you need, but also how performance generally scales at a certain level and to a certain point with the size of the disk. We can also choose what type of disk. Remember, we have the choice between cheaper but not as fast hard drives, also called standard persistent disks, or we can choose an SSD. It doesn't change our options, but notice our performance did change when we changed to an SSD disk. The performance change between SSD and iOops for a 200 gigabyte disk. Moving that to a standard persistent disk brought that down to 150 to 300. In some situations two orders of magnitude more performance out of an SSD. But remember, you'll pay commensurately. Let's go ahead and create a 200 gigabyte standard persistent disk. Once the disk creation is complete you'll notice that it's added. We can scroll down to save our changes. And once the instance is updated the start button will be enabled for us and we'll be able to continue. Let's click the start button to start our instance back up and then when it's ready we'll SSH into it to see our new disk. The instance start was succeeded so let's SSH into our instance and let's examine what volumes are attached. We'll use Linux's built-in LSBLK command to list block devices attached to our machine. You'll see the first 10 gigabyte partition that was created that's our boot partition that was created when we started the instance. You'll also see SDB which is a 200 gigabyte disk. We can use the standard Linux file system tools to format that disk and mount it wherever we please. Let's log out of the instance and stop this instance so we're no longer billed for it. Remember, we'd set the disk to be deleted when we delete the instance. Once it's stopped let's use the delete command to delete the instance and verify that the disk was deleted as well. Now that the instance is stopped let's click the delete button. It takes a few minutes for a delete to complete especially on linked disks or in accounts with a lot of resources. In this situation it happened relatively quickly. Surprisingly so. Let's click on disks and see if anything's there. Nope. The disk listing is empty as we expected as it should be. You've been able to see using Google Cloud Platform's cloud-based and web-based console is quite easy to set up a virtual machine exactly how you want it. This interface also provides you with hints as to how to use the API and the command interface if you need. In future lectures, as we learn more about the other aspects of Google Cloud Platform we'll dive deeper into the other options available to us that we took a little bit less time to gloss over in this situation. Top of Google Compute Engine Instances Kubernetes Engine simply provides a framework for running complicated Docker applications. Let's do a quick recap on containers. Traditionally a whole bunch of different apps sat on a single machine as you can see on the left. These apps shared libraries and kernel and this would cause a lot of problems because one app can hog all the resources. Then came VMs or virtual machines. Here you have total isolation between the apps. So each app has its own libraries and kernel, but this also causes other problems. Now we have a very complicated solution and a very expensive solution because we are duplicating the libraries and kernels for each app. So finally we come to containers. With containers you have the right amount of isolation because each app is packaged with its own libraries. At the same time the kernel is shared between the apps. Now containers are not a new concept. They have been there and they have been used for a long time in companies like Google. Docker came in and the public adoption of containers took off. Docker offered a lightweight container runtime with easy packaging and deployment. So this allowed you to have your app bundled with all its dependencies and easily deploy that across different environments. Now Docker only focused on providing an amazing experience for deploying a single application with a single node and a single container. Now this is not very scalable unless your project only needs one container. So then came Kubernetes. Kubernetes is an open source product. It's known as a container orchestration tool. So what does that mean? It basically means that you can manage multiple containers across multiple nodes and pods. A pod is a group of containers or it can even contain a single container. A pod is the smallest schedulable unit in the system. So Kubernetes handles scheduling health checks etc. It's also commonly known as an abstraction over infrastructure. So what does that mean? That basically means that the infrastructure operators and developers are separated and Kubernetes manages all the services like node lifecycle management disaster recovery security etc. while allowing the developers to worry about building applications. Now we have Google Kubernetes engine with GKE Google runs your clusters and it helps you run Kubernetes at scale. Now it provides a fully managed Kubernetes solution Google's version of Kubernetes is the original open source Kubernetes but with added flexibility and added functions. So on the Google's Kubernetes engine you have two main parts a control plane and then you have the nodes. Now the control plane is where all the management of the Kubernetes happens. Google has a team of site reliability engineers that are constantly taking care of day-to-day operations such as scaling security patches upgrading etc. Now GKE runs on Google Cloud Platform so you get the advantages of using other services that Google provides simply via plug and play. GKE also provides a fully managed node experience. This means it provides a base image provides updates to the base image provides services such as node auto repair and node auto upgrade. So nodes are grouped into node pools in GKE node pools are a GKE concept. There are a lot of advantages to using node pools. Node pools are simply a group of related machines where you can easily mix and match different instance types. So you can spin up a node pool with the standard configuration and you can add a second node pool with preemptive VMs or GPU, CPU and local storage. You can also target specific workloads to these node pools. Node auto upgrade is a GKE concept. Let's take a look at what happens when you want to upgrade a node. GKE automatically keeps nodes up to date with the latest stable version of Kubernetes. It also applies the latest security updates and at the OS as well as the K8s level. This happens during a maintenance window. A maintenance window is something that you can specify that's like 3-4 hours when you allow Google to upgrade your nodes. Usually this is when you have low traffic or when you have some sort of a downtime is allowable. So during the node upgrade the node that needs to be upgraded first gets drained. Then there is a block put in so that no more traffic can access that specific node. A new node that has the latest version of Kubernetes is spun up and attached to a VM. That node is then attached to the same cluster where the node pool exists. Similarly there is another feature of GKE that's called node auto repair. Node auto repair helps keep your nodes in a healthy running state. It checks for errors and unreachable nodes and it's constantly monitoring for this in the background. So if it does find any error it sets that node to not ready status and the same procedure as the node auto upgrade happens. That is the node that is drained and then a new node is spun up and then that's attached to the cluster and the node pool. As we said previously these tools and processes happen in the background in an automated fashion so that developers can focus on building applications. Google takes care of this infrastructure management in addition to those node services. There is also high availability and specific GCP connected services. With high availability the control plane is replicated to 3 regions. By doing this this increases the uptime of the cluster from 99.5% to 99.95%. This also provides zero downtime upgrades. You can also easily connect to other GCP services such as VPC networking, persistent disk load balancer and stack driver monitoring all done straight to the pod level. These features are actually free and they don't cost anything extra. Now let's take a look at the GUI for GKE. So on the cloud platform on Google Kubernetes Engine there are four main parts. There are clusters workloads services and applications. A cluster consists of at least one cluster master and multiple worker machines called nodes. These master and node machines run the Kubernetes cluster orchestration system. As we previously said the cluster master is the control plane. This can be either one or it could be up to three replicated across different regions. Containers whether for applications or batch ops are collectively called workloads. Before you deploy a workload on a GKE cluster you must first package it the workload into a container. A Kubernetes service is an abstraction which defines a logical set of pods and a policy by which to access them sometimes called a microservice. A service is also what connects pods to other GCP services like a load balancer in order to connect to the outside world. Kubernetes applications collect containers services and configuration that are managed together. GKE allows for one step deploy as well as from the full G Cloud command line. For the demo we'll take a look at how WordPress application can be deployed on GKE. Now if you're not familiar with WordPress it's an open source and free web publication publishing application and a content management system. It's the world's largest blogging tool and it's used as a backend for a lot of websites on the internet. A typical WordPress installation consists of a file system and a MySQL database. When this is deployed on GKE you get two workloads two pods and two services that are each for the file system and the MySQL database respectively. The pod for the file system connects to a load balancer so that you can get an external IP address. The deployment can be broken down into 10 steps. So if this is the first time you're deploying the application on GKE then you'll need to first set up the billing information. Then you'll get to create or select a project. Then you have to make sure you have the right permissions for the project and the user that is currently deploying it. You'll have to create a cluster or if you have an existing cluster you can select that. Then you can deploy the application check the workloads and services expose the pod with the workload to a load balancer and then assign an external IP address. Let's take a look at how this is done on GKE. Here we have the Google Cloud Platform homepage. From here you can take a look at the navigation menu and then find billing. This should be the first step in case you haven't set up your billing information. You need to provide your credit card number and your billing information like your business name, address and so on and set this up. This is usually a one-time setup that needs to be done in the beginning. Once that's done then you can check your security and permissions. That's at the IAM and admin section. Here if you want to make sure that the user that's currently deploying or managing Kubernetes has a role that's either the owner or the Kubernetes engine service agent. These would be the roles that would have the correct authentication to manage Kubernetes. Now I've already set up a project. If you haven't done this yet or if you don't have any projects, you can create or select a project from the menu here. Now let's actually go into the Kubernetes part. Back to the navigation pane and here on the compute you have the Kubernetes engine. As we have seen in the previous slides, there are four main parts. Clusters, workloads, workloads, services, and applications. Then you have configuration and storage. The first part we need to do is to go to clusters and have at least one cluster set up. Let's go ahead and do that. Since there are no clusters here, you'll get this box here saying create cluster. This is our first cluster. Let's just choose a standard cluster. And here we can name our cluster. In this case I'm going to leave it as standard cluster one. And you can choose your location type, the zone and the version of Kubernetes. We can also choose the node pools. So the default pool in this case, which by default it selects up to three nodes and the machine type and the memory can also be customized. For this example, we're going to leave everything as default here. Simply press create and wait until the cluster is created. This can take a bit of time and we will know when the cluster is created this spinning circle turns into a checkbox. OK, so the cluster has been created. You can see the green checkbox and if you click on it you can get the details of the cluster. As we have set it up it looks good. VPC has been disabled by default. We have a default pool node pool of size 3 total memory of 11.25 gigabytes and three CPU cores. We can take a look at if we have any storage, none and the nodes and these are the three nodes in the node pool. Now if you look at the workloads you should see that there are none and the same for the services at this point. So now in order to deploy our application we go to the applications tab. So here again there are currently no applications deployed on this cluster so we can do a deploy from Marketplace. Deploying from the Marketplace allows a one click or a very easy method for deploying applications. All the popular applications and code releases are available here. You can browse through these and see that currently there are 40 applications and more and more are being added. Now if you don't want to do the one click deploy you can also do all of this through the command line at Gcloud and use the API server to deploy your applications and manage them. So in this case we are looking for our WordPress and you can see that right here. So we click on that and we get to a page that says WordPress Deploy Configure and provides the details. So here we have chosen the cluster that we have previously created the namespace the default namespace and the app instance name in this case WordPress. Press deploy and now the application is being deployed. You can see the progress here. Now what's actually happening in the background is the system is creating the file system as well as a database. So the file system is where the WordPress application files reside and the database is a MySQL database that contains the images and the data that supports the application. So that's already done here as you can see standard cluster with the namespace and the labels. Now if we go back to the workloads we can see that there are two workloads that are created. One of them is for the WordPress and one of them is for MySQL. Now you can see that there are parts that are pending and there are warnings because the implementation is still happening. If you look at the services you can see that two services have been created one for WordPress and one for MySQL. Now that we came back the you can see the green checkbox for all three workloads and they have been set up properly. Now if you look at the workload for the WordPress file system you can see the actual parts that are currently being run here. So here if you click on the pod we get the details of the pod that contains the WordPress file system. Now the service that's associated with this pod is the WordPress demo WordPress SVC service. Now this has an endpoint of 10.11.247 IP address. Now this is an internal IP address that's only seen by the cluster. So if you try to access this from your browser window you won't be able to see anything here. Now in order for this application to be available to the outside world we'll have to expose this. So here you can see the expose button click on this and we have a new dialog box here where we can provide the details on how to expose this service or this pod in this case. So here the service type is where we need to click on the drop down and choose a load balancer. The load balancer is what will create an external IP address for this pod and allow it to communicate with the outside world. So now it's creating the service endpoints and you can see that it's in progress right now. So now if we go back to the services tab we should be able to see the third service here that's currently in progress. Now this is the service that will have the exposed external IP address. Click on refresh still waiting. Now if you take a look at the workloads you won't see any difference here because you still have the same two workloads one for the WordPress and one for the MySQL. So now you can see that the WordPress demo application service which has the load balancer service type has been created and this is our external IP address. Clicking on this will take you to our WordPress installation. Now from here it's the standard WordPress application that you would get if you install it in any many ways. The product and service that can handle millions of requests per second. Cloud pops up it's a scalable durable fully managed messaging system that's both real time and guarantees delivery. It brings flexibility and reliability of enterprise message oriented hardware to the cloud. Cloud pops up is an asynchronous messaging service designed to be highly reliable and scalable. This service is built on a core Google infrastructure component that many Google products have relied upon for over a decade. It provides low latency durable messaging that helps developers quickly integrate systems hosted on GCP. There are a lot of Google products that are using PubSub. Like ads, search, Gmail and even Google Home. Let's first answer what PubSub means as a service. So a published subscribed service is a messaging service where the senders of messages called publishers are decoupled from receivers of messages called subscribers. The key concepts in a PubSub service are a message with the data, that's just the data that moves through the service. There's something called a topic which is a named entity that represents a feed of messages. A subscription which represents an interest in receiving a message a publisher or a producer and a subscriber also a consumer. Published messages are categorized into topics. The publisher application creates a topic and sends messages to the topic. You can see a quick workflow on the image on the right. A message contains a payload and optional attributes that describe the content of the payload. Messages are persisted in a message store until they are delivered and acknowledged by the subscribers. PubSub then individually forwards messages from a topic to all of its subscribers. Subscribers who have expressed interest in a topic will either pull related messages or choose to get push messages from the topic. The subscriber receives pending messages from its subscription and acknowledges each one of the PubSub service. When a message is acknowledged by the subscriber it is removed from the subscription's queue of messages. The key to the service here is that neither party is required to have any knowledge of the other. Now let's take a look at another scenario. Here there are two publishers publishing messages on a single topic. There are two subscriptions to the topic where the first subscription has two subscribers and the second subscription has one subscriber. The bold letters represent messages. Message A comes from publisher 1 and is sent to subscriber 2 via subscription 1 and to subscriber 3 via subscription 2. Message B comes from publisher 2 and is sent to subscriber 1 via subscription 1 and to subscriber 3 via subscription 2. Let's take a look at some of the common use cases where PubSub is useful. Network Clusters In other lectures we've seen Google Compute Engine instances being used for Google App Engine or Kubernetes and other heavy compute products. Those products have a large queue of tasks that can be efficiently distributed among multiple workers and this process can be streamlined using PubSub. asynchronous workflows an order processing application for example a restaurant can be getting a lot of different orders at the same time but they can be processed separately by different workers. Cloud PubSub is great for distributing event notifications. Take for example a service that accepts user signups. It can send notifications whenever a new user registers and downstream services can subscribe to receive notifications of that event. The PubSub service is a part of the architecture that supports the streaming data from residential sensors to the backend servers that are hosted on cloud. For example you have Google Home or Alexa or even Siri on the iPhone or examples of residential sensors that send back a lot of data to get processed in real time and sent back to users. You can do multiple systems. A lot of the products on GCP like the compute engine instances write logs constantly in the background and later they can be queried. Reliability improvement A single zone compute engine service can operate in additional zones by subscribing to a common topic to recover from failures in a zone or region. This is how Kubernetes for example can now be enabled in a multi-zoned environment in three separate regions in order to have a failsafe service. Let's look at the endpoints. Publishers can be any application that can make HTTPS requests on GoogleAPIs.com such as an app engine app a web service hosted on Google compute engine or any other third-party network an installed app for desktop or mobile device or even a browser. Pull subscribers can also be any application that can make HTTPS requests on GoogleAPIs.com Currently push subscribers must be webhook endpoints that can accept post requests over HTTPS. Performance of a messaging service are usually judged by three metrics scalability availability and latency. Scalability a scalable service should be able to handle increases in load without noticeable degradation of latency or availability. Load here can refer to any of these dimensions such as the number of topics the number of publishers the number of subscribers the number of subscriptions the number of messages size of messages rate of messages and the size of the backlog. In a distributed end system the types and severity of problems can vary greatly. A systems availability is measured on how well it deals with different types of issues gracefully failing over in a way that is noticeable least to end users. Failures can occur in hardware such as disk drives not working or network connectivity problems in software and due to load. Failure due to load could happen when a sudden increase in traffic in the service or in other software components running on the same hardware or in software dependencies results in resource scarcity. Availability can also degrade due to human error where one makes mistakes in building or deploying software or configurations. When it comes to latency there are two main metrics for PubSub the amount of time it takes to acknowledge a published message and the amount of time it takes to deliver a published message to a subscriber. Now let's take a real world example to illustrate how PubSub works. Spotify is a popular music streaming service it had a phenomenal user growth in the past few years where the user base tripled reaching 75 million monthly active users. With the user growth came the data growth they were shipping over 60 billion events every day to their Hadoop cluster and the events were being passed through a Kafka based delivery system. All the events that are being delivered are being generated on their clients as a response to certain user actions. So what are these user actions? For example when a user creates a playlist subscribes to a playlist or when they finish listening to a song an event is generated this event is delivered to the Hadoop cluster this system worked fine until a certain scale and with the load came outages. The biggest problem was that this system is way too complex it has way too many components and on top of that most of those components were having limited coverage which is an operational nightmare. The other problem was that the event delivery system is stateless what does that mean? That means that the state is only kept on the service machines themselves and on the Hadoop cluster so if the Hadoop cluster goes down the event delivery stops functioning which is an operational burden for the event delivery team they're responsible for the uptime of the cluster and the infrastructure and that's separate from the software developers when you combine these problems with the data growth you get a recipe for disaster so they decided to look for a different system which would be easy to migrate Spotify saw that Cloud PubSub can retain the support data for 7 days and that it has at least once delivery semantics built in with these two features that would solve some of the biggest problems Spotify was already having Cloud PubSub is a globally available service that means that messages published in one region can be directly consumed from another region this is important for Spotify because that would mean that they could skip flaky cross-Atlantic internet links when publishing from their US data servers and the consumption is in the UAEU also the migration is easy because most systems use a REST API which PubSub uses as well lastly Spotify would have no operational responsibility for it as it's fully managed by Google as you can see here it's a very simple implementation in terms of the number of components before and after using Cloud PubSub the architecture is simplified with the help of Google Cloud Storage and Cloud Dataflow Dataflow SDK is a framework which provides a high-level API for writing data pipelines this is used for the ETL jobs we will see in other lectures more details about Cloud Dataflow and Cloud Storage now let's dig a little deeper into publishers and subscribers a publisher application creates and sends messages to a topic Cloud PubSub offers at least once message delivery and best effort ordering to existing subscribers the general flow for a publisher application is create a message containing your data send a request to Cloud PubSub server to publish the message to the desired topic in addition to that you can have custom attributes attributes can be text or byte strings you can have balancing where messages can be batched based on the request size and the number of messages time etc publishing failures are automatically retired except for errors that do not warrant retries threading and support for concurrency is also available but it varies by the language you are using now we have receivers the subscriber is the one that is receiving the message to receive messages published to a topic you must create a subscription to the topic only messages published to the topic after the subscription is created are available to subscriber applications the subscription connects the topic to a subscriber application that receives and processes messages published to the topic you can have multiple subscriptions but a given subscription must belong to a single topic as for the at least once delivery Cloud PubSub delivers each published message at least once for every subscription by default a message cannot be delivered within the maximum retention time of 7 days then it is deleted and it is no longer accessible a message published before a given subscription was created will usually not be delivered now subscribers who have expressed interest in a topic will either pull related messages or choose to get from the topic let's take a look at the pull subscription the image on the right shows the life cycle of this mechanism in pull delivery your subscriber application initiates requests to the Cloud PubSub server to retrieve the messages the subscribing application explicitly calls the pub pull method which requests messages for delivery the Cloud PubSub server responds with the message and the acknowledgement ID the subscriber explicitly calls the acknowledgement method using the return ACKID in order to acknowledge receipt in push delivery Cloud PubSub initiates requests to the subscriber application to deliver messages the server sends each message as an HTTPS request to the subscriber application at a pre-configured endpoint the endpoint acknowledges the message by returning an HTTP success status code a non-success response indicates that the message should be recent the rate at which these push requests are done is dynamically adjusted by Cloud PubSub and this is based on the number of success responses now when do you use a push or a pull a push can be used when you have a large volume of messages a lot more than one per second when you need efficiency and throughput of messages and the processing of these messages is critical you have a public HTTPS endpoint with no self-signed SSL certificate now for the pull you can have multiple topics that must be processed by the same webhook it's an app engine standard and used for cloud function subscribers cloud functions are discussed in detail in another lecture pull is also used in environments where Google Cloud Platform dependencies such as credentials and the client library are not feasible to setup let's take a look at the lifecycle of a subscription by default subscriptions expire after 31 days of inactivity that is for instance if there are no active connections pull requests or push successes if cloud pub sub however detects subscriber activity the subscription deletion clock restarts you can configure the inactivity duration or make the subscription persistent regardless of activity by using a policy these expiration policies are currently in beta and may not be available until GA you can also delete a subscription manually although you can create a new subscription with the same name but it will have no relation to the old one by default cloud pub sub ensures that subscriptions retain unacknowledged messages for 7 days Google Stackdriver is a monitoring and management tool specifically built from the ground up for services, containers applications and infrastructure the cloud pub sub uses this Stackdriver and its API exports the metrics to it Stackdriver allows you to create monitoring dashboards and alerts to access the metrics programmatically you have two main metrics here you can see the usage metrics that cloud pub sub reports to Stackdriver and you can see the specific details for each topic or subscription that you are monitoring you can also see the activity on the general API with the code as dashboard more details on Stackdriver are also in a separate lecture so let's take a quick look at a demo and see how all this fits together and as a reminder we need to create and have the billing setup already done in order to do any of these tasks on the Google cloud platform so you can follow along these 10 steps where we create topics and subscribers and play with some messages and look at the acknowledgement both from the Google gcloud command line as well as from the interface okay it's demo time in this demo we will give you a brief introduction to cloud pub sub command line interface using the gcloud command here we will create our first topic add a subscription and publish messages to the topic and also pull messages from the subscription so we are back to our familiar user interface the Google cloud platform so the first step is to select or create a project so we go down here and let's create a new project for this demo so now for the project name I can choose pub sub demo 100 and the project id in this case is also pub sub demo 100 now the project is being created and once the circle here stops spinning we are good to go so now we have to select the project that we just created so select project and here is the new one now just a quick note the project id is very important and you want to take note of this because the project id is the same across all google cloud platform products and services so we are always working within a project scope so now let's go to our navigation menu and find cloud pub sub so this is under all the way down under big data cloud pub sub just click on this and here this is the user interface for pub sub so you have topic subscriptions and snapshots so using the UI you can create topics and subscriptions but the first thing we need to do is to enable the api so we click on enable api we only had to do this once just the first time for each project so now the api has been enabled and now we can create a topic now instead of doing it from here we can also use our familiar gcloud interface so for that let's open our cloud shell which is this icon here now we are back to the familiar google console here window while it provisions our cloud shell and we can also see that this is a tabbed interface similar to most browsers and currently we are working within the scope of this project that we just created so now let's go ahead and create a topic now for that I have copied the previous slide the one right before this demo and all this data here is from that slide so this is the command to create a new topic gcloud pub sub topics create and then the name of the topic in this case it's my topic copy this and paste it here ctrl v and hit enter so now a new topic has been created so now let's go ahead and add a subscription the command for adding a subscription is pub sub subscriptions create the name of the subscription and then the topic the name of the topic and then we have an act copy this and paste it here ctrl v and hit enter now you may notice the dash dash act deadline equals 60 option act deadline stands for acknowledgement deadline so this new subscription has a 60 seconds acknowledgement deadline so if we look at this in a second so now our subscription has been created so let's take a look at to see if they have actually been created or not so we can list the topics and subscriptions by sending these messages so to look at the list of topics it's pub sub topics list copy this paste it here ctrl v and we can see that there is this is a topic here that has been created in the same way to see the list of subscriptions it's gcloud pub sub subscriptions list copy this and paste it here ctrl v hit enter and we can see that our subscription with our 60 seconds act deadline has been created so now the next step let's go ahead and publish messages to this topic so we'll send two messages here one just says hello and goodbye so the command for that is pub sub topics publish then the name of the topic and then minus minus then the actual message content so copy this paste it ctrl v hit enter and now for the second one let's say it's goodbye copy this and paste it to hit enter now each of these commands send a message the first message is hello and the second one is goodbye so when we so when the message has been successfully published we can see the message id's that it spits out so here we have the message id for the hello and here we have the message id for the goodbye message so now let's pull the messages so the command for that is pub sub subscriptions pull and then auto acknowledge and then limit to and then the name of the subscription copy that paste it here again ctrl v hit enter so here we see the two messages that we have just published the messages have the data hello and goodbye as well as the message id the message id is the unique id of the message that the server assigned also you can also please take note that cloud pub sub doesn't guarantee the order of these messages it's possible that sometimes you only see one message and then it might take a little bit of time to see the other one or sometimes you'll see the goodbye first and then the hello message second so after you pull the message and correctly process it you must notify cloud pub sub that you successfully received the message so this action is called acknowledge so since in our command here we pass the flag auto acknowledge so in this case the acknowledgement has been automatically processed so now we can also manually do the acknowledgement so let's take a look at that so first let's send a new message again the command is pub sub topics publish then my topic which is the name of the topic and then the message and the content of the messages thanks copy that control V to paste hit enter so now let's pull the message again the command is pub sub subscriptions pull and the name of the subscription so you can see that in this case we're not passing any flags copy that hit enter so now we can see the we can see a long message here and in this case we can see our the message ID that has been assigned the data which is thanks and we can see the act ID which is the acknowledgement ID so this is a very long string of characters now we need so now according to our initial setup we said that within 60 seconds we need to have an acknowledgement sent now if we don't do that then pubs cloud pub sub is going to think that there's an error and it's going to resend that message so now let's do the acknowledgement manually so the command for that is gcloud pub sub subscriptions and the name of the subscription and the flag act IDs and then the actual act ID which is the one that's here so let me first copy this act ID because it's really long and it's going across the screen so as usual when you when you highlight any text in the gcloud console it gets copied to the clipboard automatically so I just pasted that here so let's construct our command so it's gcloud pub sub subscriptions act name of the subscription act IDs and then the actual act ID which is this copy that and paste it so now let's copy this whole command and paste it here there so now we have successfully acknowledged the message with the following act ID so you can so instead of just one you can pass multiple act IDs because the flag is for multiple act IDs now let's take a look at the user interface of what we just did so like I previously said you can create topics and add subscriptions from here however since we already did it from the command line we should be able to see those right here so if I click on subscriptions here we can see the subscription that we just added called my sub the topic my topic with the delivery type pull and the acknowledgement deadline 60 seconds same way if we go to topics this doesn't get refreshed sometimes so you'll have to try it from subscriptions and if you click on if you drill down to the actual topic then the interface gets refreshed so if we go to topics we'll be able to see the topic that we created from the command line here so if you click on that topic to drill down we can see that we can publish the message and create subscriptions and so on from the user interface now we can also go and look at the logs real quick as usual are integrated into Stackdriver so Stackdriver is a monitoring and logging tool that comes with Google Cloud Platform so we go to the navigation menu click on Stackdriver and look at the logs here so here we can see the cloud pub sub logs these are for the tasks that we just performed where we created the topics edit subscriptions and sent messages so this concludes the cloud pub sub demo available infrastructure Google App Engine is an application platform that allows developers to build web applications and API services on top of Google's infrastructure so what is App Engine at the end of the day it's a very simple way to take your code give it to Google and run it on top of Google's fully managed infrastructure at scale you can build high performance apps which are essentially running inside of a data center that's been proven for almost the last 20 years and using the same technology that's been built for Google search maps Gmail etc it's a serverless infrastructure serverless architecture is an approach that replaces long running virtual machines with ad hoc compute power that comes into existence on request and disappears immediately after use App Engine is popular with web applications and APIs this is where it's mostly used and it has the best use cases so if you're building a user facing website be it in Node.js Ruby, PHP it's really a great destination you can build it deploy it and run it without having to think about the ops and scale it has become a really popular backend for mobile applications as well there are some pretty good examples out there like Snapchat and gaming companies that are built on top of Google App Engine it's really easy to scale up very very quickly on App Engine and it's easy to get it up and running very fast making it a popular choice for mobile backends even without mobile a lot of web applications use HTTP APIs for integration and cloud endpoints things like metering, billing logging, authentication all sit on top of your API you can use cloud functions with App Engine as they fit together to simplify complex tasks and then lastly it's great for line of business applications or LOB applications a lot of these enterprises have a ton of internal sites that need to be built quickly maintained and have a small staff they don't want to build up expertise in multiple deployment systems so here App Engine works well even within Google applications like payroll learning management systems and other internal applications are used are using App Engine so where does App Engine fit into the broader context of compute and when is it a good choice to use let's start all the way at the bottom of the pyramid here we think about virtual machines or when we think about compute engine really what you're looking at is infrastructure infrastructure as a service virtual machines traditionally provided the entry level virtualization as you would lease an entire machine in the cloud so one level up from that or one level up from that so virtual machines traditionally provided the entry level virtualization as you would lease an entire machine in the cloud that provided an isolation level for your apps to run freely and efficiently virtual machines traditionally provided the entry level virtualization as you would lease an entire machine in the cloud one level up in that level of abstraction you're going to have containers if you're an organization that's already familiar with Docker and you want to run a fleet of Docker containers efficiently across a fleet of machines Kubernetes is a fantastic option now the step up from there is App Engine this is where we start thinking less about infrastructure and we start specifically thinking about serverless style architectures you just want to hand over your code and ask Google to figure out the most efficient and the most cost cost effective way to run that code for you really that's basically the nutshell in a nutshell what App Engine does and on top of that if you have a system where you want to respond to very specific events think webhooks or every time a file is written into a gcs bucket you want to have a little bit of code that reacts to that that's where cloud functions become useful instead of thinking of the application or service level here we're looking at the function and event abstraction so let's take a look at the features of App Engine so you have logging you have monitoring you have auto scaling you have load balancing you have SSL and domain and you have health checking so with Stackdriver you have logging and monitoring where Stackdriver provides a logs API and features such as request logs, app logs shutdown logs, runtime logs and audit logs Stackdriver also monitors the health of cloud powered applications performance and the uptime auto scaling makes scaling decisions based on multiple metrics but only works with managed instance groups then you have load balancing a single anycast IP frontends all your backend instances and regions around the world the load balancing is across all GCP products and services SSL and domain offers globally distributed SSL endpoints and built in load balancing to serve your app securely, reliably and quickly to a worldwide audience health checking provides health checking mechanisms that determine which VM instances respond properly to traffic via health checks and legacy health checks the app engine standard environment has two generations of runtime environments the first generation and second generation the second generation run times significantly improve the capabilities of app engine and remove some of the limitations of the second generation run times however there are no plans to deprecate the app engine first generation and the second generation run times represent the future direction of app engine as you can see in the table here the second generation gets all the new features and support for new languages also when it comes to libraries any extension or library can be used in the second generation whereas there was a limitation in the first generation external network access was also restricted in the first generation whereas in the second generation you have full access there was no file system access available in the first generation whereas in the second generation there was no right access to the temp folder language run times you could have only modified for app engine whereas in the second generation you could have full open source run times as well as unmodified despite these differences there are still some similarities like nearly instantaneous instantaneous time to respond to traffic spikes the applications are built using the same build process the same SLA for GA services and the identical gcloud command support now for the flexible environment as the name implies you can have microservices authorization sequel and no sequel databases traffic splitting logging versioning security scanning and content delivery networks all are supported natively the app engine flexible environment allows you to customize the run times and even the operating system of your virtual machines using Docker files the supported languages are Go, Java 8 PHP 5 and 7 Python 2.7 and 3.6 .NET Node.js and custom run times the developers can customize these run times or provide their own run time by supplying a custom docker image or docker file from the open source community so you can basically bring your own docker containers into the infrastructure you can specify how much cpu and memory each instance of your application needs and the flexible environment will provision the necessary infrastructure now this is different from the standard environment because you couldn't do any customization there so custom run times as we have seen has native implementation support so in the flexible environment custom run times are used for an alternative implementation of Java, Python Node.js or Go or to write code in any other language defining new run time environments allows you to include additional components like language interpreters or application servers when you use a custom run time you must write your application code to handle certain flexible environment lifecycle and health checking requests instances are health checked healed as necessary and co-located with other services within the project critical, backwards compatible updates are automatically applied to the underlying operating system VM instances are automatically located by geographical region according to the settings in your project the Google's management services ensures that all of a project's VM instances are co-located for optimal performance the VM instances are restarted on a weekly basis during restarts, Google's management services will apply any necessary operating system and security updates they provide the service of no downtime because they're able to replicate these servers across multiple regions so let's take an example of what happens when you deploy an application in App Engine so you can go from your code to production in a single command that command is gcloud app deploy so first the code is uploaded to Google cloud storage here it's building your code along with Google's runtime then a container is built now you don't have to know anything about Docker or containers here this is all done internally under the hood and then that container gets pushed to Google container registry and now it's available for all of your App Engine applications now a Google cloud load balancer is set up this is a newer feature as the same cloud load balancer is used across all of Google's infrastructure and then they're going to take your application and run it in three availability zones the reason for this is as you mentioned before they may be trouble in an individual zone at any point if they do have a problem your application is going to automatically rebalance and keep serving to your customers and then within each of these zones they're all independently auto scale so as the use of your application grows and as traffic increases GCP is going to automatically provision and release new instances to serve your application so along the way we just talked mainly about application instances and the auto scalers there are also a bunch of other things that are also set up out of the box stuff like when you deploy an app to that individual VM you had to think about at one point or another for example you had to think about logging which stack driver seamlessly integrates now let's take, let's do a demo and then let's take a look at how App Engine looks like on the GCP console we will create a Go server and then show how iterative development and logging on stack driver looks like so there are a few steps here in order to do the demo we will have a short application which just spits out Hello World we are using the Go language here and this is the code that we will be using now there is an app.yaml file that is used to configure the environment now let's go to the demo okay it's demo time so in this demo we will learn how to connect to computing resources hosted on Google Cloud Platform via the web we will learn how to use Cloud Shell and Cloud SDK, G Cloud command so we will create a simple Go server on Google App Engine and we will learn how to update the code without taking the server down so for this we need to have a little bit of familiarity with the Go programming language and also basic Linux text editors and some simple commands so we are here back at the Google Cloud Platform dashboard now the very first thing you need to have the billing setup and then you have to create or select a project now the billing has already been setup here so we will take a look at the projects so that's right here in the drop down and let's create a new project so we'll name this demo 7 and then you can see that there's a project name and there is a project ID now the project name can be a user friendly name but the project ID is something that can't be changed and this project ID stays constant across all of Google Cloud Platform so here it just it just uses the project name and a random ID so we can change this now to just remove this and keep it as demo 7 and let's see if that's allowed so project ID needs to be between 6 and 30 characters so let's just say demo 7 go project now it's creating the project and you can see the notifications here and the little circle here will be spinning until the project is actually created I think it should be done now so let's select our project first so here we have the list of all the projects and ours is the demo 7 so now let's go to the cloud shell you can do that by clicking on this icon here that says activate cloud shell now this provisions the google cloud shell machine and at the bottom here you'll get a new window specifically a new tab as well the command line so let's go ahead and create a new directory using the familiar make directory mkdir hello world then let's change directory to hello world directory and now let's create our first application file so for that we can use any text editor or you can upload a file or you can use github to clone a repository directly to google cloud platform but in this case we'll just use nano which is a built in text editor let's make this a little bigger and here I have a code already written and this is just a very simple code that pretty much there's a main function and there's a handle response writer that just prints line hello world so it just prints the words hello world back in a browser now this code is the same code that's in the slide that's right before the demo started so you can just copy the code from that slide so let's copy this and paste it here and let's save this file to save the content in the file you can press ctrl o ctrl o and let's save this as h dot go and let's exit you can press ctrl x to close the nano text editor so now the second part is a configuration file an app engine application has a configuration file called app.yaml so let's create that as well so we go back to nano and back to our little configuration here so this is the part that's going to be in the app.yaml file so here we can see what the actual contents are so pretty much this is saying that the code runs in the go runtime environment api version go one so in the future there will be other versions available but right now this is the version of go that's available here and then every request to a URL whose path matches the regular expression slash dot store which is basically all URLs it's a wild card and it should be handled by the handle function in the main package so let's go ahead and copy this and paste it here save this as app.yaml exit out of here and now we're done so let's test the application so in order to test the application we can do a test deploy without actually doing a production deploy so for that we can call dev server dev app server dot py dot slash so this is going to simulate an instance and let's see if this worked so now okay so now an instance with process ID 434 has been created so in order to test this we can go to this icon here where it says web preview click on that and preview on port 8080 so this opens up another tab on our browser and it shows the temporary URL here and then it shows our hello world print line so this is successful so far so now while we're here we can also iteratively make changes to this while the server is running here in this window let's create a second tab so now it's provisioning another cloud shell here and we have the command prompt again so let's change directory to our hello world and let's open our h.go file so now let's make a change here so instead of hello world let's say hello world this is demo 7 let's save this now exit out of here and if we go back to our browser and refresh this we should see the difference here hello world this is demo 7 so now what actually happened was while the server is running here it's also constantly checking to see if there are any changes in the application and it's automatically updating it so now let's actually deploy this in production so let's cancel this the control c and let's use our gcloud app deploy command so that's gcloud app deploy app.yaml so now this should create an instance for us and the first question it will ask you is which region do you want to spin this up in so in this case we are saying that we wanted in let's say US East 1 that's 11 and it's now creating the app engine application so it's just gathering all the information and then we'll get to a screen where we have to say yes or no that all looks good so we'll do a yes and now the actual instance is being created so we can see that the code is now being uploaded to cloud storage and then it'll be put in a container and then push to a container registry and then sent to app engine and then attached that to a load balancer and then provide a fault tolerance as we've seen in the app deploy in our slides so this is all happening in the background right now so now it's setting up the traffic split and it's done so here it'll tell you what the URL of the deployed services in this case it's always going to be the project name and then .appspot.com so let's copy this just highlighting it would copy it so let's open a new tab here paste it and see if that works there that works so now this is a global URL that's actually available on the internet with the appspot.com domain so let's go back to our dashboard here at gcp and let's take a look at some of what's actually happening here let's close the tab for the gcloud so now if we go to the navigation menu and then let's go to our app engine dashboard so here this is the dashboard for the specific for the app engine service so here we can see that we have one instance and one version running here so if you look at the services we should be able to see one default service and which has only one version look at the versions we can see that there is one version that's serving a traffic allocation of 100% so look at the instances and we can see our one instance that's currently running with a green checkbox so now actually if we go back to our versions here so Google cloud platform allows us to create multiple versions of our application and then you can split the traffic where you can allocate a different percentage for different versions of your application so this is useful not only as a load balancing mechanism but also if you want to do for example A-B testing or just want to allocate different resources to different parts to different versions of your application so let's go back to the services here and here when we look at our default service here under the diagnose we can see tools click on the tools there are two options logs and debug click on the logs and that takes us directly to the stack driver application so the Google stack driver is a built in monitoring and logging application that application and service that provides us with logs and monitoring so this is all built in and it has been automatically set up to and attached to our new instance all this happened in just a few seconds so we can see that these are just the two log entries for us visiting the website so if we go back to our website here and refresh it and come back to our stack and load newer logs so we can see the new visits that we just performed so let's go back to our app engine dashboard so this shows the summary of our project here and then the region that it has been deployed on so this brings us to the close of this demo now a more typical example of what you might be building in the real world looks like this so you have your front web front ends you have a message queues you have backend processing you have machine language APIs scalable file storage and databases all these have to work together and process your application using APIs and Google Cloud Platform provides all these services and products as well as the infrastructure needed to run your applications failure code in the cloud in response to events originating from GCP Firebase Google Cloud Functions is a serverless execution environment for building and connecting cloud services with cloud functions you simply write single purpose functions that are attached to events emitted from your cloud infrastructure and services your function is triggered when an event being watched is fired cloud executes in a fully managed environment there is no need to provision any infrastructure or worry about managing any servers cloud functions can be written using JavaScript that is Node or Python routines on Google Cloud Platform you can also take your existing function and run it in any standard Node.js 6 Node.js 8 or Python runtime which makes portability and local testing a breeze cloud functions are like glue they provide a connective layer of logic that lets you write code to connect and extend cloud services they listen and respond to file uploads to cloud storage a log change or an incoming message on a cloud PubSub topic cloud functions arguments existing cloud services and allows you to address an increasing number of use cases with arbitrary programming logic cloud functions is automatically authenticated by default to other Google services it has access to the Google service account credential and seamlessly works with the majority of Google Cloud Platform services including cloud vision as well as many others you don't have to manage service accounts or API keys and lug all of this authentication with you cloud vision is an image recognition service offered by Google so you can for example set up cloud functions to trigger an event if someone uploads a picture of a dog in addition, cloud functions are supported by numerous Google cloud client libraries which further simplifies these integrations cloud functions listen and respond to events so what are events in this context cloud events are things that happen in your cloud environment these might be things like changes to data in a database files added to storage systems or a new virtual machine instance being created or if a picture of a dog is uploaded events occur whether or not you choose to respond to them you create a response to an event with a trigger a trigger here is a declaration that you're interested in a certain event or set of events binding a function to a trigger allows you to capture an act on events currently cloud functions supports two main types of events HTTP events and background events the first is HTTP events where they are triggered by HTTP requests and can be used for things like webhooks and creating APIs the other kind of functions are background functions they currently respond to PubSub cloud storage and Firebase events with Stackdriver cloud functions feeds its logs to Stackdriver logging and sends telemetry data memory and execution time into Stackdriver monitoring so you can review how your application is doing using familiar diagnostic tools that work across many Google cloud platform services when an event triggers the execution of your cloud function data associated with that event is passed by the functions parameters and the type of event determines what parameters are passed to your function now let's take a look at a demo in this demo we will see how to write deploy and trigger an HTTP cloud function so these are the steps that we will follow in the demo in this demo we will look at how to create and deploy an HTTP cloud function so as usual the first step is to make sure that you have your billing set up and then we have to create or select a project for that we go to our project drop down and I am going to select an existing project here called App Engine Test now you want to make sure to take note of the project ID in this case it's called Concrete Fusion 228322 so the project ID is something that's unique that's across all the products and services in the Google Cloud Platform so now let's go to the navigation menu and find cloud functions so you can find that here under the compute so the first step is to enable the API so now it's getting started because there are no other cloud functions so far so we get this screen where we can create our first function so we just click on create function this brings us to the screen where we can create a function so now we have a few fields to fill here so the first one is the name of the function so I'll just call it Hello and the memory allocated we'll just leave it at the lowest level here and the trigger here we have a couple of options and in this example we're looking at the HTTP trigger so we leave it with that and then we have a URL here so this is an HTTP trigger this is the URL where that function is going to be triggered from so if you look at the format of the URL you can see that it's the region US Central 1 and then the project ID concrete fusion 228322 and then .cloudfunctions.net and then and then the function name that we specified over here so for the source code in this case we're just going to use the inline editor so we can actually type or update the code right here in the window you can also choose to upload your code or get from a repository or from cloud storage and the runtime here we are using Node.js 6 now in this code this is a very simple code that's just going to print line hello world in the web browser so when the function is triggered you should see the words hello world and the function name is hello world as you can see here which comes from the exports. hello world there are other advanced options where you can choose the region and the timeout and the service account that you want this function to run as so for now we'll leave this as default and create once this circle stops spinning then we'll know that the function has been created ok so now it's done as you can see the green check mark so over here if we click on this there are a couple of options here so the ones that we are interested in are test function and view logs so if you click on test function you'll get to a window where you can specify specific test code here and then have that triggered through the function code and show the output here so we can try something like message so this has to be in the json format hello world test function and you should see just the same output in this case now there are a few other tabs here in the function details so you have the source where you can see the source code of the function the trigger showing that it's a trigger type of http and this is the url and then the general which tells you which gives you a nice dashboard showing the different executions that are happening for that function so now if we go to the trigger and click on the url it'll take us to the actual function being triggered here and you can see the text hello world so this is the successful http implementation of a cloud function so how does all of this fit so on the left you can see the events and triggers at the bottom of the left you can see cloud scheduler now this is a new addition so this lets you schedule your cloud functions on a timer essentially and I guess actually behind the scenes cloud scheduler still uses http or pubsub so those are still fundamentally triggering your functions the same way but you can say hey run this every minute or every 30 minutes or twice a day or whatever your schedule might be you can also have that fully managed in the cloud with cloud scheduler and on the right you can see that over 20 gcp products and services are supported and this is growing just to recap on events triggers and functions here the response to an event is done with a trigger a trigger is a declaration that you are interested in a certain event or set of events binding a function to a trigger allows you to capture and act on events functions and triggers are bound to each other on a many to one basis during deployment so in the table you can see the command line flags the command line flags are very simple and can be triggered with just one line of code so for http it's just trigger http for cloud storage you just have to specify the bucket name for cloud pubsub it's triggered by the topic name and other sources like firebase you would have to specify the event type and the resource serverless so what is serverless you can think of it as an invisible infrastructure you write code that runs in the cloud but you don't necessarily know what machines it's running on you don't necessarily know what you are not managing updating the operating system or installing security patches on the base os or trying to figure out networking and load balancing and attaching disks and there's all kinds of stuff that goes along with running your code that serverless takes care of like auto scaling automatic scaling so as traffic comes into your app and it gets higher and higher Google will scale it up for you and then it scales it down if it goes down in fact they scale it down to 0 which means you only pay while your code is actually running and doing something you get 2 million invocations a month for free but you'll have to pay for the run time this is different than maybe traditional cloud computing where you got some virtual machine that's just running and if it runs for 10 seconds a day but you still paid for the whole day unless you're manually spinning it up and down which is not realistic some of the examples of clients that are currently using serverless you have meetup smart parking and home away using their entire back ends in a serverless fashion Turmer and Simios are using real time data processing and guesswork and incentro are using it for AI and virtual assistants and chatbots so here we can take a look at some of the use cases data processing and ETL here listen and respond to cloud storage events such as when a file is created changed or removed process images perform video transcoding validating and transform data and invoke any service on the internet from your cloud functions then you have webhooks where a simple HTTP trigger you can respond to events originating from third party systems like github slack stripe or from anywhere that can send HTTP requests then you have lightweight apis compose applications from lightweight loosely coupled bits of logic that are quick to build and that scale instantly your functions can be event driven or invoke directly over HTTPS then you can use it for mobile back end you can use google's mobile platform for app developers firebase and write your mobile back and in cloud functions so here you can listen and respond to events from firebase analytics realtime database authentication and storage another common use case is for IoT now imagine tens or hundreds of thousands of devices streaming data into cloud pub sub thereby launching cloud functions to process transform and store data cloud functions lets you do it in a way that's completely serverless now let's take a look at the execution environment so here are run times cloud functions supports multiple languages and run times currently it's node.js6 node.js8 and python updates to the language run times are generally done automatically unless otherwise notified and they include any changes in the base images definition as well cloud functions can start multiple function instances to scale your function up to meet the current load this is concurrency these instances run in parallel which results in having more than one parallel function execution however each function instance handles only one concurrent request at a time this means while your code is processing one request there is no possibility of a second request being routed to the same function instance and the original request can use the full amount of resources that is cpu and memory that you requested because concurrent requests are processed by different function instances they do not share variables or local memory stateless functions as we previously discussed cloud functions implements the serverless paradigm which you can run your code without worrying about the underlying infrastructure such as servers or virtual machines to allow Google to automatically manage and scale the functions they must be stateless meaning one function invocation should not rely on in memory state set by previous invocation however the existing state can often be reused as a performance optimization by storing it then you have cold starts a new function instance is started in two cases when you deploy your function and when a new function instance is automatically created in order to scale up to the load or occasionally to replace an existing instance starting a new function instance allows loading runtime and your code requests that include function instance startup like cold starts can be slower than requests hitting existing function instances so if your function receives a steady load however then the number of cold starts is typically negligible unless of course your function frequently crashes and requires restarting of the function environment the environment running a function instance is typically resilient and reused by subsequent function invocations unless the number of instances is being scaled down say due to lack of ongoing traffic or your function crashes this means when one function execution ends another function invocation can be handled by the same instance therefore it is recommended to cache state across invocations in a global scope where possible your function should be still prepared to work without this cache available as there is no guarantee that the next invocation will reach the same function instance as we saw in the stateless function slide function scope versus global scope a single function invocation results in executing only the body of the function declared as the entry point the global scope in the function file which is expected to contain the function definition is executed on every cold start but not in the instance has already been initialized you can assume that the global scope has been executed exactly once before your function code is invoked in a new function instance and on every subsequent creation of a new function instance however you should not depend on the total number of or timing of global scope executions as they depend on the order scaling which is managed by Google now the function execution timeline a function has access to the resources requested CPU and memory here only for the duration of function execution code that is run outside of its execution periods is not guaranteed to execute and can be stopped at any time therefore you should always signal the end of your function execution and avoid running any code beyond it let's look at the execution guarantees your functions are typically invoked once for each incoming event however Google functions does not guarantee a single invocation in all cases because of differences in error scenarios the maximum or minimum number of times your function is going to be invoked on a single event depends on the type of your function HTTP functions are invoked at most once this is because of the synchronous nature of HTTP calls and it means that any error on handling function invocation will be returned without retrying the caller of an HTTP function is expected to handle these errors and retry if needed background functions are invoked at least once this is because of the asynchronous nature of handling events in which there is no caller that waits for the response and could retry on error an emitted event invokes the function with potential retries on failure if requested on function deployment and sporadic duplicate invocations for other reasons that is even if retries on failure were not requested when using multiple functions just to recap you want to make sure not to rely on shared memory variables file systems or state to share data across deployed functions you can use storage services like cloud data store cloud firestone or cloud storage alternatively you can invoke one function from another using their appropriate triggers or you can chain cloud functions so how do you build more complicated apps once you start to get to the world of 2, 3, 4, 5, 60, 70 different services and products how do you connect them how do you coordinate them so let's first talk about chaining functions so we got a bunch of these functions and we want to put them together here we have cloud pubsub pubsub is a publisher subscriber messaging system so in this example we have a new employees data getting inserted into firestore there is a cloud function listening for new employees as we want to trigger some other events when this happens a message is sent to cloud pubsub here is a new employee Brett location New York City now a new employee here needs a building business badge created for them so there is another cloud function that is listening to the new user topic and triggers the process to create a new badge for the new employee the same message can also trigger a third party system like an IT helpdesk system in this case and then for cloud monitoring and logging stack driver is built into this error logs and console logs are written automatically out of the box by default cloud functions for firebase is optimized for mobile developers cloud functions for firebase provides a way to extend the behavior of firebase and integrate firebase features through the addition of server side code it's optimized for specifically firebase SDK to configure your functions through code it's integrated with firebase console and firebase CLI the same triggers as google cloud functions are available plus in addition you have firebase realtime database firebase authentication and firebase analytics products can be used as triggers now let's take a look at the demo in this demo we will look at background services so here the objectives are to write and deploy a background cloud service and which is going to be in pubsub and then trigger that function by publishing a message to a cloud pubsub topic so these are the steps for this demo which we will go through on the UI now this is the code that we will be using in the demo and the command line in order to test the background service function in this demo we will create a google cloud function for a background service and that background service is going to be for the cloud pubsub and we will send a message to a topic on cloud pubsub and have a function triggered by that message so let's take a look at how we can do that so now we are back to our google cloud platform UI and as usual the first step is to make sure you have your billing set up and have a project created or select an existing project so we go to the project drop down here and in my case we are selecting the app engine test that we used for the previous demo so again we just want to take a note of the project ID which is concrete fusion 228 322 the project ID is unique across all google cloud platform products and services so now let's go to the navigation menu and find cloud functions so now let's go to create function and this time we will name this pub sub demo and for the trigger we will choose cloud pubsub and in this case we need to specify a topic since there are no topics set up here let's create one and now for a new pubsub topic the format is going to be projects and then the project ID topics and then the topic name so we will call this pubsub topic and create and we can choose to use the inline editor no js runtime and here we can write or update the code so instead of the boring code and the example code that's provided let's add our own code here so this code is provided in the slide previous to this demo so it's the same code that you can copy and paste in your projects so this code is a background cloud function that's to be triggered by pubsub so in this function it's looking for a message where the message represents someone's name a person's name and it's simply attaching the text hello comma and then appending the message which in this case it's expecting a name so we're going to send that message out so let's copy this code and paste it here ctrl v and we will keep this as the function to execute hello pubsub hello pubsub and create so now the function is being created and being deployed when this circle stops spinning and we get a green checkbox like here then the function has been successfully created and deployed so here we can see that it's done and we are ready to take a look at that let's just look at the details of that function by clicking on it and drilling down and it gives us the same dashboard for that specific function there's currently no data because that function hasn't been triggered yet let's check the trigger the trigger type is cloud pubsub and the topic is pubsub topic and we can take a look at the source code here and we can have a testing block here as well so now let's test our new function by publishing a topic and a message so for that instead of doing it in the block here let's launch our gcloud shell by clicking on the icon here we activate the cloud shell and the google cloud console is open here at the bottom now we just want to make sure that the correct project has been selected so in this case it's our concrete fusion 228332 project ID so now let's go back to our code here now this command gcloud pubsub topics publish the topic name and with the flag message and the person's name so this command will publish to this topic with this message so let's check the name of our topic here so this would be pubsub topic and let's say the name is james so let's copy this and paste it here ctrl v and hit enter so the message has been published and this is the unique message ID so now let's see if that actually worked can go back to general to our function and click here and check the logs now these logs are provided by stackdriver which is another google cloud platform product that provides monitoring and logging across all google cloud product and services so here we can see that the function execution has started for pubsub demo and then we can see the message here hello comma james that was sent so this has been a successful background function so just to recap what did we just do so we created a message on google cloud pubsub and then we created a function that should be triggered whenever someone publishes a message to that topic so every time someone publishes a message to that specific topic a function will be triggered so that's the background function that we created what does that function do that function takes the message from that topic and it appends hello comma to the message because we are expecting someone's name and then it sends out that message so that's what we just saw here so this concludes the demo options and database solutions that suit a hobbyist to a fortune 500 company I'll walk through a few of these options and provide a way to choose the right solution for your need almost all applications need persistent durable storage of some sort whether it's user accounts images events documents, logs or anything else different applications have different storage needs Google Cloud Platform offers 7 storage options Bigtable Cloud Datastore Cloud SQL Cloud Storage Cloud Spanner Persistent Disk and BigQuery BigQuery is a bit special because it's both a storage service and a powerful analysis tool so why Google Cloud Platform there are other options such as Microsoft Azure or Amazon Web Services or individual databases such as Microsoft SQL Server MongoDB etc the advantage here is a single API can be used across different storage classes it's scalable to exabytes of data and designed for a 99.999999999999% durability depending on the type of Datastore that's chosen there is very high availability across all storage classes and the time to first byte is in milliseconds as long as the time to first byte as long as the Datastores are on the Google Cloud Platform they all enjoy the same reliability and infrastructure that Google provides as well as strongly consistent listing now there are two categories of storage services structured and unstructured if the data you want to store can be organized into a table structure with columns and rows then it's structured data examples of structured data are user profiles event logs sensor measurements sales records and stock trade data structured data comes in many different shapes, sizes and usage patterns there's a great diversity in the way that you can store and interact with it Cloud SQL Cloud Datastore Cloud Bigtable Cloud Spanner Persistent Disk and BigQuery are all structured data unstructured data is also available in Google Cloud Platform this is simply called Google Cloud Storage you just provide a sequence of bytes to store a place to store it and a way to identify it and Google just stores them for you there's no way of knowing or there's no insight into how this data is actually internally stored all we know the way we can interact with the data is through buckets and objects an object is similar to a file however there is no file system here now let's take a look at some of these options in detail Persistent Disk is a fully managed block storage that's suitable for virtual machines and containers so if you have Google Compute Engine instances and for Google Kubernetes Persistent Disk is sort of like having your own hard drive or an SSD disk available for good for snapshots and data backup common workloads are for virtual machines and sharing read-only data across multiple virtual machines it provides a rapid durable backups of running virtual machines Cloud Bigtable is a scalable, fully managed NoSQL wide column database it's suitable for both time access and analytics workloads it provides a low latency read-write access a high throughput analytics and native time series support this is sort of similar to MongoDB where it's a NoSQL database that's great for IoT, finance, ad tech, geospatical datasets graphs and personalization Cloud Datastore is a scalable, fully managed NoSQL document database that's useful for web and mobile applications it stores semi-structured application data in a hierarchical manner and in a key value format it's great for user profiles product catalogs and to save game state Cloud SQL is a fully managed MySQL and PostgreSQL database service it also enjoys being built on the Google's infrastructure which provides reliability and scalability it's good for web frameworks structured data OLTP workloads it's commonly used for content management systems BI applications CRM and geospatical applications Cloud Spanner is a newer offering from Google that helps mission critical applications it provides a mission critical relational database service with transactional consistency global scale and high availability it's useful for financial services global supply chain retail and ad tech Google BigQuery is a scalable, fully managed enterprise data warehouse with SQL and fast response times it's good for exploring big data sets and petabytes scale applications and workloads the common workloads are analytical reporting on large data big data pricing using SQL and data science and advanced analysis now finally cloud storage is the unstructured data store that Google provides it's a scalable, fully managed, highly reliable and cost efficient object blob store it's good for storing images pictures and videos objects and blobs and any unstructured data common workloads include archives, backup disaster recovery storage for custom data analytics pipelines and storing and streaming multimedia mobile storage options are also a newer offering however Firebase and Firestore have been with Google for a while some of these specific products are still in beta cloud storage for Firebase is for mobile and web access to Google cloud storage with serverless third party authentication and authorization Firebase real time database provides a real time no-SQL JSON database Firebase hosting is a production grade web and mobile content hosting for developers and finally Cloud Firestore is a no-SQL document database that's for mobile applications with all these different options how do you choose what's the right product for your need to figure this out the first question you want to ask yourself is, is your data structured if the answer is no then is it a mobile application if it is then cloud storage for Firebase is the right product if not simply cloud storage if your data is structured then is your workload analytics if the answer is yes then do you need updates or low latency if yes Google cloud big table if not BigQuery if your workload is not analytics then is your data relational if it is then do you need horizontal scalability and if it's not is it a mobile application if it is a mobile application then cloud firestore for Firebase would be the right product if not cloud data store if you do need horizontal scalability then cloud spanner would be the right product if not cloud SQL the first part explores the basics of cloud storage and in the second part we'll look at the more advanced features so what is cloud storage it's a fully managed product that's provided by Google with its usual high reliability and cost efficient infrastructure that provides storage space for object and blob stores it's great for storing images, pictures videos any other sort of unstructured data typically it's also good for streaming multimedia storage for custom data analytics pipelines archive backup and disaster recovery integrated into all the apps and products and services offered by Google it's optimized by price performance across four storage classes you can manage that with the object lifecycle management you can access data instantly from any of these storage classes the durability here cloud goes up to 99.99999 that's up to 99 and the availability ranges from 99% to 99.9% SLA depending on the type of storage class that you choose and the amount of money you're willing to spend for your storage and also the scalability can go up to exabytes of data with a millisecond responsiveness so data in cloud storage is stored in buckets and data is stored as objects however unlike directories and folders you cannot nest buckets when you create a bucket you specify a name geographic location and a default storage class these are the only three properties that are required the location depends on where you want the bucket so store frequently access data such as data used for analytics as regional storage and you can use dual regional location when you want similar performance advantages as regional locations but with added geo-redundancy when you want to serve content outside of the google network and distributed in large geographic areas or when you need your data to be globally redundant you can use the multi region location bucket names that contain dots need to be valid the main names we will look into this in detail in the advanced lecture there are four types of storage classes multi regional storage near line storage and cold line storage multi regional guarantees two replicas by default at least 100 miles apart it's for the highest availability of frequently accessed data like website content and business continuity regional is for data accessed frequently within a region it's great for storing data for analytics or compute within a region it has the lowest latency possible within a region and it's cheaper than multi near line and cold line are for infrequently accessed content and archival storage this is comparable to glacier storage at amazon aws if you're familiar with that near line is typically for less than once a month usage and cold line is for less than once a year usage so each bucket has a default storage class that you select when you create a new bucket the objects that you add to this bucket are automatically assigned this storage class however this can be changed and only new objects added to the bucket will be assigned the updated storage class you can change the storage class for individual objects with an api you can't do this from the ui however there are some restrictions on what you can change it to based on the bucket location remember when creating a bucket there is a location and a storage class these are still two separate things now let's take a look at a demo on how to create storage buckets in this demo we'll learn how to create buckets so here we're back to our familiar google cloud platform user interface so before you begin as usual please make sure that you have your billing information set up as well as your project created or select the correct project so here I have my project selected if you need to you can create a new project by clicking on new project in this case I'm going to choose my existing project which is displayed here so now let's go to the navigation menu and look for cloud storage it's under storage and storage so we click on that and it takes us to the area where you'll see a list of buckets here here I have already three buckets from a previous project so let's create a bucket you can do that by pressing the create bucket button here click on that and as we have discussed in the lecture there are three main properties that you need to specify the name of the bucket the default storage class and the location so let's call our bucket name bucket 090 that's not taken already so it has to be a globally unique name and we'll just leave it as multi-regional and the location in the United States and press create that's it now we're in the bucket details so if we go back here by pressing the arrow key now we can see a list of all the buckets in our project in this case we can see we created bucket here let's look at a demo on how to get the bucket information now that we have created a bucket let's get some more information about that bucket through the command line so let's initiate and activate our cloud shell by clicking on this icon here this opens the google cloud console and we can write command lines here now the commands we'll be using here are listed in this file and these same commands can be seen on the slide previous to this demo so to list the buckets we can just simply use the ls command after the gsutil command so just copy that and paste it here ctrl v enter and now we can see that our bucket is listed here along with the other three buckets that are here so now let's take a look at the bucket size so for that this is the command gsutil du-s and then the name of the bucket ctrl v and then let's type in the bucket name which is bucket 190 hit enter and this will tell you that the bucket size is currently zero because there are no objects inside the bucket let's take another look at the next command so now we can change the default storage class of the bucket with this command which is gsutil def storage class set and the storage class that you want to set it to and then the bucket name so let me just copy this part here copy paste and let's so currently we can see that the default storage class is multi-regional so let's change it to let's say coldline and then we are to specify the name of the bucket we want to change that for which is bucket 190 so now that has been set let's see if that worked by refreshing the screen here by clicking on the refresh button and we can see that we have changed to coldline let's take a look at a demo on how to upload, download and list objects now let's add some objects to our bucket for that let me create a new bucket create bucket again and type call this bucket 191 and leave everything as default press create so now we have another bucket on the back button so now let's click on the bucket here and we can see the details of the bucket so currently there are no files here so if you want to add files or objects to your bucket you can directly drag files into this window here or you can also upload files by clicking the button here for example I click this and point to the file that I want to upload this case it's cat1.jpg open and the file is being uploaded here that's finished close this window and we can now see that there is one file in this bucket let's see that the object that has been uploaded also inherited the storage class the multi-regional storage class which is the default storage class of the bucket now we go back one level and we can also upload objects via the command line so in order to do that let's first create a file really quick in the command line so we can use the nano text editor type nano hit enter that opens the default text editor I'll just type in test as the text ctrl o to save the file let's select a name let's just say it's the test file dot txt and hit enter and then ctrl x to exit and now we're back to our console so now if we type ls we can list the files that are currently in this project so here we can see the test file dot txt that we just created so now let's upload this test file dot txt to our bucket in order to do that the command is gsutil cp the object location and then the destination bucket again these commands are on the slide where before this demo starts so let's copy just this part here copy ctrl v to paste and now we type the name of the file test file dot txt and then the destination bucket gs colon slash slash and the bucket name is bucket 191 and hit enter so copying one file completed let's see if that worked we click on the bucket 191 to see the details and now we have two files with the new file added here now we can also download objects from the bucket to our local directory here that it's the similar command cp and then we start with the bucket name slash object and then the destination so in this case let's just download test file dot txt back to our folder here so let's create a sample folder first the command is mkdir to create a new directory oops not here it's down here kdir let's say tstdir test directory let's see if it was actually created ls to list files and we can see that tstdir folder here now to copy our file from the bucket to local we use the same command here copy paste and gs colon slash slash bucket name bucket 191 slash test file dot txt and then we use tilde slash and the new directory that we created which is the tstdir and hit enter and now that now this file has been downloaded from the bucket 191 to our local directory testdir let's see if that actually worked so let's change directory to tstdir and then list the contents with ls and we can see the file here locally testfile.txt we can also delete objects from the bucket from the command line so for that we use the rm command so for that let's remove the testfile.txt object from the bucket bucket191 so I copy that command paste it here gsutil rm and gs colon slash slash bucket191 slash the name of the object which is testfile.txt and we hit enter and that object has been deleted so let's refresh this bucket here by clicking on refresh bucket and we can see that that file no longer exists let's take a look at a demo on moving and deleting buckets in this demo we'll see how we can move objects and delete buckets so currently we have our bucket191 with one object cat1.jpg so let's create another bucket here let's do it from the command line from the command line you only need to specify the bucket name you can optionally specify the storage class the default storage class as well as the location so in this case I'm only going to specify the bucket name so it's gsutil mb gs colon slash slash bucket name in this case I'm going to call it bucket192 hit enter now my bucket has been created so let's refresh this top panel by hitting the refresh button and we can see that our new bucket has been created with the same default storage class so now let's copy all the files and all the objects from bucket191 to bucket192 so in order to do that we can use the copy minus r flag so this is the command for the bucket so copy that and paste it here so to copy files from your old bucket to the new bucket the command is cp minus r and then the source bucket and then the destination bucket so you put in an asterisk here to specify that you want to copy all the files and all the objects from the source bucket to the destination bucket so let's copy this command and move our objects from bucket191 to bucket192 so copy that paste it here and then fill in the rest and fill in the rest which is gs colon slash slash bucket191 slash asterisk space gs colon slash bucket192 and we hit enter so we can see that the operation has completed and one object has been copied so let's see if this is true by refreshing the top panel and then clicking on bucket192 and now we can see that the file has been moved or actually copied here now we can delete the files from this old bucket so that it would actually be more like a move and not a copy so to delete files or objects from the old bucket we can use the command gsutil rm minus r and then the name of the bucket so let me copy that copy and then fill in the rest gs colon slash slash bucket191 the operation has completed and now let's go back and refresh our top panel and now we can see that bucket191 has been removed from our project now let's take a look at a demo on how to view edit object metadata in order to check the metadata of an object we first browse into the bucket identify where the object is and click on the more information icon that's all the way on the right hand side so we click on this and select edit metadata so here now we can edit the metadata for this specific object you can also add items here for your own custom key value pairs and save if you wish to you can also check this on the command line you can also do this from the command line using the gsutil ls minus l command bucket name and the object name so let's try that here going to copy this paste it here and fill in the bucket name which is bucket192 and the object cat1 dot jpg hit enter and now we can see the metadata for that object this can also be edited using the command here by specifying the key value pair of the metadata and then specifying the bucket name and the object name using the set meta minus h command where we'll discuss a few more advanced topics object lifecycle management to support common use cases like setting a time to live archiving older versions of objects or downgrading storage classes of objects to help manage costs cloud storage offers the object lifecycle management feature with this you can automate these tasks with the lifecycle management configuration the configuration consists of a set of rules which apply to the current and future objects in a bucket each rule has an action and a set of conditions if you have multiple conditions all of them have to be met in order for the action to take place actions may not necessarily be performed right away by cloud storage you can track lifecycle actions with logs and by enabling cloud pub sub notifications for cloud storage for your bucket examples of these actions include changing storage class to archive or delete an object after a period of time let's take a quick look at the UI on how to set these object lifecycle management options in this demo we'll look at how to enable object lifecycle management so we're now back at our familiar UI of Google cloud platform so let's make sure that we are in the correct project that we want to select or create a new project if you need to this we are in the app engine test project and let's go to the cloud storage section so we go to the navigation and under storage we have cloud storage now in order to enable now we can see the list of buckets that we have in this project and here we can see the details of the buckets and we can see this lifecycle column where it says none which means that it's not enabled here you can see an example of a bucket where the lifecycle management has been enabled so in order to enable it for our bucket bucket 192 we click on the none and that will open here where we can add a rule so when you click on add rule there are two parts here there are object conditions and there are actions so we can select any of these different object conditions for example age so that we can we can trigger this action if objects are of a certain number of days older or so or based on the creation date based on the storage class based on the number of new versions of that same object that have been created and it's live state or whether it's been archived or it's currently live and then you can select an option here to either set it to a different storage class or to have it deleted and then you press save and that would save and enable the object lifecycle management for that bucket object versioning allows for the retrieval of objects that are deleted or overwritten when you delete an object with it's called it gets archived as a different version you can restore archived versions or permanently delete them this is a feature that needs to be turned on for objects because it adds additional costs these costs can be managed and reduced with lifecycle management the versioning works by keeping track of the object's version and the version of the object's metadata you can add a retention policy to a bucket to specify a retention period if a bucket has a retention policy objects in the bucket can only be deleted or overwritten once their age is greater than the retention period a retention policy retroactively applies to existing objects in the bucket as well as the new ones that you add to it you can also lock a retention policy to permanently set it on the bucket this is done using a special account in order for compliance once the lock is done a regular administrator won't be able to remove or reduce the retention period you can't delete a bucket with the locked policy and you can't delete objects or remove them you can however increase the retention period or place additional holds on it this is again to comply with regulations you can use buckets to host static websites when you do that the name of the bucket needs to be the name of the C name record the objects inside the bucket are the web pages for the site so for example the URL is going to be http colon slash slash the bucket name slash the object name where the bucket name is for example dot com and the object name is the actual file index dot html you can make all the files in the bucket publicly accessible or you can separately set permissions to each file you can also set properties to the bucket to assign a page as a default like for example index dot html and then error pages like for example error 404 page not found when hosting static assets for a dynamic website you do not need to create a C name record and point to a bucket with the same name as you do for a static website in this case the URL will still be cloud dot google dot com and then your bucket name let's take a look at a demo where we create a bucket for a static website so let's host a static website using buckets so we are back to our familiar user interface for the google cloud platform and let's check and make sure that we are in our correct project or we can create a new project if we need to in this case I am in the app for this project now we open the cloud storage browser from the navigation menu let's click on the navigation menu go down to storage and click on cloud storage here we can see a list of our buckets in this project so now we create a new bucket create bucket press this button now for the name of the bucket in order to host a static website it has to be the domain name of your website including the www and the extension so in this case it's going to be www dot appcode world dot com now this domain has to already be verified and you have to prove that you are the owner of the domain before you can create a bucket with this name on google cloud platform so you press create and also if you want to have a dot in the name of a bucket it has to be a proper domain name press create and now we get to the bucket details so I have a few files that I have prepared before and this is just a basic index dot html file which is usually the first page in a website and just a content html file and an error 404 file so in order to upload these files and the two image files you can just drag and drop these files right here into the bucket and they get uploaded so now we can go and check the permissions for this bucket and so right now this bucket here that we just created is not available to the public so it can only be seen within the scope of this project and within the scope of google cloud platform so in order to change the permissions we go to the more menu and then edit bucket permissions so here we can add a member called all users and assign a role of storage and storage object viewer click on add and now you can see that public access has been granted to the bucket now let's assign specialty pages now we go back to the so now let's assign specialty pages to our website so for that we click on the more menu again and this time we click on edit website configuration so now here we have a chance to specify the main page as well as the 404 not found page so in our case it's the index.html page and error 404.html for the 404 page you press save and now we can test our site so when you go into the bucket details you can see all the files which are your objects in that bucket and we can also see that all the files have public access granted because they inherited the same permissions from the bucket and if we click on the link here this will open a new tab in the browser showing the website the index.html now you can see that this is still attached to storage.googleapis.com and that's because the CNAME has not been assigned to this bucket yet so we'll see that in the networking section of the lecture notifications there are three types of notifications that can be set so you can know when an object has changed this is usually useful to notify an application when an object is added updated or deleted in a bucket for example when you add a new picture to a bucket an application could be notified to create a thumbnail so to start watching a bucket for change notification events you can use this command watch bucket which would create a notification channel that sends notification events to the given application URL for that given bucket so you can see the workflow in the diagram below the application on the left hand side sends a watch request to Google cloud storage for a specific bucket when a user makes a change to the objects in that cloud storage bucket a notification change is triggered and through that notification channel a notification event is sent to the application for further action another way of doing this is through cloud pops up cloud pops up notifications send information about changes to objects in your buckets to cloud pops up the information is added to a cloud pops up topic of your choice in the form of messages it's implemented by adding a notification configuration rule to a bucket that specifies the topic trigger events and notification details this can take up to 30 seconds to begin sending notifications but then it guarantees at least once delivery to cloud pops up the third way of doing it is through cloud functions you can create a simple JavaScript function and make sure that it resides in the same project as your bucket this way you're able to create a lightweight standalone function that's independent of other products and services or MySQL and Postgres Google Cloud Platform offers a fully managed MySQL and Postgres database service that is built on Google's infrastructure this is specifically good for web frameworks structured data OLTP workloads and commonly used for WordPress, blogs websites content management systems CRMs e-commerce applications and business intelligence applications a simple and fully managed MySQL and Postgres database service is not a full replacement for a standalone one this means that it offloads mundane database management tasks to Google but a DBA replacement either for example it doesn't offer query optimization MySQL is the world's largest and most popular web database Postgres allows for a complex database design complex rule set use of stored procedures and procedural languages on the server side Cloud SQL offers MySQL and Postgres database services it doesn't mean that they're offering MySQL compatibility or Postgres compatibility they're offering the exact same community versions or the vanilla versions that you can download and set up yourself on your own local server Cloud SQL is easy to use it doesn't require any software installation and provides horizontal scalability out of the box we should note that the scalability is only on a single node here unlike Cloud Spanner which we will look at in a later lecture you can also access the database instance directly from within GCP via the Google Cloud console or via an IP or a proxy to run a database there's a lot going on under the hood even if you're using managed hosting you should have to worry about upgrades, patches, maintenance etc Cloud SQL automates all your backups, replication patches and updates while ensuring a greater than 99.95 availability anywhere in the world monitoring is done automatically and hooks are provided for external tools like Splunk whereas traditionally you would have to create APIs and do a lot of setup high availability clusterer a feature that's provided with Cloud SQL it provides data redundancy the configuration is made up of a primary instance the master and a secondary instance in a separate zone both of them have to be in the same region data is kept in sync between the instances via semi synchronous replication the failure is initiated if the primary instance is unresponsive for approximately 60 seconds or if the primary zone itself is failing there is such a thing as a replication health check that's done before and after there is something called a replication health that's constantly checked and if the replication lag is over 10 minutes then Cloud SQL will not initiate a failover automatically here in this diagram we can see what the typical setup looks like you can see the primary instance on the left and the failover instance on the right please take note that they're both in different zones but they're both in the same region daily automatic backups can be set up using a backup window this is to minimize any downtime or hinder any performance backups are also snapshots at block level instead of entire terror dumps backups are available on a rolling 7 days by default when you need to restore a backup you can do it on an hourly basis for granularity or you can enable binary logging in which case you can restore up to a point in time failure and up to a certain transaction now let's take a look at a demo in how to set up an instance where we'll see all the features that we just talked about let's take a look at how to create a cloud SQL database typically a MySQL database instance using Google Cloud Platform so for that we're now back on our familiar Google Cloud Platform UI so let's create or select a project first so in this case I have my project already created so I'm going to select Cloud SQL demo which is my project name and we take note of the project ID which is common for all products and services across Google Cloud Platform now we go to the navigation menu and find Cloud SQL which is under storage click on this and now we have a chance to create an instance, press create and you're offered two choices here MySQL and Postgres so we're going to choose MySQL here click on this and now we get a host of options that we can choose so let's start with the instance name it's going to call it demoDB now let's choose a root password for the instance and then we can choose the location of the data center so the region is the general area where you want to place your master database in and then the zone is the specific data center within that region so in this case let's choose East 1B and we can look at some more configuration options here when you click on advanced so here we can select to see if we want to use a different database version with this and we can also choose the type of machine that would be provisioning it so the VM behind it can be changed so currently it says that a one CPU machine has been selected with a memory of 3.75 GB let's select that to something a little more powerful let's say a 4 GB standard database sorry a CPU now we can see that there is a little more performance improvement here but if we look down here we can see that the disk out throughput and the number of IOPS the input output performance is really bad it's all the way down here so when you increase the GPUs you also want to update your storage capacity so if we add a little more let's say another 0 here we can see that the performance here has increased so google cloud platform provides this calculator for us so there are some formulas behind this calculation which we don't have to worry about and we can just play with the numbers here to make sure that we are getting the best possible performance so still that's not very good so if we choose 1000 here that might be a little too much so in this case let's try 400 so that looks a little bit better now the next step is the backups and high availability so as we have seen google cloud platform offers backup and failover services so we can have automatic backups during this specific window that we choose on a daily basis and we can also allow for binary logging so that we can get point in time recovery now high availability this is a feature for data redundancy that google cloud platform also offers so if we say we want to have failover instance also created that would be done at the same time our master is created now we do want to note that a failover database is actually a separate instance of the same size of the database and the specs that's being created in parallel to the master this means that the cost associated is also going to be double now we can also finally choose the maintenance schedule in order to allow google engineers to upgrade our server or apply patches to the operating system and do other server maintenance tasks we can leave this as it is or we can specify a certain day when the workload is going to be less on our database so we press create to have all these options and have our database set up now this will take a few minutes and when the spinning circle is spinning and there's a green check box that's when our database instance has been created we can see that our main database has been created and the failover is still being created now you can also take note that the failover is being created in the same region but in a different zone now we can see that the failover database has also been created you can take a look at the more options for our main database and we can see that from here we can also create a read replica if we want as we discussed in the lecture now if we click on the main database we can see the details of the instance that we just created and we can see the connection details here including the IP address and the instance connection name we can connect to our instance through the cloud shell by clicking on connect using cloud shell click on this and the Google cloud shell opens down here and a connection automatically gets established to our database so you can see that the command that's required here automatically provided to us so if we hit enter that automatically should get us connected to the database now since we are not using a proxy the IP address from where I'm connecting to the database from is being white listed automatically for 5 minutes so that during this time a firewall that's built in is giving me access to it now we press our password hit enter and then now we have our MySQL command prompt from here we can also manage the other aspects of our database instance like connections users databases backups replicas and other operations performance it's good to have one large database but it can get dominated by certain types of applications which demand a high number of reads for this you can create a read replica which is basically a read only copy of your master database the common use cases for this is if you have an analyst team or an analytics application that's creating reports read replicas use asynchronous communication compared to the failover which uses semi synchronous this is so as not to hinder the performance of the primary database so in practice the data on the read replica might be slightly out of date compared to the master you can also have external read replicas and replication from external servers cloud sql proxy let's say when you're working at a starbucks or a less secure public environment and want to connect to your instance you can do so by setting up a firewall and creating temporary rules or use cloud sql proxy the cloud sql proxy works by having a local client called the proxy running in the local environment your application then communicates with the proxy with the standard database protocol used by the database the proxy uses a secure tunnel to communicate sort of like a vpn with its companion process running in the server you can see in the diagram how this actually works it's extremely convenient if you'll have to constantly connect to your instance that offers transactional consistency at global scale what is cloud spanner so it's a relational database service that's provided by google cloud platform it's a sequel like database that provides sql commands and a relational table structure with tables columns rows etc it's specifically built for mission critical applications and high transaction services it offers managed instances with high availability to transparent synchronous built-in data replication the common examples for its use are financial services global supply chain retail and advertising so some of the features of cloud spanner it's a fully managed database service with global scale so it automates and provides synchronous replication with or across regions for availability and it's maintained by site reliability engineers from google it uses nodes for scalability so in our previous examples with cloud sequel and postgres and cloud sequel service that google cloud platform provides that's based purely on a single node where you can have a maximum of 32 cpus so here spanner is based on multiple nodes and each node provides up to 2 terabytes of storage however there's no backup solution available as of now so you'll have to back up spanner by perching all the data from the database spanner is being used and is still being used internally at google for really large products like adwords, google play google maps etc so let's take a look at some of the differences between a traditional database and spanner so even with the traditional databases there are mainly relational databases and non-relational databases which are more like non-sequel no-sequel databases so spanner forms a hybrid between these two between a traditional relational database and a no-sequel database so it provides the benefits of both of them like having the schema strong consistency sequel high availability and automatic replication so let's take a look at what the schema and the data model looks like so the tables look like relational database tables in that they're structured with rows columns and values and they contain primary keys and foreign keys and the tables are related to each other via the keys the data is strongly typed which means that you must define a schema for each database and that schema must specify the data types of each column of each table now here's where the difference comes in cloud spanner divides your data into chunks called splits which can move to different nodes or different servers these splits are automatically managed by cloud spanner and different parts of your table or data is handled by different servers another main difference in cloud spanner is the ability to interleave data an interleave table is a table that you declare to be a child of another table because you want the rows of the child table to be physically stored together with the associated parent so here they are literally interleaved the separate tables so in the diagram we can see three separate tables one called singers two called albums and three called songs so you can see them in with different color codes but here in the grid all three tables and all the data from those three tables are combined this kind of special way of storing data is what helps spanner become efficient and be able to scale horizontally so cloud spanner automatically creates replicas for each database split so there are three types of replicas read write, read only and witness so certain parts of the data in the replicas is owned by different nodes there is a custom algorithm that automatically manages data for any regional configuration cloud spanner maintains three read write replicas each within a different Google cloud platform zone in that region each read write replica contains a full copy of your operational database that is able to serve read write and read only requests cloud spanner uses replicas in different zones so that if a single zone failure occurs your database remains available in multi region configurations they contain two regions that are designated as read write regions each of which contains two read write replicas having more copies of your data makes the data more available to clients that want to read it now when it comes to nodes adding a node does not increase the number of replicas which are fixed for any given configuration even though cloud spanner increases the resources each replica has in that instance adding nodes gives each replica more cpu and ram which increases the replicas throughput that is more reads and writes per second can occur even though cloud spanner replicates across geographically distant locations you can still use cloud spanner as if it were a database and a single machine transactions are guaranteed to be serializable and the order of transactions within the database is the same as the order in which clients observe the transactions to have them committed this is what provides a single database experience now let's take a look at a demo where we create a cloud spanner in this demo we learn how to create a cloud spanner so we are now back here at our familiar google cloud platform interface so as usual the first step is to create or select our project so that's down here click and currently I have selected my project which is cloud spanner demo click on that and now let's navigate to cloud spanner click on the navigation menu and under storage we can see spanner click on that so now let's create an instance by pressing the create instance button let's give our instance a name spanner instance and we can choose either a regional or a multi-regional configuration and we can for now select regional and let's say europe north ok so now we can select the number of nodes and to provision our instance so in this case I'm just going to say three nodes and the cost here you can see the estimation per node and you can calculate that also if you want to know what the performance looks like you can also take a look at how the different number of nodes would matter so let's go ahead and create our instance press create and our instance has been created so the next step is to create a database so let's press the create database button let's have a name for the database demodb continue and now we can create the schema for our database so for that we can create tables so let's add a table our first table by pressing the add table button and here we can have a name for our table let's say singers as in our demo press continue and then we can add columns the first column which would be singer id and let's add another column singer name of type string now we continue and now we can set the primary key to single column and then singer id let's create that and now we have our first table so let's add a second table by pressing on the add table button again and this time let's choose albums and then now we can now we have another option here to interleave this table with another table so let's check this and then it automatically chooses the other table here which would be the parent so in this case the singers table and then we can also enable cascade delete so that if rows are deleted in this table then they will automatically be deleted in the related tables just continue and then we can add some columns here let's say album name or album id first and album name of type string continue and set the primary key and then press the create button so now we have created two tables here to create this schema so now our database has been set up with our two tables here so you can see there's a query tab and that is our albums table with the fields that we selected or created and our singers table with our fields that we created here and we can run queries SQL queries on these tables from the query tab here's a real world example Quizlet app is a popular app that provides quiz and notes for students it's a very demanding app and here's a comparison chart of using a MySQL database versus Cloud Spanner as we can see once the throughput with the number of queries per second reaches 8000 that's 8000 queries per second we can see that MySQL which was previously providing a very low latency suddenly starts to peak and it becomes almost infinite and crashes as with Cloud Spanner you can continue to maintain the latency that you want by just adding nodes so initially there were 9 nodes in green and then that was increased to 15 nodes and then to 30 nodes now this is a database that has of size now this is a database of size 700 gigabytes with 6 billion rows that's a pretty big table if you have 6 billion rows and in order to maintain that that's automatically done by Cloud Spanner using splits and the performance optimization across multiple zones TrueTime is a highly available distributed clock that is provided to applications on all Google servers TrueTime enables applications to generate monotonically increasing time stamps Cloud Spanner uses this to time stamp transactions allowing it to perform consistent reads across an entire database and across multiple cloud regions without blocking writes a traditional database that uses strict two phase locking provides external consistency in such a system every time your application wants to read the most current data which is called a strong read the system acquires a read lock on the data which blocks writes to the data being read with external consistency the system behaves as if all transactions were executed sequentially even though Cloud Spanner actually runs them across multiple servers the external consistency is a stronger property than strong consistency as it doesn't block data during the strong reads this is only available in the multi region configuration terabytes or even pedabytes of data you can use cloud big table to explore and query these types of data time series data such as CPU and memory usage over time for multiple servers marketing data such as purchase histories and customer preferences financial data such as transaction histories stock prices and currency exchange rates internet of things data such as usage reports from energy meters and home appliances or graph data such as information about how users are connected to one another cloud big table is good for low latency read write access it provides native time series support and a high throughput for analytics it's a fully managed no sequel solution that provides wide column database cloud big table is ideal for storing very large amounts of single key data with very low latency it supports high read and write throughput at low latency and is ideal data source for things like map reduce operation cloud big table compresses your data automatically using intelligent algorithm it provides incredible scalability for example it scales in direct proportion to the number of machines in your cluster it provides simple administration cloud big table handles upgrades and restarts transparently and it automatically maintains high data durability to replicate your data you simply have to add a second cluster to your instance and replication starts automatically we will look into this closer in a bit cluster resizing without downtime you can increase the size of a cloud big table cluster for a few hours to handle a large load then reduce the cluster size again all without any downtime after you change cluster size it typically takes just a few minutes under load for cloud big table to balance performance across all of the nodes in your cluster a cloud big table instance is mostly just a container for your clusters and nodes which do all of the real work tables belong to instances not to clusters or nodes the cluster represents the actual big table service each cluster belongs to a single cloud big table instance and an instance can have up to two clusters each cluster is located in a single zone in a production instance there are three or more nodes which are compute resources that big table uses to manage your data each cloud big table zone is managed by a master process which balances workload and data volume within clusters the master splits pizzeria larger tablets in half and merges less smaller tablets together redistributing them between nodes as needed the metrics you can monitor are typically the CPU average utilization utilization of the hardest node and then disk meaning storage utilization percentage per byte client requests go through a front end server before they're sent to a cloud big table node big table nodes are separate from the actual storage here we're not showing a load balancer that usually sits in between the client and the big table node multiple nodes map to data based on the metadata a cloud big table table is shorted into blocks of contiguous rows called tablets this is to help balance the workload of queries tablets are stored on colossus which is Google's file system in sst table format an sst table provides a persistent ordered immutable map from keys to values where both keys and values are arbitrary byte strings each tablet is associated with a specific cloud big table node in addition to the sst table files all writes are stored in colossus shared log as soon as they are acknowledged by cloud big table providing increased durability rebalancing these tablets from one node to another is very fast because the actual data is not being copied when a node fails no data is lost either throughput also scales linearly meaning that nodes add approximately 10,000 queries per second cloud big table stores data in massively scalable tables each of which is a sorted key value map the table is composed of rows each of which typically describes a single entity and columns which contain individual values for each row each row is indexed by a single row key columns that are related to one another are typically grouped together into a column family each column is identified by a combination of the column family and a column qualifier which is a unique name within the column family cloud big table tables are sparse if a cell does not contain any data it does not take up any space each table has only one index the row key there are no secondary indices rows are sorted lexiographically by row key meaning from the lowest to the highest byte string row keys are sorted in big indian or network byte order the binary equivalent of alphabetical order all operations are atomic row level meaning for example if you update two rows in a table it's possible that one row will be updated successfully and the other will fail ideally both reads and writes should be distributed evenly across the row space of the table so in general if you should keep all information for an entity in a single row entity that doesn't need atomic updates and reads can be split across multiple rows splitting across multiple rows is recommended if the entity data is large we will look at this closer in the performance section again cloud big table tables are sparse empty columns don't take up any space as a result it often makes sense to create a very large number of columns even if most columns are empty in most rows now let's take a look at cells each row column intersection can contain multiple cells at different time stamps providing a record of how the stored data has been altered over time so every cell is versioned with a time stamp by default cloud big table periodically writes your tables to remove deleted entries and to reorganize your data so that reads and writes are more efficient this process is known as compaction mutations or changes to a row take up extra storage space because cloud big table stores mutations sequentially so if you update the value in a cell both the original value will be stored on a disk for some amount of time until the data is compacted let's take a look at the schema design for time series data what is time series data whenever you measure something and you record the time together with the measurement you're building a time series examples include pressure over time plot of memory usage on your computer stock prices over time like day, month, year etc storing time series data in cloud big table is a good fit because big table stores data as unstructured columns in rows with the row keys sorted lexiographically retrieving data can be done by querying for a range of row keys so three design patterns for storing time series data so firstly choose meaningful names that are also as short as possible because the size of each name contributes to storage and RPC overhead for row key design in time series you should generally use tall and narrow tables storing one event per row makes it easier to run queries against your data also storing many events per row makes it more likely that the total row size will exceed the recommended maximum by default use new rows instead of column versions using multiple rows with a single version of an event in each row is the simplest way to represent understand and query your data then you have denormalization use two tables each with a row key appropriate to one of the queries this is a good solution because it results in a robust control system ensure that your row key avoids hard spotting when a row key for a time series includes a time stamp all of your writes will target a single note field promotion move fields from the column data into a row key to make writes non-contigious what's salting add an additional calculated element to a row key to artificially make writes non-contigious use reverse time stamps only when necessary prefer reverse time stamps only when your most common query is for the latest values this is because reversing time stamps makes every other query more complex indicates the overall schema so rows can be big but not infinite rows in cloud big table can contain about 100 column families and millions of columns with a 100 megabyte limit on each value stored in a column in general keep rows sizes below approximately 100 megabytes this is more of a guideline than a row rows can be larger than 100 megabytes however if you have many rows larger than this you're going to have performance issues when it comes to columns in general keep column values below approximately 10 megabytes again this is also more of a guideline keep related data in the same table and keep unrelated data in different tables cloud big table is a key value store it's not a relational store it does not support joints nor does it have support for transactions except within a single row so it's best to access data in individual rows or in a set of contiguous rows store data which you will access in a single query in a single column family all of the column qualifiers in a single column family are stored together access together and cache together as a result a query that accesses a single column family might execute more efficiently than a query spanning column families don't exploit a termicity of single rows only operations on a single row are transactional don't exploit a termicity of a single row only operations on on a single row are transactional as we talked about before transactions are also expensive meaning that a system that relies on transactions will not perform as well as it does not changes to data in an existing row should be stored as new separate row not changed in the existing row cloud big table stores data as unstructured columns in rows each row has a row key and row keys are sorted lexiographically there are two commonly used ways to retrieve data cloud big table you can get a single row by specifying the row key you can get multiple rows by specifying a range of row keys these methods are ideal for querying time series data since you often want data for a given time range for example all of the market data for the day or server CPU statistics for the last 15 minutes you can also increment values canonically append and do conditional updates now let's take a look at a demo on creating a big table instance so here we are back to our google cloud platform dashboard let's make sure that we select or create the correct project which is here down and I'm selecting my existing project here you can create a new project if you need to so now I'll go to the navigation menu and look for storage and then big table click on that so here we have no big table instances currently so we create instance click on that here we have some options the first thing we can see on the right here is the monthly cost estimate we can see that it's pretty expensive to create and run a big table instance it's almost $1600 per month for a basic one cluster one terabyte SSD big table instance so here first we can create the instance name let's call it BT instance and we can see the instance ID is also here which is permanent and then we have instance type we can choose either production or development production one will need at least three nodes and the development is a low cost option if you click on that we can see that the price here has changed to $644 from the previous $1600 and so if you choose to use a development version you can always upgrade it to a production version later on and the big difference is that with the development version you will only have one node and you can't increase the number of nodes and then you also can't have a replication cluster set up for a development instance so let's choose production and then you have a storage type we can choose either SSD or an HDD disk for your cluster in most cases the HDD is better because it provides a higher level of performance and this choice is permanent you can't change this later on so here we have the number of clusters and so this is the original cluster so we can select a region here choose Europe North 1 and we have three zones that we can select here from the nodes would be for replication purposes you choose a different zone in terms of nodes you start with three, three is the minimum number of nodes to create a production instance and here we have a choice to add a replication cluster if you need to so for now we just click done and create so now this creates our big table instance and it's created here with three nodes in the Europe North 1C zone when you create an instance with two clusters cloud big table immediately starts to synchronize your data between the zones where the clusters are located creating a separate independent copy of your data in each zone similarly when you add a new cluster to an existing instance cloud big table copies your existing data from the original cluster zone to the new cluster zone then synchronizes changes to your data between the two zones cloud big table replicates the changes automatically when you update data in existing tables new and deleted tables added and removed column families changes to column families garbage collection policy and also when you update the zones by default replication for cloud big table is eventually consistent what does that mean it means that when you write a change to one cluster you will eventually be able to read that change from the other cluster but only after the change is replicated between the clusters cloud big table can also provide read your writes consistency when replication is enabled which ensures that an application will never read data that is older than its most recent writes for read write for new consistencies the app profiles need to be configured for single cluster routing and for strong the second cluster cannot be used except for a failover if a cloud big table cluster becomes unresponsive the application makes it possible for incoming traffic to failover to the instances other cluster provides a detailed look at how you use data in your cloud big table tables it generates visual reports for your tables that break down your usage based on the row keys that you access key visualizer can help you complete the following tasks you can check whether your reads or writes are creating hot spots specific rows find rows that contain too much data look at whether your access patterns are balanced across all of the rows in a table key visualizer automatically generates hourly and daily scans for every table with a table size is over 30 gigabytes or the average reads or writes has at least 10,000 rows per second in the last 24 hours each scan includes a large heat map which you will see in a bit which shows access patterns for a group of row keys over time the core of a key visualizer scan is the heat map which is shown here this shows the value of a metric over time broken down into contiguous ranges of row keys the x-axis of the heat map represents time and the y-axis is the row keys if the metric had a low value for a group of row keys at a point in time the metric is cold and it appears in a dark color a high value which is hot appears into a bright color so white means hot and dark means cold a cloud big table table can have trillions of rows so it's not always practical to report metrics for each individual row instead key visualizer divides all of the row keys into 1,000 contiguous ranges with roughly the same number of row keys in each range these ranges are known as key buckets in general a cluster's performance increases linearly as you add nodes to the cluster for example if you create an SSD cluster with 10 nodes the cluster can support up to 100,000 rows per second for a typical read-only or write-only workload with a 6 millisecond latency for each read or write operation the two different strategies for optimizing data over time distributing the amount of data across nodes so if you have written only a small amount of data to a table cloud big table will store all of the tablets on a single node within your cluster as you can see in the first diagram as more tablets accumulate cloud big table will move some of them to other nodes in the cluster so that the amount of data is balanced more evenly across the cluster as you can see in the second diagram then you have distributing reads and writes evenly across nodes here you can see in the third diagram where there are some cases where you can't avoid accessing certain rows more frequently than others cloud big table helps you deal with these cases by taking reads and writes into account when it balances tablets across nodes for example in the third diagram we can see that 25% of the reads are going to a small number of tablets on the third node so here cloud big table will move those three tablets into a separate node so that they can get a higher amount of performance so when you are testing or troubleshooting performance some guidelines are make sure you are using a production instance with at least 300 gigabytes of data stay below the recommended storage utilization per node before you test run a heavy pre-test for several minutes this step gives cloud big table a chance to balance data across your nodes based on the access patterns it observes run your test for at least 10 minutes this step lets cloud big table further optimize your data it helps ensure that you will test reads from disk as well as cache reads from the memory make sure you are looking at the key visualizer scans for your table the key visualizer tool for big table provides daily scans that show the usage pattern for each table in a cluster key visualizer makes it easy to check whether your usage patterns are causing undesirable results such as hotspots on specific rows or excessive CPU utilization try commenting out the code that performs cloud big table reads and writes if the performance issue disappears then you are probably using cloud big table in a way that results in suboptimal performance ensure that you are creating as few clients as possible creating a client for cloud big table is a relatively expensive operation make sure you are reading and writing many different rows in your table cloud big table performs best when reads and writes are evenly distributed throughout your table which helps cloud big table distribute the workload across all the nodes in your cluster verify that you see approximately the same performance for reads and writes if you find that reads are much faster than writes you might be trying to read row keys that do not exist or a large number of row keys that contain only small number of rows let's take a look at another demo this time let's learn how we can manage tables table instance that we previously created in the previous demo we have seen how to create a big table instance now let's create actual tables and column families and so on we can see the big table instance that was created previously here if you click on this you can see the details of that instance and we can see that and if you click on tables we will see that there should be no tables found at this point so let's go ahead and activate the cloud shell so we can use the command line so we click on this icon activate cloud shell and this opens our gcloud command line tool whenever you are in the gcloud command line make sure that your project ID is correct and that you are currently in the project that you want to operate in so here we are in the correct one and so you can use the CBT command type CBT and hit enter and this will show you a list of commands that you can use for big table you can think of it as a command line for big table CBT and there is a list of different commands that you can use to manipulate your instance so for example you can type CBT list instances and hit enter and this will show you the list of big table instances that you have in this project so here we can see our BT instance that we just created previously so now to create a table we can use these commands here now these commands are also on the slide on the powerpoint previous to this demo so to create a table the command is CBT create table and then the table name let's type that in and I'll call it myTable and we hit enter and our table should be created so if we refresh the tab here we should be able to see our first table we can see here our table called myTable now in order to create a family it's CBT create family and then the table name and the family name so let's do that CBT create family table name and my family name we hit enter now let's create a second family family2 hit enter now to view the information about our table the command is CBT LS and then the table name so let's try that and hit enter and we can see that there are two families called family1 and family2 and we can also see that the GC is set to never this is the garbage collection policy which states how long before the data can be deleted meaning how many versions of the data should be stored so we can set that here as well with this command CBT set GC policy and in the table name and the family name and then max versions equals and then the number of versions we want to keep so let's try that now CBT set GC policy my table family1 max versions equals let's say 5 hit enter oh it should be max versions instead of version max versions equals 5 hit enter and that should do it now let's check the table information again with the previous command CBT LS my table and now we can see that the two families are shown here and the GC policy has been set for 5 only for family1 now in order to delete a family we can just use the CBT delete family table name and family name so let's delete family2 CBT delete delete family table name and family name now let's check the table information CBT LS my table now we can see that only one family is remaining here now when you delete a family it deletes all the data that's stored in that family as well now we can also check the monitoring to see the different statistics about our instance so if you click on monitoring as we have seen in the performance section we can see the CPU utilization CPU utilization of the hottest node and other metrics here that are constantly being checked including the cluster details and storage utilization and the error rates additionally you can check the metrics for instance tables and for the applications so if you click on tables you can specifically see the storage utilization the number of reads writes and the throughputs as well for the specific table and finally we can see the key visualizer by clicking on the key visualizer link here which opens an external link to stack driver so when stack driver loads you can see under resources here you can see the key visualizer for big table if you click on that then you get to the big table key visualizer tool however as it says here no tables in this project meet the minimum size activity requirement as we have seen in the slides that you need at least 30 gigabytes or 10,000 reads or writes for automatic scans and it takes up to 24 hours for new tables to be scanned automatically so you can't really see the heat map at this point but this is where you'd be able to see it now let's take a real world example I'm sure you all heard of Twitter Twitter has around 300 million monthly active users of those users they're generally generating around 100 million tweets per day which turns into something like 300 petabytes of data they use big table for their analytics product they can see things like spend breakdown impressions clicks and all these metrics in real time through UIs for the hundreds of thousands of active advertisers at any given time they also generally have around 5000 QPS queries per second which turns into around a gigabyte per second of data that they're processing at any given time just for query execution at a high SLO uptime of 9997 so things like give me all the impressions that my promoted tweets have generated for the last 7 days show me the spend that my campaigns have had over the last month some together or give me the impressions by day for only app installs by iOS users or how much do people in Japan spend on tweets as you can see big table can handle very demanding tasks when it comes to NoSQL unstructured data cloud big table when you create an instance with two clusters cloud big table immediately starts to synchronize your data between the zones where the clusters are located creating a separate independent copy of your data in each zone similarly when you add a new cluster to an existing instance cloud big table copies your existing data from the original clusters zone to the new clusters zone then synchronizes changes to your data between the two zones cloud big table replicates the changes automatically when you update data in existing tables new and deleted tables added and removed column families changes to column families garbage collection policy and also when you update the zones by default replication for cloud big table is eventually consistent what does that mean it means that when you write a change to one cluster you will eventually be able to read the change from the other cluster but only after the change is replicated between the clusters cloud big table can also provide read your writes consistency when replication is enabled which ensures that an application will never read data that is older than its most recent writes for read write and strong consistencies the app profiles need to be configured for single cluster routing and for strong the second cluster cannot be used except for a failover if a cloud big table cluster becomes unresponsive the application makes it possible for incoming traffic to failover to the instances other cluster key visualizer provides a detailed look at how you use data in your cloud big table tables it generates visual reports for your tables that break down your usage based on the row keys that you access key visualizer can help you complete the following tasks you can check whether your reads or writes are creating hotspots on specific rows find rows that contain too much data look at whether your access patterns are balanced across all of the rows in a table key visualizer automatically generates hourly and daily scans for every table where the table size is over 30 gigabytes or the average reads or writes has at least 10,000 rows per second in the last 24 hours each scan includes a large heat map which you'll see in a bit which shows access patterns for a group of row keys over time the core of a key visualizer scan is the heat map which is shown here this shows the value of a metric over time broken down into contiguous ranges of row keys the x-axis of the heat map represents time and the y-axis is the row keys if the metric had a low value for a group of row keys at a point in time the metric is cold and it appears in a dark color a high value which is hot appears into a bright color so white means hot and dark means cold a cloud big table table can have trillions of rows so it's not always practical to report metrics for each individual row instead key visualizer divides all of the row keys into 1,000 contiguous ranges with roughly the same number of row keys in each range these ranges are known as key buckets in general a cluster's performance increases linearly and adds nodes to the cluster for example if you create an SSD cluster with 10 nodes the cluster can support up to 100,000 rows per second for a typical read-only or write-only workload with a 6 millisecond latency for each read or write operation the two different strategies for optimizing data over time distributing the amount of data across nodes so if you have written only a small amount of data to a table cloud big table will store all of the tablets on a single node within your cluster as you can see in the first diagram as more tablets accumulate cloud big table will move some of them to other nodes in the cluster so that the amount of data is balanced more evenly across the cluster as you can see in the second diagram then you have distributing reads and writes evenly across nodes here you can see in the third diagram where there are some cases where you can't avoid accessing certain rows more frequently than others cloud big table helps you deal with these cases by taking reads and writes into account when it balances tablets across nodes for example in the third diagram we can see that 25% of the reads are going to a small number of tablets on the third node so here cloud big table will move those three tablets into a separate node so that they can get a higher amount of performance so when you're testing or troubleshooting performance some guidelines are make sure you're using a production instance with at least 300 gigabytes of data stay below the recommended storage utilization per node before you test run a heavy pre-test for several minutes this step gives cloud big table a chance to balance data across your nodes based on the access patterns it observes run your test for at least 10 minutes this step lets cloud big table further optimize your data and it helps ensure that you will test reads from disk as well as cached reads from the memory make sure you're looking at the key visualizer scans for your table the key visualizer tool for big table provides daily scans that show the usage pattern for each table in a cluster key visualizer makes it easy to check whether your usage patterns are causing undesirable results such as hot spots on specific rows or excessive CPU utilization try commenting out the code that performs cloud big table reads and writes if the performance issue disappears then you're probably using cloud big table in a way that results in sub-optimal performance ensure that you're creating as few clients as possible creating a client for cloud big table is a relatively expensive operation make sure you're reading and writing the rows in your table cloud big table performs best when reads and writes are evenly distributed throughout your table which helps cloud big table distribute the workload across all the nodes in your cluster verify that you see approximately the same performance for reads and writes if you find that reads are much faster than writes you might be trying to read row keys that do not exist or a large number of row keys that contain only small number of rows let's take a look at another demo this time let's learn how we can manage tables within the big table instance that we previously created in the previous demo we have seen how to create a big table instance now let's create actual tables and column families and so on we can see the big table instance that was created previously here if you click on this you can see the details of that instance and we can see that in if you click on tables we'll see that there should be no tables found at this point so let's go ahead and activate the cloud shell so we can use the command line so we click on this icon activate cloud shell and this opens our gcloud command line tool whenever you're in the gcloud command line make sure that your project ID is correct and that you're currently in the project that you want to operate in so here we're in the correct one and so you can use the cbt command type cbt and hit enter and this will show you a list of commands that you can use for big table you can think of it as command line for big table cbt and there's a list of different commands that you can use to manipulate your instance so for example you can type cbt list instances and hit enter and this will show you the list of big table instances that you have in this project so here we can see our bt instance that we just created previously so now to create a table we can use these commands here now these commands are also on the slide on the powerpoint previous to this demo so to create a table the command is cbt create table and then the table name stripe that in and I'll call it my table and we hit enter and our table should be created so if we refresh the tab here we should be able to see our first table we can see here our table called my table now in order to create a column family it's cbt create family and then the table name and the family name so let's do that cbt create family table name and my family name we hit enter now let's create a second family family 2 and we hit enter now to view the information about our table the command is cbt ls and then the table name so let's try that and hit enter and we can see that there are two families called family 1 and family 2 and we can also see that the GC policy is said to never this is the garbage collection policy which states how long before the data can be deleted and how many versions of the data should be stored so we can set that here as well with this command cbt set GC policy and in the table name and the family name and then max versions equals and then the number of versions we want to keep so let's try that now cbt set GC policy my table family 1 max versions equals let's say 5 hit enter it should be at max versions instead of version max versions equals 5 let's hit enter and that should do it now let's check the table information again with the previous command cbt ls my table and now we can see that the two families are shown here and the GC policy has been set for 5 only for family 1 now in order to delete a family we can just use the cbt delete family table name and family name so let's delete family 2 cbt delete delete family table name and family name now let's check the table information cbt ls my table now we can see that only one family is remaining here now if we delete a family it deletes all the data that's stored in that family as well now we can also check the monitoring to see the different statistics about our instance so if we click on monitoring as we have seen in the performance section we can see the cpu utilization cpu utilization of the hardest node and other metrics here that are constantly being checked including the cluster details and storage utilization and the error rates additionally you can check the metrics for instance tables and for the replications so if we click on tables you can specifically see the storage utilization the number of reads, writes and the throughputs as well for the specific table and finally we can see the key visualizer by clicking on the key visualizer link here which opens an external link to stack driver so when stack driver loads you can see under resources here you can see the key visualizer for big table if you click on that then you get to the big table key visualizer tool however as it says here no tables in this project meet the minimum size or activity requirement as we have seen in the slides that you need at least 30 gigabytes or 10,000 reads or writes for automatic scans and it takes up to 24 hours for new tables to be scanned automatically so you can't really see the heat map at this point but this is where you'd be able to see it now let's take a real world example I'm sure you all heard of Twitter Twitter has around 300 million monthly active users of those users they're generally generating around 100 million tweets per day which turns into something like 300 petabytes of data they use big table for their advertiser analytics product they can see things like spend breakdown, impressions clicks and all these metrics in real time through UIs for the hundreds of thousands of active advertisers at any given time they also generally have around 5000 QPS queries per second which turns into around a gigabyte a second of data that they're processing at any given time just for query execution at a high SLO uptime of 9997 so things like give me all the impressions that my promoted tweets have generated for the last 7 days show me the spend that my campaigns have had over the last month totaled up in sum together or give me the impressions by day for only app installs by iOS users or how much do people in Japan spend on tweets as you can see big table can handle very demanding tasks when it comes to NoSQL unstructured data on highly available structured data at scale it's a scalable fully managed NoSQL document database for your web and mobile applications it's good for semi structured application data hierarchical data and durable key value data you can use cloud data store to store and query these kinds of data product catalogs that provide real time inventory and product details for a retailer user profiles for a customized app experience based on the users past activities and preferences and transactions based on asset properties for example transferring funds from one bank account to another so what are some of the features of data store cloud data store can execute a set of operations where either all succeed or none occur high availability of reads and writes cloud data store runs in Google data centers which uses redundancy to minimize impact from points of failure massive scalability and high performance data store uses a distributed architecture to automatically manage scaling flexible storage and querying data store maps naturally to object oriented and scripting languages and it exposes to applications through multiple clients balance of strong and eventual consistency cloud data store ensures that entity lookups by key and ancestor queries always receives strongly consistent data all other queries are eventually consistent encryption at rest cloud data store automatically encrypts all data before it's written to disk and automatically decrypts the data when read by an authorized user while the cloud data store interface has many of the same features as traditional databases as a no-SQL database it differs from them in a way it describes here data store stores data in entities properties and kinds entities of the same kind can have different properties and different entities can have properties with the same name but different value types if you compare that to a traditional relational database kinds are tables entities are rows properties are columns and keys are the primary key equivalents data store entities of the same kind can have different properties and different entities can have properties with the same name but different value types cloud data store writes scale by automatically distributing data as necessary it reads scale because the only query supported are those performance scales with the size of the result set as opposed to the data set which traditional databases do cloud data store does not support for join operations inequality filtering on multiple properties or filtering on data based on results of a sub query cloud data store is schema less it doesn't require entities of the same kind to have a consistent set of properties when you create a new cloud data store instance you will have the option to create a cloud data store database or a cloud fire store database additionally a cloud fire store database can run in one mode native mode or data store mode cloud data store creates a cloud data store database with existing features and limitations whereas cloud fire store in data store mode uses cloud data store system behavior but accesses cloud fire store storage layer does removing some of the limitations like eventual consistency all cloud data store queries become strongly consistent now transactions are no longer limited to 25 entity groups and writes to an entity group are no longer limited to one per second we should note that these products are still in better so if you need an SLA then you should still opt for the original cloud data store until the cloud fire store in data store mode and the native modes become GA and remove the beta status now let's take a look at a demo where we can learn how to create a data store in this demo we will learn how to create a data store so now we are back to our Google cloud platform UI let's make sure we have selected the correct project or create one if necessary so in this case my project name is cloud data store demo click on that and I have my project name and project ID here which is the same across all Google products and services so now we go to the navigation menu and click on that and browse to storage and we see data store click on that and this takes us to the main cloud data store page so here we can see there are entities, dashboard indexes and admin so let's go ahead and create an entity by clicking on create entity we can leave that name space as default and let's create a kind called pens let's add a property called blue pen of type string and with the value big blue pen let's add a second property called on sale of type boolean and have the value as false so now let's go ahead and create our entity by pressing on the create button and now on the main page we can see that our entity has been created now let's go to the query query by gql click on this here we can query our data stores we can do this by using a sequel like language so we can do something like select star from pens now this has to be case sensitive here so you have to use a capital letter so and you click on run query and we can see one result here with the data that we specified we can also use a where clause to filter our results like for example say where on sale equals true now when you run the query we should get no results let's change the true to false and run the query again and we can see our result here so like we discussed in the lecture the kind which is pens here acts as the table in a real database the blue pen and on sale act as the column names and these are property names and you can see that the ID is the the primary key in this case now let's take a look at a real world example Pokemon Go by Niantic is a popular phone game it's a very demanding augmented reality game that requires GPS location and player data for every user at its peak the game had over 20 million daily active users these users were all demanding the servers for getting player data as well as their GPS location and the opponent's GPS location in order to do battles for this on the server side Niantic was using over 10,000 CPU cores in a Kubernetes solution in order for the servers to handle this traffic in real time and serve the GPS locations as well as keep track of player data a traditional relational database would not be sufficient the latency would be too high so in order for that to work you would need not just a no-SQL database but also a solution that would be able to distribute the load in real time and scale efficiently across multiple geographies so this is where Datastore came in and was the perfect fit Compute Engine, Virtual Machine Instances GKE Clusters and App Engine Flex Instances and other cloud based resources and services a VPC stands for a Virtual Private Cloud a Virtual Private Cloud is a logical isolation of a section of your cloud it's a cloud equivalent of your traditional network that you would run in and your data center but with the extremely important added benefits of running on infrastructure that is highly optimized for scale, performance and reduced network costs using Google Cloud means you're using the same online network infrastructure that powers all of Google services like Search, Maps, YouTube and Gmail VPC is a programmable interface for distributed networking functions on this interface it allows you the capability of creating virtual machines anywhere in the world in many regions and having private connectivity between all of these virtual machines by default some of the features include a single VPC across all regions no peering of VPCs is required and no cross region VPNs are required so you don't need to give your Virtual Private Machines public IPs in order for these to communicate with each other even though they are across different regions a single global VPC that is shareable in nature allows you to administer your network centrally as well while allowing thousands of developers and development teams in the organization to create separate projects and to create and manage compute resources the traffic that goes from the virtual machines to the storage for example would use the network backbone and would not use the internet so the data is encrypted while in transit there is a product called cloud armor that Google also provides which secures the VPC perimeter distributed firewalls and distributed networks with no choke points using cloud router high bandwidth and availability using the Andromeda control plane which supports Kubernetes via GKE here are some of the VPC concepts VPC networks and their associated routes firewalls, rules are global resources they are not associated with any particular region or zone a network must have at least one subnet before you can use it subnets are regional resources though each subnet defines a range of IP addresses when you create an instance or resource you select a zone a network and a subnet GCP assigns the instance an IP address from the range of available addresses in the subnet the traffic to and from instances can be controlled with the network firewall rules so here we can see on the left a traditional VPC and on the right what a Google VPC looks like so in the traditional version you can see two separate regions US west and US east each of these regions have a separate network and an application server on each of their networks you can see that in order for those two application servers to communicate there needs to be a VPN gateway that provides communication across the internet that's where the public transport happens and that's how traffic is routed between these two completely separate VPCs on the right you can see the Google VPC where you have two separate subnets and two separate VPCs with two separate application servers but they're both connected with the Google cloud platform a single Google cloud VPC can span multiple regions without communicating across the public internet so the containers where the VPC lives is a global construct for on-premises scenarios you can share a connection between VPC and on-premises resources with all regions in a single VPC so you don't need a connection in every region the Google cloud platform offers two types of VPC networks the auto mode and the custom mode when an auto mode network is created one subnet from each region is automatically created within its predefined IP ranges and new subnets are added when new regions become available each project starts with a default auto mode network the predefined IP ranges of the subnets do not overlap with IP ranges you would use for different purposes except for manually added ones when a custom mode network is created no subnets are automatically created this type of network provides you with a complete control over its subnets and IP ranges you get to decide which subnets to create in which regions to choose and using IP ranges you specify for example the price of creating a Kubernetes container cluster involves selecting a zone or region depending on the cluster type a network and a subnet the subnets available for selection are restricted to those in that selected region now let's do a demo where we create an auto mode network in this demo we'll look at how to create a VPC now we're back here at the Google Cloud Platform dashboard that should be familiar to all of us by now so let's select or create our project here by clicking down and in this case selecting my project here VPC and subnets demo or you can also create a new project if you need to so here I can go to the navigation menu and select VPC under the networking options which is here VPC network and VPC networks click on this so this brings us to the dashboard for VPC networks so on the left here we can see different tabs for external IP addresses firewall rules routes VPC network peering and shared VPC for now we'll create a VPC network by pressing this button here so to create a VPC network first we can choose a name we'll call it demo VPC description and for the subnets we can either choose custom or automatic so if we choose automatic then we get a message that says that the ranges are going to be assigned automatically based on the region here and if you do choose custom then we can actually create multiple subnets where we can choose a name for a subnet and then the region and then the IP address range we can also create secondary IP ranges here by clicking on this we can choose if we want to allow private google access if this is on that means all the VMs in this network will be able to communicate with each other without requiring external IP addresses for now we'll just choose automatic also when you choose custom you get to choose whether you want dynamic routing to be regional or global so for now let's just choose automatic and if you scroll down we get to firewall rules here some of these are already selected for us these can be either enabled or disabled and these two cannot be disabled these are the deny all incoming and allow all outgoing the ingress and egress rules that come with by default so here allowICMP is just for error reporting it's a network layer protocol this one is for internal communications and RDP is for remote protocol SSH is for secure shell and so we can choose to allow these and we don't need to provide a DNS server policy at this time so we can just press create to create our VPC so here you can see that our VPC is being created ok so now it's created and we can see that the different ranges are also created for the different regions if we click on the VPC you can get the details about our VPC network so here we can see there are multiple tabs here we can see the static internal IP addresses if there are any our firewall rules routes VPC network peering and if there are any private service connections subnets and IP ranges when you create a subnet you must define a primary IP address range and optionally up to 5 secondary IP ranges the primary address range can be used for virtual machine primary internal IP addresses VM alias IP addresses and IP addresses of internal load balancer the secondary IP ranges are used only for alias IP addresses each subnet has 4 reserved IP addresses in its primary IP range there is a network default gateway second to last reservation and a broadcast address let's take a look at a demo on subnets where we can learn how to add subnets and list and describe subnets GCP resources such as computer engine, VM instances forwarding rules GKE containers and app engine rely on IP addresses to communicate if you have multiple services running on a single VM instance you can give each service a different internal IP address using the alias IP ranges the VPC network forwards packets designed for each alias IP to the corresponding virtual machine we learn more in depth about IP addresses in a future lecture you can add multiple network interfaces to a VM instance each interface resides in a unique VPC network multiple network interfaces enables a network appliance VM to act as a gateway for securing traffic among different VPC networks or to and from the internet this diagram shows a situation where we can see the interplay between the two features in this case VM1 has a network interface in both network A and network B network B is paired with another network C IP1 is where the network interface with IP1 address from network A which is the primary interface IP2 is a network interface with IP2 address from network B which is a secondary interface so here IP3 can send traffic to IP2 because IP2 is a network B and network B routes are automatically propagated to network C when the two networks appeared we will look at peering in a bit but for IP2 to send network traffic to IP3 you will have to configure a policy routing on the IP2 interface IP1 flows are not installed in network C so therefore network C cannot access IP1 routes tell VM instances and VPC network how to send traffic from an instance to a destination either inside the network or outside of GCP each VPC network comes with some system generated routes to route traffic among its subnet and send traffic from eligible instances to the internet there are two types of system generated routes the default route which defines a path for traffic to leave the VPC network and provides general internet access to VMs and provides the typical path for the private Google access a subnet route is created for each of the IP ranges associated with a subnet every subnet has at least one subnet route for its primary IP range and an additional subnet routes created for a subnet if you add secondary IP ranges the subnet routes define paths for traffic for each VMs that use these subnets each VPC network also has an associated dynamic routing mode that controls the behavior of all its cloud routers each VPC network implements a distributed virtual firewall that you can configure firewall rules allow you to control which packets are allowed to travel to which destinations every VPC network has two implied firewall rules that block all incoming connections and allow all outgoing connections the default network includes a number of firewall rules in addition to the implied ones including the default allow internal rule which permits instance to instance communication within the network rules that come with the default network are also presented as options for you to apply to new automo networks that you create using the GCP console for instance to have outgoing internet access firewall rules must allow egress traffic from the instance and it must have an external IP address once the VPC is set up and secured the next thing to do is to connect your VPC running in the cloud with your on premises data center so there are multiple ways of doing this if bandwidth and the overhead of IPsec encryption is not a problem then you can just use a VPN you can either create static routes to your VPN tunnel to basically allow you to statically specify the prefixes that should be routed to your on-prem or you can use the cloud router which runs as a BGP daemon to exchange routes with your on-prem in a dynamic way so if bandwidth is a concern though you can use interconnects if you would like to directly peer with Google if you have a public ESN and you want to basically connect with them at one of the POPs or points of presence then you can use a dedicated interconnect if you're however doing the same thing using a carrier's infrastructure and you're paying the carrier for the service then you can use something called a partner interconnect Google Cloud Platform load balancing gives you the ability to distribute load balanced compute resources in single or multiple regions to meet your high availability requirements to put your resources behind a single anycast IP and to scale the resources up and down with order scaling there are three types of load balancers the global external load balancer for mostly HTTPS SSL proxy and TCP proxy and then two regional ones one for external network load balancing and one for internal load balancing shared VPC so you can share a VPC network from one project called a host project to another project in your GCP organization you can grant access to entire shared VPC networks or select subnets this allows you to provide centralized control over a common network while maintaining organizational flexibility shared VPC is especially useful in large organizations now a shared VPC was designed with the intention of having private connectivity between virtual machines in one organization but if you want to connect two VPCs in two different organizations then you can use VPC peering with VPC network peering all communication happens using private IP addresses these are subject to firewall rules and VM instances and each peer network can communicate with one another without using external IP addresses peer networks only share their subnet routes the network administration for each peer network is unchanged the network and security admins for one network do not automatically get those roles for the other network in the peering relationship if two networks from different projects appeared project owners editors and compute instance admins in one project do not automatically receive those roles in the project that contains the other network so let's take a look at an example here we have a shared VPC and we have four projects project one is the host project the VPC network peering allows peering with a shared VPC a shared VPC host project is a project that allows other projects to use one of its networks so the network shared VPC is in a shared VPC network in host project P1 service projects P3 and P4 are able to attach VM instances to network shared VPC the peers with network A as a result VM instances in shared VPC service projects that are using the network which is the VM3 and VM4 have private internal IP connectivity with all endpoints associated to network A VM instances associated to network A will have private internal IP connectivity with any endpoints associated to network SVPC regardless of whether those endpoints live in the host project or in a service project it's also possible to set up VPC network peering between two shared VPC networks and how they communicate with each other in GCP you can assign an IP address to certain resources for example you can assign an internal and external IP address to compute engine virtual machine instances similarly you can assign an internal or external IP address to a forwarding rule or external load balancing respectively to communicate between instances on the same virtual private cloud network you can use an instances internal IP address however to communicate with the internet you must always have the instances external IP address unless you have some kind of a proxy configured use the external IP address to connect to instances outside the same VPC network unless those networks are connected by a VPN both external and internal IP addresses can be either ephemeral or static a forwarding rule is required for network, global and internal load balancing the forwarding rule must have an external or internal IP address depending on the load balancer that you are using you can assign an external IP address to an instance or a forwarding rule if you need to communicate with the internet sources from outside the GCP VPC network can address a specific resource by the external IP address as long as firewall rules allow the connection only resources with an external IP address can send and receive traffic directly to and from outside the network Compute Engine supports two types of external IP addresses static external IP addresses and ephemeral external IP addresses you can reserve a static external IP address which assigns the address to your project indefinitely until you explicitly release it this is useful if you are dependent on a specific IP address for your service and need to prevent others from being able to use the address you can reserve a new static external IP address or promote an existing ephemeral external IP address to a static external IP address now within static external IP addresses they can either be regional or global a regional one allows resources of that region or resources of zones within that region to use the IP address in this case, VM instances and regional forwarding rules can use a regional static IP address now a global static external IP address these are available only to global forwarding rules these are used for global load balancing you can't assign these to regional or zonal resources now an ephemeral external IP address is one that does not persist beyond the life of the resource so when you create an instance or a forwarding rule without specifying an IP address the resource is automatically assigned an ephemeral external IP address so for VM instances if you stop the instance the IP address is also released so whether you restart it or stop it it's the same thing now every VM instance has one primary internal IP address that is unique to that VPC network you can assign a specific internal IP address when you create a VM instance or you can reserve a static internal IP address for your project and assign that address to your resources so if you do not specify an address compute engine assigns one automatically in either case the address must belong to the IP range of that subnet so if it's an auto mode VPC network the address comes from the region's subnet if it's custom you must specify which subnet the IP address will come from if your network is a legacy network then the IP address is assigned from the network's global internal IP range the engine supports two types of internal IP addresses static internal IP addresses and fmrl internal IP addresses static ones are assigned to a project long term until they are explicitly released and they remain attached to the resource until they are detached for VM instances static internal IP addresses remain attached to stopped instances until they are removed fmrl IP addresses remain attached to a VM instance only until the VM is stopped and restarted or the instance is terminated for internal load balancers you can assign a static internal IP address specify an explicit fmrl internal IP address or let GCP assign an internal IP address randomly now let's take a look at a demo in assigning a static external IP address to a VM instance in this demo we'll learn how to assign a static external IP address while creating a VM instance so we're back here in our google cloud platform dashboard so here first we select or create a new project by clicking down here in this case I'm choosing the project IP addresses demo click on that make sure we're in the right project so now we go to the navigation menu and select compute compute engine and VM instances click on that this takes us to the VM instances page so for the first time you would create a new VM instance you press create this takes us to our VM instance creation page so here we have we have several options so first we just have the typical VM name and the region machine type container access and so on down here we have a section called management security disks networking and sole tenancy so if you click on this we can see more options here and here we can see the networking tab so we click on that and we can see the networking details here so here we can see that the default network interface has this IP address and if you click on this we can make changes to the network interface that's being assigned to our VM so here we can see that the default subnet is slash twenty and the primary internal IP address is ephemeral that's automatically being assigned and the external IP address is also ephemeral because we haven't specified anything otherwise so in order to change this we can click on the down arrow here and press create new IP address now if you already have reserved IP addresses they will show up in this list so here I'm creating a new one so this opens a new window where I can assign or I can reserve a new static IP address so I'll call this VM IP1 and reserve so now we can see that an external IP address 34 73.80.7 has been assigned here so it has been reserved for us to be able to select this here and when we press create this creates our VM so here now our VM has been created with an external IP address of 34.73 80.7 and this will allow us to connect to the outside world now let's take a look at a demo on changing the external IP address in this demo we'll learn how to change the external IP address for a VM instance in the previous demo we have seen how we can assign an external static IP address to a VM instance while creating it now we have already created the VM instance now if we need to change this we can click on the VM instance so we can get the details of that instance and then press the edit button here now we can make changes here under network interfaces we can see the default network interface card Nick01 and we click on this to make changes here and here we can see the external IP address press the down key and we can set this to ephemeral if we need to so here we get a warning saying that the static IP address will be detached from the VM so now we scroll down and press save now if we go back to the VM details we can see that the IP address has changed here to 473.16.125 now we can see that specifically when you click on the details here that this is an ephemeral IP address meaning that it's a default that has been allocated by Google Cloud Platform and if you stop or restart this VM instance this IP address will be dropped and a new one will be assigned so it's not static anymore now let's take a look at a demo on new static internal IP addresses in this demo we'll learn how to create a static internal IP address and assign it to a new VM instance so we're back here in our Google Cloud Platform dashboard make sure you select your IP project or create a new one go to the navigation menu and we go to our compute engine VM instances so here we can see a list of VM instances in this project so let's create a new instance pressing create instance and here let's choose a region let's say Europe North One and the zone North One A scroll down and here under management security, disks, networking and soul tendency click on this now here we can see the networking details when we click on the networking tab and click on edit network interfaces back here from as we have seen in the previous demo and in this case we're going to choose the network here the demo VPC the demo VPC is a virtual private cloud network that was created in a previous lecture so this has an allocated IP addresses and a subnet and in this case we can choose a primary internal IP address here so here an affirmable one is automatically assigned when nothing is specified so you press down here and then reserve static internal IP address click on this so let's give this a name int VM and we'll allow this to be assigned automatically click on reserve and now the address is being reserved for us so now we have this IP address 10.166.0.2 now this is an internal IP address which means that only resources that are within the network can see and communicate with this VM with this IP address however it's static meaning that it won't be dropped or changed unless we specifically detach it from the resource so we can press done and create so now we have created a VM instance with a static internal IP address alias IP ranges Google cloud platform alias IP ranges let you assign ranges of internal IP addresses as aliases to a virtual machines network interfaces this is useful if you have multiple services running on a VM and you want to assign each service a different IP address for a VM network interface the alias IP must be from the same subnet resource that provides the IP address for the primary network interface you can't select a primary or secondary CIDR range from another subnet resource CIDR stands for classless inter domain routing and it is used for IP addresses and routing alias IP ranges also work with Kubernetes parts when an alias IP ranges are configured GCP automatically installs VPC network routes for primary and alias IP ranges for the subnet of the primary network interface card your container orchestrator does not need to specify VPC network connectivity for these routes so this simplifies the routing traffic managing your containers you don't have to add a route for every IP alias and you don't have to take route quotas into account either so using alias IP ranges container IP addresses can be allocated from a second CIDR range you can use this to separate the infrastructure from the services and you can use separate firewall controls for each of these ranges GCP automatically configures internal DNS for the primary IP of the primary interface of every VM instance GCP however does not associate alias IP addresses on the primary interface with the host name and it does not associate any IP addresses of secondary interfaces with the host name firewall source tags are not supported for alias IP addresses so when you configure source tags in firewall rules the source tag matches the VM primary IP address but not the alias IP addresses in a static route the next hop IP address must be the primary IP address of the virtual machine instance an alias IP address is not supported as the next hop IP address in VPC peering you're allowed to peer to VPC networks so that the VMs in the two networks can communicate via internal private IP addresses we can see in the diagram here of network peering both primary and secondary IP ranges of a subnet are reachable by VM instances in a peer network subnet overlap checks across peer networks ensure that primary and secondary ranges do not overlap with any peer ranges now when it comes to alias IP addresses within auto mode VPC networks you can allocate alias IP ranges from the automatically created primary CIDR range or you can add a secondary range to allocate the IP addresses from the alias IP addresses you can also create a new subnet with the secondary ranges in the auto mode VPC and then create VM instances in the new subnet and allocate those alias IP address ranges you can have up to 7000 alias IP ranges across all VMs in GCP now if you're using a custom mode network all of the subnets are created manually so one primary CIDR range is mandatory and the other secondary and settings and subnets are up to you to how you want to configure it now let's take a look at the demo on creating a VM with alias IP in this demo we'll learn how to create a VM instance with an alias IP range so we're back here in our Google Cloud Platform dashboard let's make sure we're on the right project by clicking down here or creating a new project if we need to I'm selecting IP addresses demo as my project now we go to the navigation menu and select compute engine VM instances so here we can see a list of our VM instances currently in this project now let's create a new instance by pressing create and down here and let's select a region let's say Europe West 2 and when you scroll down here we can see management, security disks, networking and soul tendency, click on this and then click on the networking tab and the default network interface so here we can see the network let's select the demo VPC, demo VPC is the virtual private cloud that we created in a previous lecture with this IP range and a subnet so here we can see an option for show alias IP ranges so if we click on that that expands to the further options so here we can see the subnet range and the alias range so let's choose 10.154.1.0 slash 24 as the alias IP range and this has to be an unused range from the primary so we go ahead and create so now we can see that the internal IP address here is 10.154.0.2 so let's take a look at this detail and we can see that we have an alias IP range defined here now let's take a look at a demo on adding an alias IP existing virtual machine now if we want to create an alias IP range for an existing VM instance it's similar to what we have done previously now I'm here back at the VM instances dashboard of google cloud platform and we can see all the VM instances that we have previously created so let's click on the first VM instance and see the details and here under network interfaces we can see that there is no alias IP range defined so we can click on edit up here to edit this VM instance and click on the network interface show alias IP ranges and here we can define the alias IP range that we have done previously while creating a VM instance in instances based on a configuration you specify when you create a gcp firewall rule you specify a vpc network and a set of components that define what the rule will do the components enable you to target certain types of traffic based on the traffic's protocol ports sources and destinations while firewall rules are defined at the network level connections are allowed or denied on a per instance basis gcp firewall rules exist not only between your instances and other networks but between individual instances within the same network has two implied firewall rules which permit outgoing connections and block incoming connections each firewall rules action is either allow or deny the rule applies to traffic as long as it's enforced which means it can be disabled for troubleshooting purposes each firewall rule applies to incoming which is ingress or outgoing ingress traffic but not both gcp firewall rules are stateful so if a connection is allowed between a source and a target or a target and its destination all subsequent traffic in either direction will be allowed as long as the connection is active in other words firewall rules allow bi-directional communication once a session is established so what are always block traffic so you have GRE traffic like all sources all destinations including some instances using internal IP addresses GRE stands for generic routing encapsulation which is a tunneling protocol that works in the network layer also blocked are protocols other than TCP, UDP ICMP and IPIP always allow traffic include DHCP DNS name resolution and instance metadata now what are the components of a firewall rule so you have a priority which is determined if the rule will be applied only the highest priority rule whose other components match traffic is applied if there are conflicting rules the lower priorities will be ignored then you have action the action on the match which is either allow or deny this determines if the rule permits or blocks traffic the enforcement status of the firewall is you can enable or disable the firewall rules without deleting them the targets and egress rules apply to traffic going to specified destinations from targets the source for egress rules or destination for egress rules and then you have protocols and ports such as TCP, UDP ICMP and then the port now let's take a look at a demo in listing the firewall rules of our VPC as well as our VM instances in this demo we will see how we can take a look at the firewall rules and routes so we are back here at our Google Cloud Platform dashboard from here make sure you select the correct project or create a new project if you have to I am clicking on firewall rules and routes demo so from here let's go to the navigation menu and go to networks VPC network and then firewall rules so here we can see the list of firewall rules that are currently on our VPC network and here we can see specifically which network these are all so these ones are on the demo VPC network created previously and these are the default network firewall rules now if you want to check the firewall rules of a specific VM instance then let's go to navigation menu and computer engine VM instances now let's click on our instance 1 here we can see the details of our instance and under network interface we can see the network details so we press on view details and here we can see the firewall rules of this network interface that has been attached to our VM instance and as we have discussed in the lecture we have the type the description, the targets and the filters protocols, ports action and priority and let's take another demo where we can create firewall rules in this demo we learned how to create a new firewall rule so we here back in our google cloud platform dashboard and from here we will go to our navigation menu and go to network VPC network and firewall rules as we have seen previously so here we have a list of all the firewall rules now we can create a new firewall rule by pressing on the create button this brings us to a new form so here we can specify a name for our firewall rule I'll call it f1 and then you can choose to either generate logs or not I'll choose yes and then we can select the network that we want to apply this to so either we can choose default or make it specific to our VPC that we created so I'm going to select it to our VPC and then we can select a priority so the priority is the lower the number the higher the priority would be so in this case I'm leaving it as 1000 then you have the direction of traffic either ingress or egress and then the action to match allow or deny and then the targets so here we can either select and say all instances in the network or apply to only specific target tags so I'm going to choose all instances now the source filter IP so we can choose which ranges to filter the source from and then I'm going to choose subnets and select our one of our subnets from our VPC here so I'm going to select this one and finally we have the protocols and ports and then we can specify which ports this firewall rule should apply to so we can select TCP and then we can either choose specify port numbers with a comma separation and we can also choose we can also type a range of ports like 30 to 80 and then we can press create and now we can see that the firewall rule has been created and it specific to this subnets and to these ports and with the priority of 1000 and applies to the demo vpc network firewall rules logging allows you to audit, verify and analyze the effects of your firewall rules you can determine if a firewall rule designed to deny traffic is functioning as intended you can only enable firewall rule logging for rules in a vpc network and for TCP and UDP connections so you cannot enable firewall rule logging for the implied deny ingress and implied allow egress rules or for automatically created default rules in the default network connection logging limits are expressed as a maximum number of connections that can be logged in a 5 second interval that's usually 500 per virtual CPU now let's take a look at a logging example and see what the log looks like so here we have traffic between VM instances in the example vpc network in the example project the two VM instances here are VM1 which is in the west one zone with the IP address 10.10. 0.99 and that's in the west subnet then you have VM2 in the US east one zone with the IP address 10.20.0.99 in the east subnet so here we create two firewall rules the first rule is an egress deny firewall rule has a target of all instances in the network a destination of 10.20.0.99 which is the VM2 and applies to tcp port 80 then you have rule B which is an egress allow firewall rule has a target of all instances in the network and a source of 10.10.0.99 which is our VM1 and this applies to tcp port 80 as well so if VM1 attempts to connect to VM2 on port 80 you get this firewall rules log so the log entry for rule A from the perspective of VM1 is generated because VM1 attempts to connect to VM2 so because rule A actually blocks the traffic even considered so there is no log entry for rule B so this one is specifically for rule A and you can see that in the field disposition it's denied now let's take a look at routing so there are four main route types the default route the subnet route static route the default route and subnet routes are system generated when you create a vpc network google cloud platform creates a system generated default route this route serves two purposes it defines the path out of the vpc network including the path to the internet in addition to having the route instances must also meet other requirements if they need internet access and then it provides a standard path for private google access the system generated default route has a priority of 1000 this is because the destination is a broadcast possible which is 0.0.0.0 and only this will be used if a route with a more specific destination does not apply to a packet we will look at the routing order in a bit where this will be a little more clear you can however delete the default route if you want to isolate your network from the internet now subnet routes are system generated routes that define paths to each subnet in the vpc network each subnet has at least one subnet route whose destination matches the primary IP range of the subnet if the subnet however has secondary IP ranges gcp creates a subnet route with a corresponding destination for each of those secondary ranges now because the subnet route must always have the most specific destination in cases where destination ranges would overlap its priority does not actually matter the priority of all subnet routes is fixed at 1000 now the subnet routes for subnets in a peer network cannot be removed unless you break the peering relationship when you break the peering relationship all imported subnet routes from the other network are also automatically deleted then you have custom routes these are either static routes you create manually or dynamic routes that are maintained automatically by one or more of your cloud routers the destinations for custom routes cannot match or be more specific than any of the subnet routes in the network static routes can use any static route next hops you can create them manually with the gcp console when you create a cloud vpn tunnel that uses policy based routing then you have dynamic routes these are managed by one or more of the cloud routers their destinations always represent IP ranges outside of your vpc network and their next hops are always bgp peer addresses a cloud router can manage these dynamic routes for cloud vpn tunnels that use dynamic routing system generated routes apply to all instances in the vpc network the scope of instances to which subnet routes apply cannot be altered however you can replace the default route custom static routes apply to all instances or specific instances depending on the tag attribute of the route the static routes with the tag attribute are applicable to instances that have a corresponding network tag so if no tag attribute is specified the static route applies to all instances in the network dynamic routes apply to instances based on the dynamic routing mode of the vpc network if it's in regional dynamic routing mode all cloud routers apply routes they learn within their respective region if it's global then all the cloud routers apply routes they learn within the whole network in terms of routing order the procedure used to find and select the next hop the packet is as follows subnet routes are considered first this is because gcp requires that subnet routes have the most specific destinations matching the ip address ranges of their respective subnets if the packet does not fit in the destination for the subnet route gcp looks for another route with the most specific destination if more than one route has the same most specific destination gcp considered the priority of the route so in this case if a single route with the highest priority is available the packet is sent to its next hop if more than one route has the same priority and is available traffic is distributed among multiple next hops using a 5 tuple hash for affinity the hash is calculated from the protocol the source and the destination ip address and if no applicable destination is found gcp drops the packet replying with an icmp destination or network unreachable error now let's take a look at a demo in inspecting routes in our vpc network as well as the system generated and the custom static routes and then we'll take a look at the route analysis for our vpc in this demo we'll take a look at routes and route analysis so we're back here in our google cloud platform dashboard with the correct project selected now we go to our navigation menu and go to network vpc networks and routes so here we can see a list of all the routes that are in our project and we can see the specific routes that are created for a specific network here this is the demo vpc and these are for the default routes and we can see that these are default routes that have been created by gcp let's take a look at this one so here we can see that these are created because of the virtual machine instance that were created instance 2 and instance 3 that we've created previously now if we want to take a look at the routes for a specific virtual machine we can do that by going back to our navigation menu go to compute engine vm instances so from here let's select our instance 2 click on it to get more details and here under network interface we have the details of the network interface that are attached to this vm click on view details of the network and here we can see all the details of this network interface it's all the way down here at the network analysis you have 3 tabs ingress analysis egress analysis and route analysis so if you click on route analysis you can see all the active routes that are created for this specific network interface and the order of these that are listed here is the order of priority for the routes now let's take a look at a demo on adding a static route in this demo we'll see how we can create a static route so we're back here in our google cloud platform dashboard with the correct project selected now we go to the navigation menu networking vpc network and routes so here we can see a list of all the routes that are in our project to create a route we click on the create route button now here we can specify a name for a route call it we click on the network where the route should be applied to so either our vpc network or default the destination ip range to define the destination of the route and then the priority which is only applicable when determining routing if the routes have equivalent destination then we click on the tags that's optional and then we can specify the next hop for the route so here we can see different options so the default internet gateway creates a route to the internet specify an instance allows you to select an instance by name so traffic will be routed to that even if its IP address changes specify IP address allows you to enter an IP address of an existing instance in the vpc network and finally specify vpn tunnel allows you to select an existing cloud vpn tunnel as the next hop so you can select one of these and then you press create in order to create our route dns domain name system service that publishes your domain names to the global dns in a cost efficient way in cloud dns the managed zone is the resource that models a dns zone dns is a hierarchical distributed database that lets you store ip addresses and other data and lets you look them up by name cloud dns lets you publish your zones and records in the dns without the burden of managing your own dns servers and software all records in a managed zone are hosted on the same google operated name servers these name servers respond to dns queries against your managed zone according to how you configure the zone cloud dns offers both public and private managed dns zone a public zone is visible to the public internet while the private zone is visible only from one or more vpc networks that you specify you can create a private zone with a different set of dns records specific to your vpc network called a split horizon dns so public zone is visible to the internet cloud dns has public authoritative name servers that respond to queries about public zones regardless of where the queries originate you can create dns records in a public zone to publish your services on the internet for the dns records in a public zone to be resolvable over the internet you must update the name server settings of your domain registration at your registrar cloud dns assigns a set of name servers when a public zone is created here is an example of an a record for the dns name www.example.com with a time to live 300 seconds and the public IP address private zones enable you to manage custom domain names for your virtual machines load balancers and other gcp resources without exposing the underlying dns data to the public internet a private zone is a container of dns records that can only be queried by one or more dns networks that you authorize a private zone can only be queried by resources in the same project where it is defined the vpc networks that you authorize must be located in the same project as the private zone you specify the list of authorized vpc networks that can query a private zone when you create or update it you can use private zones with shared vpc private zones do not support dns security extensions such as dnssec request for dns records in private zones must be submitted through the merdata server which is usually 169254 169254 this is the default internal name server for vm's created from google supplied images dns forwarding you can setup dns forwarding between your non-gcp name servers and gcp's internal name servers configuring bidirectional forwarding allows instances in your vpc network to look up the addresses of hosts in your on-premises multi-cloud networks this allows hosts on linked networks to look up addresses for resources in your vpc network google cloud platform supports dns forwarding in these ways one vpc networks support both inbound and outbound dns forwarding using dns server policies this must originate from systems in networks connected to the vpc by cloud vpn tunnel or cloud interconnect cloud dns private managed support outbound dns forwarding by means of forwarding zones each vpc network provides dns name resolution services to the vm instances that use it by default the vpc networks name resolution services are not available outside of that network you can however make them available to systems in on-premises networks connected using cloud vpn or cloud interconnect by creating a dns policy to enable inbound dns forwarding to the vpc network when a dns policy is configured gcp allocates an internal ip address in the subnet this will be used as a proxy for inbound dns requests you can specify a region or let gcp choose one automatically each dns policy for inbound dns forwarding allocates a single proxy ip address you can configure your on-premises name servers to forward to the proxy address as necessary so you can configure a dns policy for outbound forwarding by specifying an alternative name servers with internal ip addresses of other gcp vms in your vpc network or systems you can also specify alternative name servers using public ip addresses which must be accessible on the internet you can also set up outbound dns forwarding to the definition of a forwarding zone all matching queries for a forwarding zone are forwarded to a set of destination dns servers split horizon dns you can use a combination of public and private zones in a split horizon dns configuration split configuration is useful if you have separate development corporate and production vpc networks like for example you can define a private zone and authorize access from a development vpc network so that queries from vms in that network for dns records in that zone directed to other vms in the same network and then you can define a second private zone serving the same dns records with different answers authorizing access from a corporate network and finally you can define a third public zone serving the same dns records with appropriate public answers suitable for production let's take a look at an example of this so suppose you have created both a public zone and a private zone for gcp.example.com you configured the registrar for gcp.example.com to use cloud dns name servers so that your public zone is accessible to the internet and your authorized access to the private zone from your vpc network so in the private zone you create a single record foo.gcp.example.com this is an a record and it points to an internal ip address 10.128.1.35 and then in the public zone you create two records two a records one for foo.gcp.example.com and another for bar dot gcp dot example dot com they both point to public ip addresses so now a query for foo.gcp.example.com coming from a vm in your vpc network returns 10.128.1.35 which is the internal address a query from the internet however returns 104.198.6.142 because that's the public zone a query for the second one bar.gcp.example.com coming from a vm in your vpc network returns an error because there is no record for bar.gcp.example.com in the private zone and then finally a query for bar.gcp.example.com coming from the internet will return a successful 104.198.7.145 because there is a public record for that which is the last line in the second image table now let's take a look at creating a managed public zone and then we look at a demo for creating a managed private zone in this demo we learn how to create a cloud dns public and private zone so here we are back in our google cloud platform dashboard let's select or create a project first click down here and I'm selecting my project cloud dns demo you can alternatively create a new project if you need to so now my project is loaded go to the navigation menu go to networking and networking services and cloud dns click on this and here in cloud dns we can see dns zones and here we have a button to create a dns zone because there are none currently available click on this and in the create dns zone it's a pretty short form you have a zone type either private or public a zone name a dns name dns sec option and a description so for a public let's create one called pub test a zone would be pub dot my domain I'll let the dns sec to be off and then press create so now my public zone has been created and you can see the details here so now we can go back to the dns dashboard and we can create a new dns zone in this case I'll choose private so here I'll name this private test the dns name private and my domain name and I'll say I would like to forward queries to another server in that case I'll have to provide an external IP address so in this case I'll create a forwarding zone and then for the networks you can choose which networks you want this private zone to be visible to so here I can choose my demo vpc vpc which we've created in a previous lecture and now I press create so now we can see that the private zone has been created now if we go back to our dashboard we can see that there are two zones here one is public and one is private and we can see that the private zone is in use by one network which is the demo vpc that we've created previously now let's take a look at how to create a new record to point the domain to an external IP address and then create a C name record for the www subdomain in this demo we'll learn how to create records for our DNS zones so let's go to the navigation menu and go to cloud DNS which is under networking network services and cloud DNS click on that and this takes us back to our cloud DNS dashboard and we can see our two zones here that we previously created now let's click on pubtest1 which is the public zone and we can see a button here that says add record set and we click on that and here we can choose the DNS name and the record type A and the time to level 5 and the IP address that we want to create the record for so this creates an A record with the IP address that we choose now if we want to create a C name record it's similar to what we just did we click add record set again and under DNS name we type www and then for the record type we select C name under canonical name we enter the domain name followed by a period so in this case it would be www.mydomain.com and then there should be another dot at the end here and then we press create and that should create the record as we can see here C name record that's created with our domain name for www is available on the Google Cloud Platform with the load balancer Google Cloud Platform load balancing gives you the ability to distribute load balanced compute resources in a single or multiple regions it provides a high availability solution where resources can be behind a single IP and scale up or down with intelligent auto scaling cloud load balancing is fully integrated with cloud cdn for optimal content delivery cloud load balancing can respond to over 1 million queries per second it's a software defined managed service which means it's not instance or device based so you do not need to manage a physical load balancing infrastructure you can use global load balancing when your users and instances are globally distributed your users need access to the same applications and content and you want to provide access using a single IP address then you have regional load balancing this is used when users and instances are concentrated in one region global supports IP version 6 while regional only supports IP version 4 then you have external and internal load balancers external load balancers distribute traffic coming from the network like the internet to your gcp network internal load balancers are used for distributing traffic within your gcp network the type of traffic you need your load balancer to handle is also another factor in determining which load balancer to use for HTTP and HTTPS traffic a global external load balancer is required TCP traffic can be handled by global external load balancing external regional load balancing or an internal regional load balancer UDP traffic can be handled by external regional load balancing or the internal regional load balancing now Google load balancing is implemented entirely in software that's done with Google front-ends the Google front-ends are distributed globally and load balance traffic is synchronized with each other by working with Google's other software defined systems and the global control plane the Google front-ends terminate your user traffic as close as possible to the users and direct load balance traffic to the closest healthy back-end that has the required capacity as you can see in the diagram the global load balancing terminates outside of the regions for regional load balancing you have a separate load balancer for each region now this has an internal TCP UDP load balancing and it provides a standard tier of network services it provides standard tier of network service tiers although only IP version 4 is available the regional load balancing is also implemented entirely in software the instances are in a single GCP region and the traffic is distributed to instances within a single region network TCP UDP load balancing is used to load balance external traffic and internal TCP UDP load balancing is used to load balance internal traffic as we said previously HTTP and HTTPS traffic requires global external load balancing so that's where the HTTPS load balancing is used and it provides a global load balancing for HTTPS requests destined to your instances another type is a SSL proxy this is a load balancing service used for SSL traffic and it provides end to end encryption cloud TCP proxy is another load balancing service and that's also intended for non-HTTP traffic network load balancing is used to balance the load system based on incoming IP protocol data such as address port and protocol type at the network level the internal TCP UDP load balancing is a regional load balancer that enables you to run and scale your services behind a private load balancing IP address this is only accessible to your internal virtual machines these are the main components for an HTTPS load balancer you have global forwarding rules which route traffic by IP address port and protocol to a load balancing configuration that consists of a target proxy a URL map and one or more backend services each global forwarding rule provides a single global IP address that can be used in DNS records for your application target proxies terminate HTTP connections from clients one or more global forwarding rules direct the traffic to the target proxy and the target proxy consults the URL map in order to determine how to route traffic URL maps define matching patterns for URL based routing of requests to the appropriate backend services a default service is defined to handle any requests that do not match a specified host rule or path matching rule in some situations such as the cross region load balancing example you might not define any URL rules and rely only on the default service if it's a content based routing then the URL map allows you to divide your traffic depending on the URL components which we'll see in an example later on now if you're using HTTPS load balancing you must install one or more SSL certificates on the target HTTPS proxy SSL policies give you the ability to control the features of the SSL and your HTTPS load balancer and how it negotiates with the HTTPS clients backend services direct incoming traffic to one or more attached backends each backend needs to be in an instance group an additional service sync capacity metadata each backend needs to be in an instance group and needs to have additional serving capacity metadata the backend serving capacity can be based on CPU or request per second each backend service also specifies which health checks will be performed against the available instances now let's take a look at a demo in setting up an HTTPS load balancer as we can see in the diagram here we have two instances that need to be set up one for video resources and one for just resources so these are these will both be the backend services now they will be an URL map that needs to be set up that splits the traffic depending on the URL components whether it's for video resources or if the request is for just regular resources we'll be setting up a forwarding rule that uses a target proxy which uses the URL map for this this whole part is going to be the load balancer these are the steps that we'll be following in the demo as part of the demo we'll also be using a small script while setting up the VMs so the scripts that we'll be using are also here so you can copy and paste this if you're following the example in this demo we'll learn how to create a content based HTTPS load balancer so here we'll configure URL paths so that they start with video are routed to a certain instance group and the others are routed to a different instance group so we're back here in our Google Cloud Platform dashboard and let's make sure we select or create a new project so here I have selected my project load balancer demo click on that so the first step is to create our instances so here I'm going to create two VM instances so here we go to our navigation menu and go to our compute engine VM instances create a new instance press create we'll call this www set the region to us central 1 zone to central b and under networking we will add some tags here http hyphen tag and under management let's add a startup script here which will help us set up our VM so here this is going to be for the first one and it's simply setting up our web server Apache 2 and creating an html document called index.html so you copy paste this and let's press create ok so this is done so let's create another instance for the video so now I'm going to call this www dash video and similarly we go to we make sure that this is in the same region as well central 1 and we are going to go to management allow http traffic networking do the same thing added tag and use our startup script in this case we use the second one and the only difference here is we're creating a folder called video and we're creating an index.html file in that folder here so we copy this paste it in the startup script and create ok so that's done so now let's create a firewall rule so for that we go to navigation menu networking vpc network and firewall rules so here we create firewall rule and we'll call this www dash firewall and for the targets specific target tags and here we type our http tag the source filter source filter for IP ranges and the source IP range 0.0.0.0 slash 0 and select specified protocols and ports TCP we choose AD now if you're using HTTPS then you would use port 443 so let's create this ok so now we need to create a global static external IP address for our load balancer so for that let's go here to external IP addresses and then reserve static address and let's call this ib ip 1 set it to ip version 4 type global press reserve so that's created now and now we have to create instance groups for our virtual machines so let's go to the navigation menu compute engine instance groups click on that and we create instance group let's call this www hyphen resources set the region to us central 1 and 1d and under group type unmanaged instance type and under network we select default and under VM instances let's select our www instance let's create this now let's create a second instance group for our second VM so we'll call this www dash video central 1 b unmanaged network default and we choose our other instance here press create ok so now we are ready to create our load balancer so let's go to navigation menu networking services and load balancing so here let's create load balancer so here we have a choice of http tcp and udp so we want an http one so we press start configuring so for the name let's choose www dash map press create and let's check backend configuration so here let's create backend service and here we will choose web dash dash service instance group let's select our resources instance group and our port 80 and we'll just leave these as defaults for now and the health check we have to create a health check we have to name the health check http basic protocol port 80 and let's save this so now we can create our backend service now let's do the same thing for the other resource we'll call this video service select the other instance group port 80 and http basic check video set the protocol to http and save and continue and let's create this so now we have both our backend configurations done so now let's set the host and path rules click on this so this one is already the default which is any unmatched and paths so here for this one we have to set it to star for the hosts and for the paths let's set it to slash video and slash video slash star now let's configure the front end so here we'll name this as http content rule in the protocol we leave it as http so for ip version we leave it as ip version 4 and for the ip address we choose the ip address that we created previously this is the external static ip address and we select port 80 because this is for http currently and for http s then we'll have to provide the ssl certificate here so if we chose http s for example then we get additional fields here where we have to set the ssl policy and select or create an ssl certificate and the port will be 443 so in our case we'll just leave it with http for now and make sure this is the external ip address and we are done here so now we can review and finalize the all the options so we click on review and finalize so here we can see all these steps the fields for these steps that we populated here so we can see that there are two backend services here one and two and then this is the hosting path rules and this is the front end so now let's go ahead and create this press the create button and now our load balancer is being created so our load balancer has been successfully created here and so let's click on this to see the details so here we can see that the instances are not healthy if it says 0 of 0 that means it's not good it should say 1 of 1 refresh so now we can see that the the health check shows 1 of 1 and 1 of 1 so sometimes it takes some time once you create the load balancer for this to refresh and get set up on the backend so now let's go to our ip address here copy this and we can just go to this tab so here we can see our page has loaded this is the index.html page for the resources so now here if we type video and press enter we can see that the other html page has loaded from the other vm instance with ssl proxying for your ssl traffic you can terminate ssl sessions at the load balancing layer then forward the traffic to your virtual machine instances using ssl ssl proxy can handle https but this is not recommended you should instead use https load balancing for https traffic this is because https load balancer has additional functionality like support for a cloud cdn or the ability to forward requests to different instance groups based on url host and path as we've seen in the example and it provides better instance utilization ssl proxy is a load balancing service that can also be deployed globally if the closest region is at capacity the load balancer automatically directs new connections to another region with the available capacity for the best security you can use end to end encryption for the ssl proxy deployment to do this you should configure your backend service to accept traffic over ssl this ensures that the client traffic decrypted at the ssl proxy layer is encrypted again before being sent to the backend instances scaling an https load balancer spreads load across backend services which then distributes traffic among instance groups now within the backend service you can define the load balancing serving capacity of the instance groups associated with the backend when you attach an auto scaler to an https load balancer the auto scaler will scale the managed instance group to maintain a fraction of the load balancing servicing capacity auto scaling only works with maximum cpu utilization and maximum requests per second per instance this is because the value of these settings can be controlled by adding or removing instances when an instance group reaches the serving capacity the backend service will start sending traffic to another instance group when you attach an auto scaler to an https load balancer the auto scaler will scale the managed instance group to maintain a fraction of the load balancing servicing capacity that's been set up