 Hello everyone. So now we have a talk on analyzing COVID vaccine popularity in Europe using fast API Postgres and Azure. Our talk will be by Ravya Imad. She's a program manager at Microsoft with a focus on building great cloud products for Python web developers in Azure. She has a degree in computer science and astrophysics from Harvard and is currently studying data analytics at Georgia Tech. Hello Ravya. Hi Yasmin, nice to meet you. Thank you very much. We're very happy to have you today. Wonderful. So yeah, is it time to just go ahead and get started now? Yes, definitely. Awesome. Awesome. Well, hello everyone in Europe and elsewhere who are joining me today. I'm so honored to have the privilege of your audience and I have a lot of exciting things for you today. So first off, a huge shout out to the Python community from Microsoft. We'd love working with you and building great tools for you. And thank you, Europe Python, for having us here. As Yasmin mentioned, the title of the talk that I'm going to be speaking to today is really analyzing COVID vaccine distribution by type in Europe. I'm going to be using a few different technologies like Fast API, Python Fire, Azure Web Apps and Azure Database for Postgres SQL, a flexible server edition for this. Some of these technologies are new and I'm going to be giving you a little bit of a tasting on all the different things that these technologies bring. But before I do that, I want to give you just a little bit of context about me. My name is Rabia as Yasmin mentioned and I'm a program manager at Microsoft and I work on the Python team. And so I am really a good contact for you if you want to learn about anything that's going on in the Python world at Microsoft. I've left my Twitter and my LinkedIn in the slide deck. You can also download this separately. And yeah, don't hesitate to reach out, message me directly or otherwise and yeah, we'll stay in touch even after the conference. So with that said, let's go ahead and get started. And I'm just because we have a lot of ground to cover today in 30 minutes, I'm going to give you a brief tasting of what we're going to be building today. And really what I'm going to do is build the architecture that's shown in this diagram. So I'm going to transfer COVID data to Azure Postgres. That's the database that's shown on the right hand side. And I'm going to connect it to Azure App Service, which is basically going to host a website. Then I'm going to use Jupyter to talk to that website and then visualize the data in a meaningful way so that I can get insights. And then of course, I can use my computer to go to Jupyter and actually see the results and share them with you. So ultimately, that's the goal of this talk today. And yeah, let's see what we need to go ahead and get started. There are a few things that I already installed on my machine before this talk. So I have Python 3 installed. I have Visual Studio Code installed. I also have an Azure free account. You can go ahead and get some free credit so that you can also try this exercise yourself and learn as you go along. And then I also have the Azure CLI installed. This is going to be important as I run certain Azure specific commands in my terminal. I've also already cloned the code that I'm going to be using from GitHub repository. And I have the dependencies installed as well as the virtual environment. And don't worry so much about following and retaining all of the information that I'm going to present to you in your head. Detailed instructions on what I'm going to be doing are going to be in the clone repository in the readme. The slides are going to be available for you. So just come along for the journey and have some fun and see what wonderful things Python can do. So with that, let's talk about data. Let's talk about what we need to do to understand data to formulate some interesting questions and filter some relevant components. So when I was thinking about the COVID vaccines and Europe, I wanted to go and get some data. And I did that by going to the website of the European Center for Disease Control that's shown over here. And I downloaded the data from the ECBC to my local repository. And there were a lot of columns in this data set. It was very rich. But there were really three that really piqued my interest. The first one was the number of doses that a country received from a given manufacturer, the name of the country and the type of vaccine, the name of the manufacturer. And if you look at the raw data, you know, it has a lot of abbreviations like the country name or the vaccine manufacturer names. They're not really spelled out completely. So I did a little bit of digging and I figured out what those meant. And so I made dictionaries in Python to kind of use them so that, you know, when I share my results with you, they don't have all these acronyms. So over here, I have the vaccine types. And you can see that this J-A-N-S-S, this is the Johnson & Johnson vaccine. And this A-T reporting country is, in fact, Austria. So once I got this data, I really got curious about, you know, what are the most commonly used vaccines for every country in Europe? And what are the least popular or least common ones for the country? And really, what is the difference between how common they are? Are they more evenly distributed? Are they not? Does it really change by country? So there are lots of different questions that I wanted to answer. And so with that, I had a few things to go right off of. And so, yeah, time to go and analyze the data and get the answers to these questions. So the very first thing that I needed to do was actually set up the database and the local connection variables so that I could actually do something with this database later on, such as actually transferring the data maybe and maybe retrieving it. So the very first thing that I'm going to do in order to set up the database is I'm going to come back here to my terminal. And you can see that my virtual environment is activated and I've installed all the dependencies. I mentioned this already in the prerequisite. And instead of actually writing down all the commands myself in a live demo, I have a really cool script so that I can just write step zero and it's going to tell me what it's going to execute. And then that way I can actually spend less time making typing mistakes and more time actually explaining things to you in a clear and concise manner. So step zero is something called AZ login. You can see this over here. And what it does is I'm trying to actually log into Azure from my terminal so that I can access the resources that I have there. And you can see that it's asking me to sign into my account. And after a brief period of time, it says, oh, yeah, you've logged into the account. You can close your window or just go back to your step. So great. We've already done the login. And now we have to do something else, which is run this command, which is Azure config param persist on. Now, you might not know what this means, but essentially what happens is that whenever you're deploying resources in Azure, there are a lot of different parameters that you need to set up. And you need to mention them in your commands. And sometimes that means that the commands can appear to be rather long and repetitive. So if you have AZ config param persist on, you can use shorter commands and it will remember some common things like the names you used and things like that so that you don't have to be typing them all the time. So once that is done, I'm ready to actually go ahead and talk to Azure. And the first thing I'm going to do is I'm actually going to go ahead and create a database in Azure. And that's going to be here in step two. And the command that I'm going to use is this one. It's AZ Postgres Flexible Server Create. Now, what this is going to do, like this part of it is it's going to tell Azure that I want a database that's a Postgres database and I want it to be of the flexible server type. Now, that's a Microsoft Azure specific type. And what it is, is it has a free offering included in it. And so that's why I want to use it because I want you to have the best, but the freest possible resources for you for your experimentation. And then you can also see that I've opened the access to this database to everyone using dash dash public access all. Now, generally, whenever you're procuring a database, you want to have a restricted access, but I didn't want to delve into firewall rules just yet in this lightning talk. So I'm just going to open this for now, but please look at the docs and see how you can configure the firewall in a more restricted manner. And then finally, there are some other parameters here, and those are really just needed to make sure that I can use the free tier because Microsoft provides a 12 month free version of Postgres Flexible Server if you procure it using these parameters and I certainly want to do that. So that's why I have these there. Now, you can see that as soon as I executed the command, you know, lots of things were being printed and this command is running. Now, the run for this command actually takes a few minutes in line with the run it would take on any other cloud. But I don't want to spend time on seeing this run completely for this demo because, you know, I want to be precise about it. So what I did before the demo is that I actually went ahead and I ran this in advance and I'm going to switch the tab now and you can see what happens when I complete finish completely finished running this test. So what happened over here is that once it's done running, Azure gives you an output, which is which is this green output. And this has some very important information that you want to keep. And so what you can do is you can copy this and put it in a safe, non checked in location on your computer, just for reference later on. So you can see that there is a database host name, a database password, a user name, and then also over here, it will say that it's creating a Postgres flexible server DB. And that is really the name of your database. So let me also save that here and then go back to my very first terminal, which is still going to be running. And I'm just going to go ahead and kill the process right now. Since I've already shown you what the results are. So with that, what we now need to do is we have a database created in the cloud, but I need to actually go ahead and connect to it so that I can maybe talk to it from my computer. And the way to do that is going to be shown to you in the next step. And I'll walk through what that is. And it's by executing these four commands. And what I've done here is that I'm exporting something called the server name, the admin username, the admin password, and the DB name. And these were all things that I got from. This is the database name. This is the host name. This is the password. This is the username. And you might notice that the ones that I have here are a little bit different. Those are really for security purposes. I wanted to do a demo with different passwords than the one that I was going to show in the slides. But anyway, with this done, I now have some variables that have been exported. But again, these variables are just in my local terminal. I actually need to do something with them. And what I need to do is I need to create a connection string. And a connection string is basically the combination of these things, the server name, the database name, the username, the password, and stuff like that. So that can be done by executing this line of code. And once this line has been executed, you can run echo on the connection string variable and make sure that it actually looks proper, which it does. So I'm going to go ahead and copy this connection string. And I'm going to paste this again in my little notepad that I have not checked in, just for now. And it's going to become very relevant in a few seconds. Now, why am I naming my variables, connection string and all of those things? That's because of the design of my code. And I'm going to get to that in a brief moment. But first, before I do that, let me do a quick recap of what we just learned. And what we learned was that we need to actually activate the virtual environment and install the dependencies, log into Azure and persist the parameters. Then what we need to do is create the Azure database for Postgres SQL flexible server. We had this command. And then there were all these credentials that we got from that command. And then we use the export name, sorry, we use the export command to export these variables and finally create a database connection string. And really, the role of this connection string is to connect me to the Azure database for Postgres SQL flexible server here, which is at the very bottom of the slide. And I'll show you how that can be done in a brief moment. So next, what I'm going to teach you is how to load the data that you downloaded from the ECDC website and actually put it in your cloud database and then retrieve it to verify that you've done the right thing. And to do this, I'm going to actually be calling a function called load data. And this function is defined in my data work dot py file, you can see that it's over here. And it's really using a file called COVID data dot CSV. This is a giant CSV file that I got from the ECDC website and it has many columns. But as I mentioned earlier, we are only going to care about a few different columns. And so what load data really does is that it creates a table called vaccine data, this is the most relevant part of it. And what this table does is it takes the number of doses that you received, the reporting country information, as well as the vaccine type information, and it puts it in a data table called vaccine data. For references also keeps the raw data in a separate table just as a best practice, but you don't really need to have that. And then finally, what it does is it tells you that the data was loaded successfully. Another very important thing that I'm going to be doing is actually verifying that the data was loaded correctly. And that's by calling something called get all data and using it to literally print out the three columns from the vaccine data. Now, these two functions are really important, but I want to be able to call them from the CLI. And this is really where Python fire comes in. And what Python fire does is that it lets you call these functions through the CLI using commands like this. So if I write load data from the CLI, it's actually going to call load data. If I call get all data, it's going to call the get all data function. So that's the way of calling all your functions very quickly and in an elegant manner directly from the CLI. And we're going to use Python fire for that. So let's let's do that. Let's go on to the next step, which is step six. Oh, and one more thing. Remember how we talked a lot about connection string and connection string is really important because in order to load data, you have to connect to the database and in my code, it actually gets an environment variable called connection string and then use that along with Psycho PG2, which is a Python driver for Postgres to actually connect to the database. So that is the importance of having that connection string variable. So now that that's done, let's execute step six, which is simply this command Python three data work dot py load data. And what this is doing is actually going ahead connecting to that database and putting all the data both third and raw into the database. And it says that the data was loaded successfully. So that's great. And now I want to make sure that this is that the data actually looks like what I think it should look like, which is three columns. And so I'm going to call step seven, which is going to get all data. Hang on a second. You can see this is the command here Python three data work dot py get all data. And you can see that Yep, it's a lot of tuples that are three in number. So with that done, I'm actually able to connect to Azure from my local machine using a Psycho PG2, as well as all of these connection strings that I had. So let's do a quick recap. What we did here was we loaded the data, we refined it, and then we retrieved it using some code that we use Python fire to call using load data, you could pass in a specific CSV file, or you could just use the default name, either one works given the way this code is written. And then again, we did data verifications using the get all data flag. So now we have our database piece really complete. Now maybe let's do some interesting manipulations with the data. And that is really the next section of this talk, which is gathering insights and serving that slowly with the backend AI that built using fast API. And the way to do that is, let's go on to a quick demonstration of the things that you can do. So I've talked about how, you know, we got lots of different types of data. But really what I'm interested in was the vaccine percentages. And so I have a function here that actually goes ahead and computes the percentage share of every vaccine manufacturer for every country that's in the data set. And I'm not going to go into like how the function works. It's generally just simple arithmetic that it's doing. It's counting the numbers, and it's, it's spitting out a percentage. And you can actually verify that it can do that by again invoking, let's see, oops, we already did step seven, we can do vaccine counts. And then what I just did was simply call the function that would give me a vaccine count for every country. And I piped it using jq so that it's pretty printed. And so for example, you can see for Luxembourg, these are the counts of the doses that have been received per unit for Luxembourg and for all the other countries. So this is really cool data. But you know, maybe the counts are actually a bit overwhelming to process. So let's see if I can get some, sorry, some some percentages. And the way to do that is to go ahead and run this the same command. So the command that we just did. But instead of get country vaccine counts, you can actually even get country vaccine percentages. And so I'm just going to go ahead and do that here. And now I can see, for example, for Austria, AstraZeneca has 24% share, Pfizer is at 63, G&J at 2, Moderna at 9.76. So this is much more useful data. But now what I want to do is, you know, it's all well and good that it's on my machine. But you know, maybe and like a website that can build actually has the data so that I can analyze it and others can analyze it. And this is really where fast API comes in and fast API is a wonderful framework for building Python back ends. And the best part about it is, well, it's fast because it's asynchronous, it supports async code natively. And it also allows you to force a certain response type in your API responses. And so that can also really help you in making sure that you get the right type of data. And fast API is just like, wonderful to work with, I will highly recommend, particularly people who are new to building API is to certainly check it out. And so I have something here, which is I'm initializing fast API, and I have an app, and then there is a backslash pair, which is the root. And root is supposed to give you get country vaccine percentages. I also have something called data, which gives you the same. Then I also have an endpoint called most common, which gives me the top vaccine by the country, as well as the least common, which gives me the least common vaccine for the country. So with that done, let's figure out a way to actually get this up and running. And to do that, I accidentally ran this command quickly before. And I suspect many of you might not have actually seen it. So I'm going to go ahead and kill the port that may have been using that command, if it's live, and it looks like it. Yes, it is. So you can ignore this part if you are seeing this demo later on. And what you can do is 56078. Okay, great. So it looks like that our server is not running. But what I can do next is I can run step eight, wait, no, it's not step eight, step nine. And what step nine does is that it runs the server locally, and it does that by calling you the corn, which is an as the server, it's saying, go look into the main file and start the app, an app called app, and then have an all refresh. And so if I go to localhost 8000, like I'm just going to copy paste this link, and go back to my computer, zoom it in, and then go to this website, you can see that it is actually serving the data that I wanted. So that's really wonderful. And if I actually go to the other endpoints, like say most common, which was which I mentioned, you can see that it looks like Pfizer is a very common one in Europe, with the with the exception of Lichtenstein. And then we also have the least common one. And it seems like also, yeah, it looks like there's a pattern, where perhaps J&J is the least common or unknown types are the least and Lichtenstein is that Pfizer BioNTech. So really interesting data there, I wonder why all of this is happening. But at least now we can we can take a pause and let's just review what we've learned so far, before we go on to the next step. So let's review really quickly. Remember that we talked about getting the vaccine percentages by executing a command like this. We also created APIs to show the data. So there was the data, then there was most popular, I renamed this to most common. And then we serve them locally using UV corn. And then finally, navigated to the local host to see the results. So you can see this is me, I'm connecting to my own local computer, which is connecting to a database. And my own local computer is running a fast API. So again, that's all good. But let's take it to the next level and launch this website actually on the web. And I'm going to do that using Azure App Service. And App Service is just one of my favorite things. So it's Postgres flexible server. And I'll tell you a little bit on why. Now, if I want to package all of this information and put it on the web, all I have to do is run this command. That's it. Easy web app up. And because I want to be specific about the name of my site, calling Europe COVID vaccines. But if you remove this flag entirely, it would generate a random one for you and you wouldn't even have to do that. So easy web app up what it's doing right now is that it's taking all the code in my repository. It's and is going to zip deploy it to Azure. And Azure is really going to make sure that everything is up and running for you. So you don't have to provision containers or anything like that. It's really going to install your requirements for you. All of that cool stuff is going to be done automatically, which is really really cool. But while this is happening, remember that we're still going to need to connect this website to to to that Postgres database. And remember, I once asked you to save this connection string. And I'm going to tell you where this is going to come in useful. I'm just going to go ahead and copy paste it. And I'm going to go now to the Azure portal. So that's portal dot azure dot com. And I am going to zoom in very quickly. And I'm going to search for something called app service app service and Azure web apps. It's really the same thing. So I'm going to go ahead and click it. And I'm going to search for the app that I just built, which is called Europe COVID vaccines. Just because I have a lot of apps and you can see that it says that it's running below it is still deploying. And what I need to do now is I need to configure the parameters. And to do that, go into the settings and click configuration. And then within the configuration, you're going to see this sign on new application settings. So let's click that and call it connection string, and then put the entire string that we had over there. Sorry, hang on, I'm going to hide this and then click OK. And then click save. It's very important that you click save once you're once you've done updating the setting. And then the next thing that you have to do is you actually have to set up the server that is going to actually run your app. And app service uses Gunicorn, but you can specify the class and specify it to be UV corn. And so what I've done is that I'm saying that when this app wakes up, use Gunicorn with one worker, that's this flag, and use the class of UV corn workers and open up the main file and then execute the app. And once that's done, it should be able to run beautifully. I'm going to click save again. Very important to do that. And then click continue. And it's going to tell me that it's updating the app settings. Now let's go back and see if the code deployment was complete. And it looks like it was over here. It says that the website is up and running. And so let me see if we can browse it. And I'm going to go to step 11. And all that happens here is that I go to AZ web app browse, and it opens up things for me. And if I go hang on a second. Yep, if I go here, you can see that I have the website up and running Europe COVID vaccine dot Azure website dot net you can access it right now learn a little bit about the percentage shares in your own countries, or any country of your interest. And then you can even go to the most common and least common endpoints that I showed you. So with that, your back end is now live, which is really interesting. And now the next thing I'm going to do is do a quick recap. Remember, at this stage, the things were pretty simple. We ended up executing AZ web app up with a name. And then I set up a startup command in the Azure portal, as well as added a connection string. And then I went to view the website. And actually the one that I have right now is more pretty printed. But I use the non pretty printed versions like a squeeze all this information in in into the deck. So with that done, I am now talking to Azure app service, which is now talking to Azure database for Postgres flexible server. So at this point, I'm fully in the cloud, which is great. And now my app can see just about can be seen by just about anyone in the world. And so, you know, that's great. But now what I want to do is actually create a visualization so that the numbers make more sense to me. And I'm going to do that using a very simple juperscript that I'm going to show you. And that is going to be in step 12. And all step 12 does is that it opens the visualization notebook that I have on my GitHub. And here it is, you can see that I'm making a request over here. I'm making it to a backup website that I had for this talk. But if you wrote your COVID vaccines, it's the exact same thing. You can see that the data is exactly the same from that endpoint. And what happens is that I picked a country that I'm very interested in visiting. That's Norway. I love the fields there. And so I wanted to know, well, I wonder what Norway is doing and what the distribution of the manufacturers looks like. So I went and I created the data and I got the country information for for Norway. And that was really cool. But I really wanted to visualize it because it just I like to see things in a pie chart format. And so I ended up making a pie chart out of this. And what I have here is the distribution for the vaccines for Norway, we can see that more than half the population seems to have that that's vaccinated seems to have Pfizer. We have a sizable amount of unknown types, which is 25% about a quarter of the population, not sure what happened there. And then we have a lots of Moderna AstraZeneca and GNG as well at five at seven seven and 5% approximately. So with that done, we are good in terms of the technical aspects of this demo. And I'm just going to review the last piece really quickly. Remember, I created a visualization by calling a backend API. And really what I had in the end was me talking to Jupiter, which was talking to Azure Web Apps, which was talking to Postgres, and really getting a nice pretty graphic at the end of it. And so so that's what I have here. And we built it using all these different technologies, which is really exciting. And I've put in all the resources, including how to deploy and web app and database together from the portal. I want to give a specific shout out to this because I know a lot of people don't like doing things in the terminal. So if you're a more visual learner or just a programmer, feel free to follow this link and try and doing this. I also have the code here, the readme instructions, as well as the data. And finally, I want to thank you all for being here. The one thing that I really love if you could do is please follow me on Twitter. I'm trying to build my Twitter app and also try and network with me on LinkedIn, comb the repo, play with the data, tell me what you've learned, tell me what your insights are. The code is available for you on GitHub. And most importantly, have fun. And it was so nice meeting you all today virtually. So that is it for me. I'm going to go back and talk to you now. Thank you very much for your talk. It was very informative. And all of us, we've definitely learned a lot. Thank you again.