 The fourth and last part of this lecture is now to make our application persistent. So we have deployed it to Heroku. We have implemented all the different endpoints. Now the only problem we have left is that we have only used local variables. So whenever we restart, they're lost, so re-initialized. It's hard to distribute, so let's say our our load increases. We have a lot of users and we might want to have multiple instances of our endpoints of our server, then they all have their own data, and it's not very good. So instead we would like to use a database that then for example is persistent. We can restart the data is still the same and we can have multiple applications that could access the same data and they all have always the current state of it. We'll use MongoDB for this. You could use MySQL just as well. I use MongoDB to show you something else. I use it because it's very common and I use it because it's for free included in Heroku as well. So we can also use it in the deployed backend. MongoDB just as most databases has a number of things you get for free. You get unique IDs that are auto-generated by MongoDB, so you don't need to deal with this. You of course get efficiency. Databases are very well optimized for these kind of use cases. We deal with to quickly find something. You don't need to write the for loops yourself or so on. And there are some things which I put in parentheses here. There are some additional plugins you can use for Mongo and for also for relational databases that for example give you input validation. You can define ranges and types for attributes and you don't need to deal with all the if-else statements to check whether the values are correct. And another thing I've already mentioned that sometimes you do not want to return the entire resource, maybe only certain attributes, and again that's something that a couple of plugins on top of Mongo allow you to do very very easily. Now we use the MongoDB module for Node.js. That's the official one. It's very basic, so it allows you to do all the different things. If you want to use MongoDB in production, so if you want to keep playing around with this stuff, I would check out Mongo, which is a module on top of MongoDB that allows you to do a lot of things in a very efficient way. For example, this kind of input validation that I've talked about. So that's strongly recommended. The official website is MongoJS.com and then I also attached here a tutorial that shows you how to do this and you can just install it in the regular way, but we'll use MongoDB. Now there are a couple of things you need to consider when you want to write a persistent application. The first is we actually need to connect to our database and for that we need to have it running. So that's maybe where I can start here. I need to have a MongoDB installation. I have installed it on my computer. You can do the same and then you run it just like every other server. This is a blocking command, so you see that it's just running. That's fine. Then we go into a different tab. Here I will start my application. Now in the application itself, I need to first of all include Mongo so that you have seen before. I do require MongoDB and I actually import the Mongo client. That's just a part of this framework. Then I have to have the URL. In my case, that's MongoDB. You see this is a different protocol. We're not using HTTP. MongoDB colon double slash localhost 2717. That's the regular address for MongoDB. You can change this. You can run it on a different port, but that's the standard. Then you need to connect to it. You need to use the Mongo client for this. Mongo client connect to the URL. Then as we're used to it, we have a callback function that gives us either an error or the database object. This is then the object that we use to, for example, insert new to-dos in this case. If I cannot connect to my database, I'll throw the error. I'll just display it. I crashed the application. It's maybe not the best way, but it's okay. Otherwise, I'll get one of my different databases. On Mongo, you can store a lot of different databases in parallel. I'll just use this name. If you don't have to create this before, Mongo does it automatically if it does not exist. The first time I start this, this database will be created. Then this is just the same command as before for starting Express. I actually make sure that I have a connection to the database before I start my HTTP server, my Express application. Please note that the DB object here, this is something I have to find up here because in my application, I now want to access this for actually using the database. Now, I'll walk you through the different operations, get, post, put, delete, and show you how these things work. I'll go through them rather quickly. We might do a bit more in class, but it's just to show you an example of what the difference is to before. In MongoDB, you do not have tables. Instead, you have collections. That's something you have in document databases. It's a very similar thing to a table. If you know SQL, you can just imagine this as a table. When I want to get all my users, I get the collection of all the users, the table of all the users, and I get all of them. I basically use a find command that's comparable to a select in SQL. Instead of the star in SQL, I just have the JSON brackets here that are empty. Basically, I find without any condition, find everything. Whatever is returned, I then convert to an array. This is again a callback function thing, but once I get this array back, I return it. These are two lines more than before. Before that, I only had a single line, but I'm actually accessing the database. I get everything back. What I have not done here is error handling, and that's maybe something that is important. Let's just look into it. Here, I have the entire application, so you can also look at it yourself later. Get users. I return 200. If there is an error, then maybe I should return 404, and I just return the message, users not found, or something like that. Maybe this is something I need to investigate further in which cases I actually get an error here. This is from the to array function. Maybe this should actually be a 500 error. Error 1, accessing database. Something is wrong in my collection. I cannot convert this to an array, so I'll just give the error back to the user. This should work. Let's try it. I should update my code because I have some old statements that don't work anymore with MongoDB, so I should do that. You see there is no error. ExpressApp is listening. This seems to work. Let's try to do a slash users. I have actually used this before. It's probably from last year, so I seem to have three users in my database. What you might notice here is that the ID looks slightly different. In MongoDB, the underscore ID is basically the primary key. That's the auto-generated ID, and you see that it's this strange string of numbers and characters. That's my unique ID. That has worked. That's great. That was again the easiest one. The important thing is we call the right operation on the database. If this would be an SQL database, you would have your select statement here. Then we would just return the array. If we want to create, and now this is really where the power comes in, where it gets much easier, we create our user array. We create a new object with username and age. Both of them are in our body. Then we just say db.collection is the same as before. Insert one. Please insert one new file, and we just give that function our JavaScript object. Then we again have a callback function. If there's an error, we do something. If we get the user back, this means it has worked. Then we return that one. The user object looks a bit strange, so we can look into that. There are a couple of other fields, but essentially the important part is this one here. Then I return 201. We have created it. We will see the important thing is what I put in as username and age. What I get back is actually different. I get username and age back, but I also get the underscore ID. I actually get something that the database has worked with. That's the request down here. This part, important. I still do the check whether all the parameters are there, so that has not changed. I still need to make sure that the request is correct. Then I insert it. Again, I throw the error here. That's not very elegant. Instead, I should maybe return that something has gone wrong. I'm talking about that. I also have errors in here. I should return if something is bad. Again, this is a server error. Something has not worked. It's not declined usually. I should send 500 back. Again, let's try this. Post requests. I have to go into postman. Users. Now what I need to send is username and age. I send and I get something back. First of all, I get 201 back. It's successful. As you see, I get the username and the age back just as I've entered, but I also get the auto-generated ID. If I now do a GET request, I should have the user here as well. That has worked. Also, the important thing is now we have persistence. If I close this and if I start it again and I do the same GET request again, then I'm still here. This is now much, much better. I do not anymore have my objects just in the memory. What I do is I create a regular JavaScript object that has all the right attributes and then I insert it. With the standard Mongo module, there is no check whether this is any how conforming to a certain standard. I can insert something that is completely different here. For example, I can remove age and I can put something else in there. The database does not check that. In MongoDB, you can insert objects that have different format, no problem. I can do a lot of mess here. That's again, where I would like to point to Mongoose. There you have, for example, ways to say anything that's inserted into users has to have a username and has to have an age. The age has to be positive or similar things. Here there's no check, so we can do whatever we want. As I said, the important thing is that MongoDB actually auto-generates this so-called object ID here. Now, let's do a specific one. We want to find a user with a specific ID and we get that ID from the parameters. We first have to convert it to an object ID so it has to have the right format. That's, again, very specific to this case. Then we use, again, find, but this time we say find one. Find me a single element from our collection that's equivalent to a select statement with limit one in SQL. Then we have here this curly braces underscore ID colon ID. This is basically a query. This is saying find me one element that where this condition is true. This is, to summarize, the same as if you would do in SQL, select star from users where underscore ID equals ID limit one. That's essentially the thing. Then the same as before, if we get an error, send back something. If we get the user then send the user back. Again, that's the code that's here. There is nothing specific here. Right. Now we want to update something. What we need to do here is we need to find the right thing, the right object. Someone has requested to update the user with user ID. Again, we convert it to an object ID. Then we create a new object that has the new values, username and age. They have to be in the body. This is now something that is, again, specific to Mongo. It might look different in different databases. We say use the set. This dollar here is an operator. It basically tells Mongo set the following values to whatever we provide here. Then we say find one and update. Find the entry with underscore ID equals to ID and update it accordingly according to this object. There are other, these kind of operators, if you're interested as, for example, want to remove certain things, which can be good if you have a collection. You have an array and you want to remove one element out of it. It requires a certain update operation which we provide here. That's then what we do. We just update the specific things. You can implement the patch in exactly the same way. The only difference is, again, that you need to check which of the two parameters you actually have available. Finally, we want to delete. Deleting in Mongo is using a delete many. That's for deleting more than one object from a collection. If we use the empty curly braces, it, again, means no condition. Just delete all of them and we get something back. The weird thing in this standard Mongo module is actually that we want to return all of them, but delete many actually does not return anything. What we do here is we first get all of the users and when that is successful, we delete them and we return the user array. That's what we have done. All the other endpoints here are similar, so there is no specific thing. I will upload this so you can have a look at it. Note that I have not implemented all of these endpoints. Just some of them. You'll still see that there are some for loops to find the right things. If we would have used a more, okay, this one does not even use the persistence that explains it. If we would have used a more advanced database module like Mongo's, then a lot of the stuff you get for free. For example, you see that finding a specific user ID does no longer require us to loop through things. That's very nice. This works reasonably well. Now, we want to get it back to Heroku. The issue is that, again, we have something hard coded. Before that, we had the port. We could not access the port because 3000 was the wrong value. Here, it's the same with our database URL. Most likely, we cannot directly access this because MongoDB is installed somewhere else. In Heroku, we again, similar to the port, we have an environment variable that allows us to access the right URL. We have to get that. To actually get a Mongo database on Heroku, you can use MongoLab. That's a service that you can use to get an application database. If I let's see whether this is still up to date or whether they have changed it, create MongoLab. It actually now is creating a new database. I can change. I see that they're using a different URI, a different environment variable. Probably, they have changed the way this works. Let's go into the application and have a look. If I go to Resources, it says there are no... Okay, I ended up on a different application. Let's go there. I see. It probably has added a... There was already one. That's not the one I want. I have already created one. If you do not have any, you can also do it here on the web interface that you can just search for MongoDB and you just get a new database. This is, again, a plan. You see here, it says free. If you want a real database that has, for example, in this case, the free version does not have any backup. If you lose the data as your problem, there are plans that then cost money. The cheapest one is $18 a month, which is quite some money, but you get it. Basically, you don't need to think about backup and stuff like that. Good. Let's try whether this works. Now, I have already modified it here. You see, this is my port that's made for deployment. Here, I have introduced the MongoDB URI. It's the same as with the port. I just have an OR statement that says, if this environment variable is not available, then just use this one instead. I can use it locally. That's why I already had a database because I was in the wrong folder. Let's change it to our application that I worked with before. I just used the Mongo file, the one that I just changed, which means I need to update my package.json to actually use that one. The dependencies need to be updated. I have expressed body parser that's already there, but I also need to add MongoDB. That seemed to work. If you look at the package.json, we now have the MongoDB dependency as well. Now, I can add the database again. Since here, I should not have one yet. Now, it's added and they say the MongoDB URI gives me that information where it is. Now, I should just be able to redeploy. I, of course, need to add the file that I have just created here. I commit this, db added. If I have done everything correct, then this should work out of the box. It's being launched and it at least does not crash. That's a good start. If I go to slash users, I get some kind of error and that's not good. Let's see what it says. If it says something, yeah. Here, you see another issue that most likely is the problem here that, as I said, I have used some kind of strange way of connecting to it that apparently does not work anymore. That might simply be the problem. It might also be that there are simply no users. We can try to add one and then we'll see. Again, we are back to debugging how does our application work. Do a post and I want to send username and age. Again, there is something wrong. This seems to indicate that the access to the database is actually not successful. What I could do is now go into the different logs here to try to figure out what is going on. I could go to MongoDB to MLAP and look at the log whether there's anything that has gone wrong. I won't be doing this in detail now here. You see that this is now the next problem that apparently does not work as much as we would like to. It could also be the database itself. There are lots of things to debug always. That's simply what you have to live with, that when you go from a local environment to a production environment you have a lot of different pitfalls that you might have to look at. That's as much as we'll do in terms of deployment. One last thing I wanted to mention is the use of middlewares because Express allows you to do that. In fact, we have been doing that all the time. What a middleware is, you hear that word a lot in networking applications, networking technology, is something that is somewhere in between in the middle of our request response chain. We send the request, we get a response back, and somewhere in between there we do something. What a middleware does is it receives both of them, request response. It does something with them, so it can modify them, it can trigger some other functions, and then it just passes them on, it continues. That means that you can stack or chain them. You can run five middleware functions in between, and it does not change anything. It's like imagine the layered architecture, you just put layers in between that have the same input and output. You can put as many of those in between, it will not change the format anyhow. Express has an option to do that out of the box, and I have actually added it here. What I do is I call app.use and app.use, I have already mentioned that, is simply take in whatever HTTP method you send in. It doesn't care, so it can be a get or a post request or something else, and whenever it catches that it just runs a function, and the function has as before request and response, and it has a so-called next parameter. What we do in this case, we log something, we log the method, and we log the URL at the time, so which method is called, on which URL, when, and then we actually set something. We set, and this is a header, we are setting x logged to true. Just in case you wondered, this does not mean anything, it's just additional information in the header that I can then, on my client side, I can read for example. And now comes the key part, we don't send any response here, we don't do rest status 200 or so, we don't send anything back to the client, instead we call next, that's our parameter up here, and that causes express to just continue with the execution, so it does not send anything back, it just continues here and checks, okay, is this a get request, then do something, is it a post request, do something else. And the nice thing here is really we have done, we have added some functionality that has been called, and it does not affect the remaining endpoints, they just behave as before. In our case what we have done is we have added logging functionality, so now our server, and you have seen that in the logs, we are actually logging every single request, so we see exactly which request comes in at what time, and that can be very good if you want to debug or if there's security issues you can check that, so that's a common case for using a middleware. The other thing that is interesting here, maybe we can look at that as well, I have already mentioned we are modifying the header, so if I go into, if I go into any of these requests and I look at the response, no, at the headers, the response header include x logged true, so this is what I have added, it's just additional information, and you can use any you like. The convention here is usually, that's why I use the x, everything that is not standard HTTP, you just put an x in front of you, so for example here you see x minus powered by express, that's essentially advertising that ExpressJS adds to say this application has been implemented using express, so that's the second line here, I just set the header, you could also do other stuff like modifying the request, I can set the method to post, so no matter what kind of request comes in, I always make it a post request, that's not necessarily a good idea, but you can do that, you can mess with the request if you want, and then you call next, and indeed that's exactly what is hidden in here for example, so when we tell express to use the body parser to give us back a nice JSON body, internally the body parser is implemented as a middleware, so it does something, it actually modifies the request, it modifies the body so that it looks, that's formatted properly, and then it calls next and we just continue, so that's exactly what's happening in here. Okay, so that was the last part of this lecture, we have done a whole lot of things, we have implemented an entire, almost entire RESTful API, we have deployed it, we have made it persistent with MongoDB almost to the end, I have not debugged why the connection doesn't work, and we have looked at middleware very briefly, and of course as you already see, that the whole deployment part is really lots of configuration and lots of debugging, why things work locally, but not on that machine, and that's a very common issue you have whenever you essentially distribute applications that you run into this, so this was just a small primer to give you some idea what is waiting when you want to deploy things, and that's it for this lecture, we continue the next lecture with web security, we start with a discussion of attack surfaces, so essentially what is possible, how can attacks happen to a web application, and then we start off discussing authentication and authorization, so far everything we have programmed is just open, everyone can access it, ideally of course we would like to protect certain things, and that's what we look at then, and then in the second web security lecture, lecture 19, we look at vulnerabilities and actual attacks to websites, thank you for today