 So first up, we have Prabhupada Sadalke. He is somebody who I work with in my organization, somebody very knowledgeable about agile practices, but specifically around how you apply them to databases as well. He's written a book a few years ago called Refactoring Databases, something a lot of us refer to when we are puzzled about how do we actually write database-oriented code, which actually can be maintainable as well. So he's going to talk about practices for agile database development. Over to you, Pramod. So what does the agile manifesto says? Like one of the things it says, individual interactions are more important than processes and tools themselves. It also talks about working software or comprehensive documentation. And it also talks about customer collaboration over contact negotiation. The most important from a database perspective is responding to change over following a plan. So in the waterfall method, we are used to following a plan. You get requirements, someone sets and creates the schema for you, gives it to someone who creates the actual DDLs, and then you start developing. Because here, stuff is going to change under with you that waterfall way of designing the schema up front is not going to work at all. Because even if you design, stuff is going to change. Requirements are going to change, your schema is going to change. So designing up front will become a waste. Because maybe of the 100 tables you design, 50 will not be used at all. Then the time you put in designing those 50 is waste. Or is it wasted? So we'll talk a lot about this and how to cater this from a database perspective. So the obvious question is why agile? So the manifesto says that great, but why do I do agile? One of the things is businesses are changing concept. Like 20 years back, businesses didn't change that fast. They did change, but the horizon of change was longer, like three years, five years. Now businesses change every two months, like some radical business change every month. Some of them change in six months. But that's a good horizon to look at. Like stuff is going to change in six months. So even if you went by a waterfall way of doing things, that will change in six months. So my philosophy has been change is going to be there, no matter what you do. So why not get better at managing change itself? So if I'm better at managing change itself, then I can deliver stuff fast. There is also this whole idea of design is not a predictive phase anymore. Like there's no phase which says this is the design phase. And you don't design after that ever. You design constantly. And it's emerging design requirements change. Sometimes let's say if it's a startup in the initial, you just want to put the product out. So you do whatever, like it's a PHP hack, who cares? I have a website up running, people are buying. If it goes hit, it goes viral, a lot of people are doing. Then you sit back and say, now how do I handle scale? How do I do this? Urefactor, rearchitect, whatever, all right? So similarly, that's where it's a continuous evolving activity, right? Making decisions before they are actually needed is kind of ways, right? So the same, let's say the same startup example, in the beginning you say, hey, I'm gonna hit like million users or 10 million users a day. I'm gonna have this cluster of awesome hundred machines, blah, blah, blah. You put the website online and nobody sees your website. So that's what we are saying is, if you have the requirement to handle a million users, at that time sit down and do that. If you don't have that and you can't really prove it, then it's basically a waste. So a lot of this also comes from the Lean startup book where they talk about how you can get to that spot when you need to and you don't have to actually design for it before it, right? So application requirements, as you go, they are gonna change. So when the application requirements are changed, your persistence layer will have to change. That means your database will have to change, right? So this, we are talking strictly relational database systems. When we actually go into the no SQL space, that's a whole different paradigm you have to deal with. So I'm not gonna talk about that no SQL as such. But if you visit my blog, I talk about no SQL also as a separate matter, how it affects your schema design or migrations and stuff like that. But that's a separate topic, not addressed in this particular topic, okay? And what decisions need to be made upfront, right? So even in an agile way, you are making some decisions upfront. They are not far-reaching decisions. They are not decisions that you basically write and say thou shall not change ever, right? That doesn't happen. But you do take some decisions and you look at that decision on ongoing basis and change as it goes, okay? And dealing with emerging design. Design is gonna change on you. How do you deal with that change in design, right? So to do all of that, you need some good practices in place, right? Because rapid change in schema means a lot of dependent systems, dependent applications, dependent code, dependent stored prox, dependent triggers. They're gonna get affected. You need good practices to make sure that your working software is always working because that's one of the thing that the manifesto says, you, your software should always be working. Delivering working software is more important than code or documentation, right? So without those good practices, that is gonna be difficult, right? So what are those practices, right? So these are some of the practices I'm gonna show and like from here on, you will actually be getting to do it, okay? So the pairing, let's start with pairing, for example, most of the companies are siloed in the sense like there's a DBA team and there's a development team and when the development team needs something, right? What do we do? We send an email or we say, I have a meeting with the DBA team and say this is what we want and then they argue why you want it, how you want it. No, that column should be that and after all the argument, you get a schema back, right? And then the schema, you need a change in that schema or you need some other stuff. You have that meeting again, this goes on forever, right? So one of the things you need to do when you are going agile is to you need to break down the silos, right? It doesn't mean that you should dispense the whole DBA team. They could still have one DBA team, right? That could be taking care of operational stuff like maintaining production or doing events if something happens, backups and all that kind of stuff. But you need a representative from the DBA team working with you on your team, right? So you don't have that go to a meeting phase at all. Just bring the portion to your team. Sometimes it happens that is not possible, right? So other alternative for that is maybe have that portion come to your team once a day for one hour or two hours, whatever works for your team, right? So that will allow you to break down the silos. Most of the time sometimes the DBAs will let you do the schema design themselves or yourself, but have this schema review phase after iteration or before deployment or something like that. And usually they don't like the schema you design at all, right? They'll come back and say, okay, change this, this naming convention is not right. I would put a index here. Have you thought about this query and all that stuff? And that is coming too late in the game for you before deployment. You can't react to that change. Then you say, oh, forget about it. We'll deploy this and we'll worry about it in the next release. Next release, everything's deployed as stuff is going happy. Nobody cares about that stuff. When release two is about to be deployed, you come back to the same meeting, schema design, review meeting, and same stuff happens again. So you are carrying a lot of technical debt from a database perspective, but you don't have that feedback loop from the DBA team frequent enough, often enough or early enough to react to that, to cater to that, reducing the technical debt, right? So pairing with the DBA team actually gives you continuous review of the schema as it is designed and reduces the technical debt on the database side, right? A lot of times developers are known to basically get everything from the database and loop in the code, right? Or all other things that happen. This is a good way for the DBA team to give the feedback saying how this will affect the performance of the application, performance of code, add all other things. In the early phase, right? That's the key. Like knowing it on the last day before it's going live, it doesn't really help anyone at all, right? So knowing that early is important, right? And the third thing or the last thing is put the database code, the application code, or whatever that the application needs in the same repository. Don't have your code in one place and the database stuff is script or something in some DBA's machine or some the Oracle database somewhere that is backed up and that is your version control. When you do that, you cannot link the version of the application with the version of the database, right? So let's say you want to go back in like, I want to go back as of release two because there was some bug reported against release two which is in production at some customer X. If you want to recreate that environment, the only way to do that is take the version control copy of the release in production and ask for a backup of the database as of that date so that you can get the schema for that particular release. That creates a mismatch and creates a lot of ways. Somebody has to go fetch those tapes, somebody has to go reinstall that database somewhere and then you can start talking to it, right? That's one of the problems. The other problem says, as databases are deployed constantly in the agile environment, you are bound to have times when you want to deploy frequently, right? QA wants builds often. UAT wants a little slower. Production maybe is a little slower than that. So all these three phases are at different phases of deployment or different builds as they want. So as you go deploying these, some DBA is sitting somewhere and doing a diff of two scheme as I was saying, what needs to be deployed there? He'll take the diff, deploy it and let's say the QA tomorrow comes, say that bug had that build had a bug, give me a different build. They have to do that diff again because something changed, right? So that is all manual work that somebody has to sit somewhere and do, right? But as the changes are happening, the changes are happening on the developer side. So why not just take the change itself instead of putting it in a bucket and then diffing the buckets, right? So that's where the database stuff will happen. Configuration management is, this is where like, if you put your database code as well as your application code in the same place, it's all in the same place. So the DBAs can actually check out or update their version every minute or every day and do a review of what got checked in, what SQL got checked in, what indexes got checked in and all of that stuff can be reviewed continuously. If they change, if someone changes and if someone, some other person changes and they don't like the change, they can look at history and see what changed, who changed, when it was it changed, all that auditability is there, right? And this is what the DBAs like too. In a sense, they like that auditability of who changed, why it was changed, what was the story that it was changed for, right? All that is good. So that allows, configuration management allows you to do that if you have code and database in the same place. All database artifacts like DDL or data for some tables, like for example, what is the credit card type? Like Visa, MasterCard, there's a list usually put in the database somewhere or like what kind of cities like somebody, like if you're doing an airline website, they'll be a list of airports they fly to, right? All that stuff should be belonging to a source control repository, right? That should be including setup and config data. And similarly, when CI publishes your artifact, we are used to publishing the jar or the Warfile or the DLL, whatever we publish, we are used to publishing that, but we are never used to publishing the database, right? Because some DBA will take a copy and propagate along. Those are two different timelines it's moving against and coordinating those timelines will be very hard, right? So if you publish the database as part of your CI build, then you can go to the CI artifact page and download the database as well as the code that matches with the database in one go, yeah? You will get to do it yourself, okay? So when we get to the CI demo, you'll get to see it, right? So as you publish, let's say you publish a jar as part of CI, you compile, you run your unit test, integration test, everything's good. You publish the jar, along with that, publish all the database, like the DLL, everything. So for this particular application version or build number 152, this is the database that matches with that build 152, right? So if someone downloads that and recreates the database based on the DLL, the application will roll perfectly against that. No issues of column missing, table missing, view missing, something like that, right? So that's what I mean. And when you're deploying to production, yeah? Yes, whatever changed, yes, yeah. So when you're deploying to production or to QA or UAT or whatever or the other developer who is not current yet, you're basically only deploying checked into it. That means from your application side as well as your database side, right? So if someone creates an index directly on the database, if it doesn't make it to the next environment, it's his fault because he didn't check it in, right? So don't have this phase where somebody creates stuff on the database. So then somebody has to remember to move it, right? So don't have that phase, just have everything in subversion or whatever else you are using, right? So let's try this, right? So if you have a machine fired up, if you have subversion going, just basically do a SVN checkout of that particular repository, right? So SVN 10.012 is the IP address of the machine. As LDB as SVN repository name, we are checking out trunk and that's just a folder name you're checking it into. And if you don't have a machine, I'll show it to you what's happening here, right? So that's where I'm checking it out, right? So checkout. So I basically got all the code, all the database that went with it, right? So when you checked out, you saw a bunch of stuff came down and most of it was Java code. There was also a lot of SQL in there, a lot of jars and all that other stuff, right? So let's go see what's inside. That's where we checked out, right? So you see the DB folder coming in and has a bunch of other stuff, with some SQL, some migration scripts. If you have a data model or whatever, all the data model is checked into, some test data is there, some triggers and all that kind of stuff. At the same time, there's also like all the source code, all the unit test, all of them, right? All of them are the same repositories. So if you change a database, that will get checked in at the same place. If you change code, that talks to the database, you're checking, it's at the same place. If you change database code and application code at the same time and check in, it's again in the same revision, right? So when you check out that particular revision, you are getting all the data, all the database and all the code that goes with it at the same time, right? Does it make sense? I mean, nobody likes losing the data, right? That's the first rule of database is if you lose your customer's data, they'll be pissed at you, right? So that's the first rule of databases is don't lose customer's data, right? So that's what we did. We saw what came down. Now, when it came down to you, you could change whatever you wanted within that environment and not commit and not affect anyone, right? So that's one very powerful incentive you have to experiment. Previously, if you had a common Dev database, you want to experiment in it? You screw up something, you have like a wrong trigger, the trigger doesn't compile, or you put a wrong not not not somewhere, everybody will be shouting at you. What did you do to the database? And then the next step that happens is the database says all changes will go through me, right? Because he doesn't want that stuff happening again. And that basically kills your innovation layer totally, right? Because now you can't innovate, you can't experiment, you can't try different things and innovate, right? That basically goes away. So this will give you that power, right? So any change you make here is local to you. Do whatever you want, like you can change scripts, try different stuff and see what happens. If you are not happy, you just revert your back to normal, right? So the next obvious thing is, okay, I got my own stuff. Now I need my own database to play with it, or else just writing scripts in a file doesn't really help me, right? So that's where you basically give everyone a database sandbox, right? So a database sandbox from, like let's say from a Oracle perspective would be just a schema in a database that is running common to everyone. In a MySQL sense, you could just run it on your machine if you wanted to. In a SQL server sense, it could be a database that's running on a common instance, or you could run SQL Server Express. You could even run Oracle Express on your local box multiple different ways to make that happen. But a database sandbox is basically a database that your application can run against successfully, right? And only you are connected to it, or your instance of the application is connected to it. Right? Yes, so anywhere, like whenever you are running unit test or when you are testing as a developer testing the application instance, you are running your own instance, right? It could be web logic, or it could be a firing up jetty, or putting the war in the Tomcat in your local instance. You are testing it for yourself, but connecting to a common database, right? I'm saying, run your own application instance, connect your own database. Right? It could be that, or it could be like in the same database instance, I could have different schema, one for you, one for him, one for him, whatever. I'll show you a good diagram of that. Yes, it will be multiple copies. Yeah. Okay? So this basically what it does is it reduces waste and waiting time. Like let's say some new user joins the team. If you have this common Dev database setup, he'll basically connect to that, just start using. But then you run into the problem of not being able to innovate, making mistakes and screwing up everyone's time. Because think of it like by mistake, if I drop a table, everybody who's using that common Dev database has lost that table. So productivity goes down the drain, right? So if you have your own sandbox, you make mistakes, you're affecting only yourself, not anybody else, right? So that's where you're reducing waste as well as waiting time. You can automate every database task that is usually done by a DBA, right? The thinking that he needs to is in design and performance tuning, not in creating schemas. Creating schemas is automation script, not a thinking task, right? So anything that is automatable, automate the heck out of it, right? And you will see a bunch of those. You can allow local changes and let people experiment with what they want to do, right? It improves productivity because if I break my stuff, it's only me or my pair that is affected, not the rest of the team is not affected at all, right? So that's where it improves productivity. And what this does is when someone requests for a new environment, like QA, like there are some types of QAs that only do daily, they can take a new build daily. There are some QAs which are very interesting in life cycle of a product. Like let's say you're doing insurance products, then like, okay, what happens after a month? Does a new inverse get generated? Do I get alerts? That kind of QA needs long-term build stability. So you can't deploy frequently to those environments, right? Or you can't change the database frequently. So you will need multiple environments. So when you ask in a traditional DBA shop, like I need three environments, they'll say, why? I can't use one QA because setting up environment is very hard. If you go through this automated way, setting up new environments is very easy actually because you have automated so much, you can literally give one environment per QA and say do whatever, right? Yes, so because you are assuming I get app changes through Subversion and database comes through some other layer. You saw the previous step, the app and the database are the same place, right? So when it changes the app and the database, you update, you get the app and the database at the same time in your Subversion repository. Now obviously you have to apply that change to your database, there's automation step, I'll show you that also, right? So spinning up new environments become very easy because you can apply those changes, right? So this is where you are, don't be there, basically be here, right? So this is what we're saying, like the pair uses their own schema in a common database, right? A lot of times most of the shops are concerned about licensing like Oracle Power Developer Machine. So they say, oh, let's use a common, you can also use Oracle Express if you're not using those advanced features. If you are using the advanced feature, use a common database and then inside that use schema per pair or schema per developer, schema per and one. So within this schema, you will have your own customers, you will have your own list of addresses, you will have your own list of tables, you will have everything will be your own copy. So if I drop this schema off, nobody will be affected. If the question relates to size of the data, we'll come there, okay? So this is what helps in this pair experimenting with whatever they want in that schema and not affecting anyone, right? And when they arrive at like a good design of the object or good design of the table or good design, whatever they make the change, they create a script, they check it in and then Jim, Jack or whatever, whenever they update, right? This is not forced upon them. Think of it this way. If Andy and Joe make a change here, everybody is forced to take the change. There is no if then else there, right? Here Andy and Joe make a change. If they don't want the change, they don't have to update, right? That's a choice the rest of them have, right? And these guys will only check in when they are comfortable with the change, they check in, the CI build passes if it's green, then everybody else says, okay, I can take that change and update and get the change, right? When they get the change on their local machine here through subversion, right? They can just run the scripts against the database and the database will be current for them, right? So that's how it works. It doesn't affect the others. Now, obviously you are thinking now everybody's gonna make a change, how am I going to track? Who made what change, where it is at? So there are a bunch of tools out there, like DB deploy, that's what we are gonna see in this particular scenario, but tools like liquid base, DB maintain, ibat is, the rails migration, fly away. These are all tools that actually track changes against your database and will apply them if the change has not already been applied to the database, only the change, right? So there's no question of losing your existing data or rebuilding the database, right? You'll just apply the change, just apply the delta. You are talking about the technique of baseline. I have written that out in the book. So there's a book called recipes for continuous database integration. That technique is known as baselining. So once you release something, you don't want that deltas as a list. You just want one script consolidated, that is release one script. So you can do a consolidation like reverse script back from the database and that becomes your baseline for the next release, right? So as you go releases one, two, three, four, you continuously keep baselining as you release and whatever you did as new development is only in deltas because the rest is all baselined out, right? So let's try this, right? So if you have checked out your stuff, there's a buildout properties.template file in the checked out code, right? So let's go see. So that's the file. So what I really want you to change is the username, the password and there's a host name there. That's the host name of the database, right? So make sure you get there and change the username and password. So I'm gonna copy that buildout properties template as a buildout properties file. I'm hoping you guys understand what ant is and how it works and all that stuff. So I'm just gonna say and dbcreate, right? There I get created a schema form, right? It's a blank schema, but it created a brand new schema for me, right? So no sending email to DBA, no, nothing like that, right? You can pair with the DBA to create this ant target, but once it's created, the DBA is not spending any more time. You're actually saving time for the DBA, right? So obviously you're thinking, oh, great, I got an empty schema. What do I do with it, right? I'm running Oracle inside of Windows VM here. So let's connect to the database I just created and you will see there's nothing in the schema, right? I did a select star from user tables. There's nothing in the schema because it's an empty schema, right? The next thing I'm gonna run is ant dbinit, right? So it did a bunch of stuff, doesn't really matter, we'll get back to that, but I got whatever I wanted, right? So bunch of steps happened here. I created an empty schema and then I ran a tool called as dbdeploy against my schema, right? So bunch of things it does, like this is just ant target programming. I have multiple, one it did was a dbclean, right? I don't know why it's only showing up, right? So it did a dbclean. In the dbclean, I am basically running a clean script. It will basically clean out my schema, right? If there were tables, it will drop all of it. Then it will do a dbinit, which is basically saying my working schema or working user fame is Bangalore, right? And it runs a bunch of stuff against it and it is saying that currently there is nothing applied to the database. I have scripts one to nine available to be applied is one to nine because nothing was applied before and it will basically execute all of that against and basically combine them and execute them against my schema and it will give all the changes to my database or applied to my database, right? So if you look here, these were the nine changes we had inside here, right? They are numbered. So dbdeploy is a tool that lets you write scripts which are numerically ascending, right? It doesn't really have to be sequential. They just have to be numerically ascending. So as you write a new script, you basically take the next number or the next higher number that you desire, put that name in the file, right? So for example, here we put 009 vehicle, right? Put that number there and put whatever you want inside that script file, right? Whatever script it is and check it in, all right? And then dbdeploy will figure out that this change was not applied to the database and I'll apply that one change, right? So that's how you create a sandbox. So obviously now questions such as what if there are multiple peoples doing different changes? They'll just create files with different numbers. If by mistake, like let's say he has checked out and he's seeing nine and you have checked out too, you are seeing nine, nobody has committed. So both of you create a 10, right? And check it in. One of you will fail because dbdeploy will say there are two files of the same change number and the build will fail for one of you guys. It's like standard development stuff, it happens. When you conflict in the subversion, conflict has to be resolved by someone, right? That's how you make changes to your database. We will later see what you write in the file when we actually come to scripting the change itself, right? So that's what we did. We did it in the India folder. We copied the template file to the properties file. One classic mistake people do is actually check in the build.properties file. They'll do that because if you check in with your username and password in there, somebody will check out and just run and db in it and it will clear your database underneath you, right? So just check in the template file, don't check in the build.properties file, right? So that's a good technique to safeguard against accidental, right? So we change the username, password, we put that IP address in there and we did the db create and db in it, right? So that gave us the schema that our application wants, right? So on to the next one, right? So this is, how many of you know what VDD stands for? Yeah, right? So you can apply the same techniques of behavior-driven development against database too because the table has a behavior like it will raise exception when duplicate stuff is put in, it will raise exception when not null stuff is put in, it will default your columns to something when something is provided as null on insert. These are all behaviors you are asking of the table when you create the DDL, right? So you can actually write tests against this. I have a paper on methods and tools which tells you how to describe all this stuff, right? That you can look at. But you can go to my website, the links there. So let's see if this is our requirements, right? Like store vehicle with a unique win number, model year 2005 and above because we don't want to deal with model year 2004 and below, we don't sell them, whatever, right? Model year cannot be null, name cannot be null, make cannot be null and miles are not above. Like I don't sell cars that are more than 10,000 miles already driven, for example, right? That's just made up requirements. So if that is your requirement, you could write behavior tests in your unit testing framework like this, like should not say duplicate win, blah, blah, blah. Those are your behavior tests that you're writing on the persistence layer, right? You can even write it in like the database, unit testing layer if you wanted to. But since it's always good to test the integration layer from the people who are actually using the integration layer, right? So persistence layer is used by your domain object. So test it from that on. So you can actually exercise the persistence layer itself, right? So if that is our test, right? We can actually create the table like this that will make those tests pass, right? Now if after sometime someone comes in and makes that null, right? For example, the test will fail because your design assumption on the table is changing now. So you have two options, right? One is was the change by mistake or was the change on purpose because that's what the new requirements are. If that is new requirements, then you basically go back and change your test, right? If that was not the requirement of just a mistake, you are stopped from doing mistakes before they actually hit any further beyond then your own schema, right? So that's like outside in testing theory that behavior-driven development uses. You apply the same to the database so that if you have dependent systems talking to the database, right? So especially that's what happens. People use the database as an integration database for multiple applications. If you have that kind of thing going on then doing this kind of testing really helps you to change the database in a safe manner, right? We could try this. I'm gonna skip this for time purposes but I have the vehicle that I just showed defined in that 009 vehicle.sql. So if you have checked out code, look at that file and then actually you can do all this stuff, right? So on to the next topic here which is schema design, right? So in this iterative fashion or iterative way of development, stories are coming at you at different rate because this iteration some stories may come and next iteration some stories may come that may relate to the same table, maybe not, maybe yes, whatever, right? So let's assume we have this set of stories and let's assume this group is one iteration just for simplicity sake, right? So in the first iteration you got this story it says look up a car by a win and see the accidents reported against the car. If that's the first thing that came to you then you just know about that much because you don't know what's coming here. Even if you knew you don't know if that will be played or not because the customer might decide oh, that's not important, let's take it out. Then whatever you designed for this is wasted at that time, right? So given this story list, let's say how the schema shapes up, right? So the story list is there in the background, right? So this is how I would start, like create a car table, put a sequence on it and put a sequence constraint to the owner. You would think where is the owner requirement here while creating the owner, right? So this is where you mesh agent techniques and best practices at the same time. The car always has a owner and as a person who has designed any decent RDBMS schemas before you know there will be multiple owners on the car you just create a owner table with a foreign key constraint even if it's a one to one in the beginning, right? So that's where like having the DB on the team helps, he'll help out with this best practices at the same time, right? So I put a owner table and a sequence on it, right? And then there's this accident. So I put an accident thing in there and I am doing the FK constraint to the car table, right? And then I want that event to be unique. So I put a unique index on it. That's a new change that I did to the database, right? The change. Then basically everything that is coming on the slide new is basically a change that is applied to the existing schema, right? So at this point of time, the schema looks like this. Whatever new changes come in is a change to the schema applied to a script, right? So I say I apply the registration type because I'm seeing car and truck here. So I say, okay, let's put a registration type there that will say car and a truck on the same table, right? And then I say, oh, I want purchase date and stuff applied. Then somebody thinks, oh, the car, the table is named car but actually we are storing vehicles there. So let's rename the car table to vehicle, right? So this kind of series of changes keep happening. I have a vehicle type now because I introduced the vehicle. I need to know it's a car or a truck. So I introduce a vehicle type. This is where you will have data going into the database, right? Because a list of vehicle types like car, truck, commercial truck, rickshaw, all this is basically fixed data that you put in the database, right? So that's where I basically put insert statements inside the change script itself, right? Then I basically remove the column because a car can actually have multiple owners over its lifetime. So I basically have a owner going on there. So this is how schema design happens, right? This is not ad hoc design. People are actually putting thought into the design as requirements are coming to you, right? So this is not like, a lot of people have this confusion like I will never design the schema because this is ad hoc. It's not. As thing is coming to you, you are refactoring the databases to match to your new requirement. Similar to what you do on the code side, right? So a bunch of stuff is gonna happen here. Like I have event type, registration, and then registration table because car could be registered to multiple people, all that kind of stuff. The key thing to look here is the database schema is developed as a series of changes, right? It's not done one big schema DDL. It's basically a series of changes that are dictated by changing requirements, obviously, right? Nobody is just doing it just for the sake of it. But the key thing is those series of changes are basically migrations that you apply to the database, right? So that brings to the next topic of refactoring, right? Those series of changes are basically refactoring so you're applying to the database to prove its design so that you can take it. Like way back, Martin wrote this, a refactoring is a change you do your code to improve its design, improve its semantics or whatever, right? So you can take this same definition and apply to database refactoring by saying a database change actually changes the DDL or the data, right? Sometimes you change the DDL that affects data, but sometimes you just change data also, right? But the interesting angle on the database side, it has two dimensions. Like code refactoring just has one dimension. You change code, it compiles, it all works fine. You just deploy it. It doesn't have state with it. A database carries state with it. You can't just rename the customer table to client and say, oh, I'm done, right? Because there is code talking to the database and that code still thinks you have the customer table, not the client table. And then when you make the code talk to the database, it will say, oh, where is the customer table? And it will say the table not found and that kind of stuff, right? So that's where it's important to think about behavior as well as informational semantics of the database and its interaction with your application, right? So let's see how that affects, right? Because this is where your database is, your enterprise, right? And there are a lot of other things that are talking to your database. If you are in a decent size database enterprise that has been in production for a decent number of years, people will start writing stuff against database. Like somebody will want data out of the database. Somebody will want to push new customers through a batch feed. Somebody will like to do, like, oh, let me serve this data as a service. All kinds of stuff start happening against your database. And that is where you're working. So if you change the database, potentially all of this other stuff can get affected, right? So how do you keep the database alive or working for other people, but at the same time, make your change go through, right? That's the key behind this, right? So that's where the techniques of refactoring come in, right? And let's talk about a timeline, for example. Like, let's say you wanted to make a change. You basically start making the change. You deploy the change, right? But at the same time, you have this transition phase in the middle where, let's say, take the example of client and customer. You keep the customer table alive. At the same time, you have a client, right? So obviously now you're thinking there are two tables. How do I synchronize that kind of stuff, right? You can also implement the same change by renaming the customer table to be client and introducing a view called customer, which is a one-on-one view on top of the client table, right? So now people who are looking for customer will find it through the view. People who are going against the client will find it through the client table, right? So this is the encapsulate table with view refactoring applied to rename a table, right? Now all of your old stuff will keep working because there is the customer table, right? At least via facade over the client table. The new stuff will keep working because there is a client table, right? Not really. Like splitting a column, view will not work, right? So in the transition phase, you will have that view as well as the table in the system. And after some time, you will take out the view because everybody else has migrated to using the client table, right? So that's when the transition phase ends and you basically, your refactoring is complete. This will apply only if you are in a complex environment where multiple people are talking to your database and you have to keep multiple versions of the application alive and all that kind of stuff, right? Now there may be scenarios where your application is the only application talking to your database and there's only one version of that application ever. Then you don't need this. You just refactor, you're done because your new application knows about the new schema and you don't really care about the old stuff, right? So when I talk about refactoring, keep this diagram in mind as we go through this, right? So obviously there are types of refactoring, structural refactoring, so when you're changing splitting columns, tables, doing all kinds of stuff. There's data quality, like refactoring, triggers to do CI, like RI implementations, or you are actually making the format of the phone number enforced via some check constraints, all of that falls under data quality pattern. In architectural, you are basically making big decisions like introducing a read-only table via batch job, for example, is a architectural refactoring, or like making some of your stored procedures, you are extracting it out and putting it in the application for whatever reason, that's architectural refactoring, right? A method refactoring is all of refactoring that Martin talked about in his book, apply it to your database code, basically, right? So for example, introduce parameter, or rename variable, all of those will apply to your database code too, because the database code needs to look better, needs to be maintainable, needs to have proper separation of concerns, all of those apply to your database code too, because that is still code that you have to make, right? Store procedures, triggers, right? And transformations are basically things you use as part of a refactoring, right? So let's say, for example, move column is a refactoring, you actually put a column in a different table, remove the column from the previous table, and the data is moved using update statement. So the update statement here is a transformation you use as part of, right? Okay, so let's look at some refactoring here, merge columns, that's what I got. I want to be here, like basically country and phone, that's all I care about. So right now I have country, area code and phone. So that's what I will do, like I'll introduce a phone number, and a phone area code and local are still around, and after some time I will take out the phone area code and phone local, and there's a new number going on there, right? So that's the transition phase. You can implement this by having a before update trigger on the database, like if it's Oracle, if it's some other database, some other way of mechanism of doing that, but when the old applications are inserting data here, you are dynamically updating that. When the new one is inserting data here, you're dynamically updating that. So old and new are surviving at the same time, right? We can make a column nullable. Purposefully I didn't put the transition phase here just to make the diagram bigger, I guess. So there's customer, and I want to make the first name not null, so that's my resulting schema after that, right? I can add a foreign key constraint by the same token. I can have introduce read-only table. This is architectural refactoring, as I say. Like for example, I want to search customers going through each table or whatever, going through a view is performance-wise is low or doesn't work. So I may decide to put like a materialized view here or a table that is dynamically updated to a batch job like hourly or whatever, right? That's what I would do as a read-only thing, yeah. To be very frank, I wouldn't do most of this stuff in the database like business functionality. There is some functionality that triggers help you with, like for example, transposing columns here and there, because that's just a transient thing that I would do, right? But to have stored procs or not is a religious discussion, I think, and I don't want to get into that right now, okay? Database code, like I was saying, like all of your stored procs triggers should also be like clean-decoded, clean have proper variable names and all that kind of stuff, right? So let's try this, right? So let's implement the rename column database refactoring and that's applied to the vehicle.name and applied to the vehicle. I basically want to change the vehicle.name to vehicle.model, right? And we'll see how that goes. So if you have code checked out, I have this intl.j going on that will basically let me, so I'm gonna just make this name change to model, right? I'm doing this on purpose just so that you know how dependencies will get affected, right? So the DB init will work fine because the table got created, that's fine. So the database got created, no problem, right? So obviously the next question would be will the code work, right, the code didn't work because the code is looking to write to the name column and I don't have the name column anymore, right? So this is how dependencies can also be enforced by having a proper set of tests going against your database via the domain layer, right? You can test the database just through unit testing on the PNSQL side. You can use utpnsql to unit test your database layer, but as long as your domain layer doesn't talk to your database, as part of a test, you run this risk of your domain layer being out of sync with your database layer, right? So that's why it's always good to have at least a set of tests, call them integration tests that talk to your database from your code, right? Sure, I mean, that's why we call it integration test. You can create a separate suite out of it, unit test that you are using mock objects can run first if they pass, then you can run the integration test. If you have not changed the database at all, then you can just run the unit test check-in because I didn't change the database at all. Then you should be fine. If you change the test, change the database, make sure you run the integration test before you check-in. On the CI build, make sure you have this pipeline concept going where the first it checks the unit test. If that passes, then it runs the integration build. If that passes, then you can do other stuff, right? So there are techniques to do. Like, if you just Google for build pipelines, you will get all that information, right? So now, obviously, my build is breaking because I changed the vehicle table to be have. So I'm gonna just go here, make the change. Let's see what if that works, right? That's still gonna fail. Are you seeing the pattern of how dependencies are being cascaded, right? I changed just the database from name to model and now that forced my test to fail. I changed the hybrid mapping. Now that's making my domain change and that will make my test change, right? So this is what I'm saying is you can make this cascading changes. This is all local to you. You didn't affect anyone. Neither did I affect if anybody else is using their own database. I didn't affect them in any way, right? And let's say I was not happy with this at all. I could just go here and revert, right? So now I'm back to normal. It doesn't matter what changes I had, right? So this is what it gives you locally as well as gives you the dependency built in. You don't have to really use, like, some people will say, oh, let me see if I change this column. How will it affect the database? So a lot of DBAs will show you like a right click show dependency and it will show you nice graph of what tables, what stored products will affect it. But it will not show you how your code will get affected because it's only in the database, right? So we changed the vehicle table. We run and test and then fixed in the build passes. We didn't fix. We reverted just for time purposes. But you could just keep going till the domain layer got changed, till the unit test got changed and everything was passing again. And then at that point of time, we can check in. So when you check in, you checked in the SQL file. You checked in the Java files you changed. You checked in the hybrid mapping file you changed, all as one unit of work in the revision control system. So when you update, you get all of that as one change. And whoever has that updated that change can run the same targets. And their database will get updated and their code will get updated and all of that will be one unit, right? So obviously now I'm making changes and I can't just make random changes to existing files and hope that people will pick up change, right? Because developers don't really care losing about their data. They can rebuild their database. But when you deploy to QA and forward, they care about their data. Because QAs take a lot of effort to set up their database, right? Like a proper customer set, like a customer who is like, if it's an insurance product, a customer who is 60 years old, a customer who is three years old, a customer who is a male or female, they'll spend a lot of time creating that kind of data. And their policies are in different states of the life cycle and all that kind of stuff. And if you drop that data by mistake, they will really be not happy, right? So obviously each change has then to be written as a migration itself, right? So make sure it can handle data migration also as you change the media. Like let's say I rename the column and I rename the column, then data gets moved on its own. But if I move a column from a table to a different column, I put the column in the new table, make sure the data moves along with it and it's a complete object. So the object can be recreated without trouble, right? So migration scripts are development time activity, not a deployment time activity. So that's why I was talking in the beginning, like make sure you're not diffing databases all the time because people will want different versions in different databases all the time. So you can't be in the diffing game at all because you'll be diffing forever. The other thing it also does is who's gonna test the diff, right? And once you write the migration scripts, you can package the migration scripts as part of CI build and create it for automated deployment so that there is no manual intervention at all, right? And use the same scripts for developers. If I make a change, I write a script, I check in, the developers checks out or updates, runs the migration scripts against the database, it works fine. You use the same to apply against QA, UAT and production. So there are specialized cases where you would want to consolidate just for performance reasons or like, there's a DBA team sitting in a different country at client's side, you can't send the migration scripts, you want to send them package scripts, all that kind of cases can be handled when you're going to production, right? So an example of change would be like this, like for example, I made the change directly in the create vehicle table. You could actually write this alter table vehicle and add model and basically drop name and that kind of stuff, right? If there is no transition period, you would just drop it, right? A better way of writing the same migration scripts is this way, right? So you write the forward going change, you can even write the backward coming change, right? Sometimes you deploy in production and there's a major bug or something like that, you want to roll back that particular change, you can write the undo part also, right? So forward going is from nine to 10, backward coming is from 10 to nine, right? So if you deploy this change, you want to undo that change, this is what you would do, right? Lot of tools I talked about in the beginning like DB deploy, IBATIS and if you're using rails migrations, you can do forward and backward at the same time and this is what tells DB deploy that whatever is following here is basically the backward migration, right? So let's try this, right? So let's implement a rename column database refactoring as a change scripts. Previously we just wrote it directly in the file, in the SQL file. Now we create a new file called rename name to model. Now somebody was asking if I can just update. So looks like the rename script I wrote is not right, right? Does that make sense? So I wrote a wrong script, I ran it, the end target itself told me whatever I told or whatever I wrote, I had a syntax error in it, so I corrected that, re-ran it. Now DB deploy is looking at my database, right? Currently it's saying one to 10 are to be applied, right? So that's what it basically applies the diff and then it goes for it, right? So this is how you write one particular change, right? So this, I'm just renaming the column so no need to move data around or whatever I write. Now obviously I still have to change code because the code is still referring to name and I changed the column name to be model, right? Does it make sense? So I still need to make that change but for time sake we'll not get there right now. So we did the rename as a change script so we just added 10 data rename that was the file name that we did, right? And we changed, if there was any data to be moved around we would have written update statement, right? In the same file, if there are multiple tables involved we would have moved columns around multiple tables, return whatever migration scripts was needed to make the schema change as well as keep the data intact semantically, right? And run DB in it to make sure that the database change works and then run and test to make sure the application works against the new state of the database, right? So that would, we would basically keep going till the build passes locally. Once the build passes locally you would check in, right? When you check in people get that 10 as a change on their updates and that will update their own database, right? So obviously this begs the question like how do I do continuous integration with this, right? We are testing the application code and the database in one place because the same application domain layer is talking to the same database. We are generating code and database artifacts, right? And we are gonna integrate the application and database changes in an independent environment, right? In the share the common developer database we have this model of we all work against the same database schema and when we check in the CI build we'll also work against the same scheme, right? So because databases are not going through. That's a wrong place to be in because you don't know what changed, right? So that's why that independent environment is important and we'll show the current state of application and database to everyone, right? So when they click on the build junk in space or whatever else you are using it will show them what happened to the database as well as the application, right? So this is hopefully you're using CI maybe not database CI but at least CI this is your state currently, right? So you have this locally you're working here you have this dev database the DBA team makes the changes in the dev database, right? And then you have this dev database it's the same database here that the CI engine is talking to, right? And the database the DBA team is in control of that database also then the DBA team is also deploying to your QA your client as UAT and production systems, right? Because they made the change and that kind of stuff, right? So what you want to be as the next step when you want to do CI with databases is don't do that, right? You write this migration scripts at developer time, right? Check in those migration scripts in source control, right? It's a version of whatever you're using the same migration scripts are picked up by the continuous integration engine and applied to a different database called the integration database, right? This could literally be a different schema in the same common database but it's a separate schema for the CI engine or the CI that you're using, whatever you're using, right? The same migration scripts are applied to that integration engine if build is good then you basically publish, right? You publish the jar but along with that you publish the migration scripts, right? So that is your artifacts. Now when it comes time to deploy this against your database either QA or CT or production you basically apply the same migration scripts to your CT, QA or production environments, right? And you take the jar or war from here and apply to the environment that the database is deployed. Interesting thing happens is from here on they're all linked together. There is no question of what application version works with what database version, right? If it's a successful build that means the database was working for that application, you publish that there is no questions about will it work or not work, right? It's actually linked right here and passes through all this testing and then when it is published you have this linked artifact. So the DBS will not have to go to the database to make a diff at all, right? And since the diff was actually done here came through all this testing you have the confidence on the script that it will run good in production, right? Or whatever you want, yeah. Yes, you can't update a stored product. You just have to recreate them, right? Be it's, if you look at the statement there is no alter stored product command itself in Oracle. That's a create or replace stored product but no alter stored product, right? Because that's just code you can package and say apply all of this stuff, right? So that's the CI cycle. We can definitely try this, right? In the local build let's go back and do the same rename column for the sake of simplicity. I'm just gonna take a so that I don't have to change all code. Yeah, yeah. So especially if you have like large sets of data that you're dealing with like what I have done personally on projects where we are dealing with database sizes of above a terabyte or so what we did was the migrations went through the normal CI build. After the CI build pass we took the migrations again and applied it to a copy of the production database and saw how they performed. Every migration went through that and if it was slow or like timeouts or whatever we made sure that we performance optimized the migration. Every change went through that route because we wanted to make sure that while deployment we don't come up with surprises, right? So like sometimes you write update or something like that and that will update like a million or 10 million rows. You run up into this rollback space issues and all that kind of stuff that happens in a database. Then what you do is run it against that and see if it fails. If it fails, then you come back with come back and say, okay, can I write a PLSQL script instead of a straight update and batch it up like 1000 rows at a time or whatever. So you do that kind of techniques, right? So we have this change running now. So if you look at we got a new file let's just add it to the subversion, right? New change got checked in. So I'm running Jenkins. I don't know how many of you know Jenkins but that's one of the CI tools in the market. So I am running a project called evolutionary database here and hopefully the build will pick up on its own or else we can force it in a minute or so on. So while it's doing that, I will show you this other stuff that has to all across the world. The latest build I am generating the application jar which is basically a compilation of all the Java files that I got and there's a dbupgrade.zip that I'm generating which is a zip of all the changes that I've done to the database so far, all of them, right? Not just the last change, all of them because when you go to deploy you don't know what version that particular database is at. So you want everything, right? So it could be like production maybe at version three, right? But the current version in development maybe version 10, right? So you don't know what version the database that you are going to upgrade is at. So you send everything and dbDeploy will pick what needs to be applied and apply them. And pick up so that's what's happening there. The build is good, right? So we got a database and that has a dbUpgrade. So let's go back to our presentation and we'll see what to do with that dbUpgrade zip when we get there, right? So we did, we just created a new table instead of the rename column because I didn't want to waste time in changing all the code, right? So we watched the build at that particular URL and our build was good. So we took, we saw that it generated artifact and a database artifact also, right? So obviously the next ultimate question is like I made all these changes, it has to go to production or else it's pretty useless anyway, right? So we do the database migration or upgrade just like what developers will do in their own sandbox, right? So we'll use the same task because it's a development time task, not a deployment task. The reason for this is as stories are being developed, right? Developers are making the change. They have the context of what is changing from what to what, right? So instead of doing that on a common schema and after three months, the dba going and doing a diff at the, after that three months time, they don't really have the context of what changed and what was the reason that changed, right? So they have to go ask, hey, why did this happen? Why this column move here? How did it move? Did you split it? And they have to understand all of that and then code the diff, right? Instead of that, when the diff is actually being implemented, if you code the migration right there and then you don't have that knowledge loss, first of all. So you are not stacking up the changes and then giving it to someone. Again, like lean comes in as the change happens, code the change right there instead of batching it up for later, right? So that's why it's a development time task, not a deployment task. Package all the migration scripts during CI build and basically publish them off, right? And apply this migration scripts to the environments, right? So this is a good copy, like this guy is working on the latest version. These guys are all still on 48. This developer here is fixing some bug from some previous version, so he's a little behind. QAs are different version, like this QA is at 48, this QA is at 47, UAT one of them is at 44, the other is at 45 and this guy is production is at 43, right? So these are your change numbers, file-wise, like we had that 10, right? So these are your change numbers and migration scripts basically keep going from your left to your right, right? So that's how we deploy. And every database knows what version it is at because DBDeploy maintains a table inside your database that tells DBDeploy what all changes have been applied to your database, right? So that's what DBDeploy maintains in every schema it works against, right? So anybody could be at any version doesn't really matter. If you have all the change files, you can upgrade them to whatever version you want, right? So here it says I am till version 10, right? So especially if you have like a large client base, like one of the projects I worked on, we were doing dental office software and there are like 4,000 clients they had. We can't really know what all versions they are at, right? So we just publish everything and they upgrade to whatever version they want. It's all inside the install process itself. The doctors don't even know that this stuff is happening in the back. You could think of it in terms of iterations or releases too like this is iteration, this is your week numbers or whatever or build numbers. And when you go live here, you basically that's what is in production, but you're still keep writing changes as they go. And when you're time to go live again, you basically take that line and deploy it again, right? So the change numbers from here to here is what you package and send off, right? So there's no new step in there which says create the disk script. This can work off-shore, on-shore too, like works like a charm, right? Because everything is being zipped through here, not through a database that's off-shore, one on-shore and then you're trying to consolidate changes. Because whatever is coming from here, that's what. Because when you check in on the on-shore side, it's going to the same subversion. When you check in on the remote, it's going to the same subversion. So if you have two teams working on-shore off-shore, they work in their daytime checking everything. You come in your daytime and when you update subversion code, you get everything, you just run and dv in it again, your database gets upgraded automatically and off you go to work, right? The other thing someone said they deploy like weekly, bi-weekly, that's awesome place to be. But a lot of people don't deploy that often, like six months, one year, that's too long a cycle. So I try to portray that image as a deployment metaphor or a boat metaphor, right? So if there's a boat going from one shore to the other shore and the boat is going, let's say, only once in six months, right? A lot of people will want to be on that boat because if they miss that boat, the next boat is in six more months, right? So a lot of people will be clamoring for the development team to put their change in, fix my bug, give me my feature. They will be really clamoring for that, right? And then you'll be forced to do a lot more and then you're batching up six months of work, deployment will be high risk because there will be a lot of changes, right? Instead of that, if you went to a model where you're deploying weekly or even daily, right? Then what your change you did in that one day is that's always deployed and most likely it's just one script or two scripts. That is very low risk, right? It will run very fast, it's very low risk. You know exactly what you're deploying, two lines, something goes wrong, rolling back those two lines of change is also very easy, right? So you can do all of that work in five minutes and go home back to FM instead of sitting through the night or the whole weekend, right? So that's where deploying frequently helps because you're reducing risk drastically instead of accumulating a bunch of work and then deploying it in one go, creating more risk, okay? So let's try this deployment part, right? So let's download the latest build. We'll unzip and run, right? So go back here, we'll take this DB upgrade zip. We'll download it into the downloads folder, right? So in the downloads folder, we have a DB upgrade zip. I'm gonna unzip that, right? So I got, maybe I'll show it here, right? I got all the changes, right? All the way from one to 10. I have a build.xml published along with it, the DB deploy jar and the jar that talks to the Oracle database and I have three properties files here, right? Each property file is actually talking to a different database, right? So let's look at what is in one of the properties files. So if you look at the QA properties, I have the user name is QA password is whatever is all different. It's talking to a machine. So we need to change that to match with our IP address over there. And so this is basically a QA environment database properties file. Can this something? Take out the at, that's what you're saying. Okay, so we changed that. Let's go back to our presentation here, right? So we did, we took the latest build, we unzip and we are just gonna run that command, right? We add minus property five, is it equal to, right? Right, obviously it's not running because the QA database is not there. I did not create the QA database before this. I can even just do, right? That created the database. Now I can just say upgrade, right? All of that got applied. That's my QA database got created. Obviously when the next change comes in, I can just say upgrade and it'll just apply that one 10 change. So let's go and do that so that you guys see exactly how that happens. So let's apply 11, call it product or something like that and I'll just change this to product. So that's a commit, CI should pick it up. So once CI picks it up, it will publish everything again, right? So it will have all the way from one to 11, not just 11, right? Because at deployment, you don't need to worry about what changed and code what is changed on the fly because it doesn't get tested. It doesn't go through your CI cycle. It's this last minute movement, last minute coding that people do. That's very high risk. Instead, if you do the development of the change itself during the early phases, it gets tested through CI cycle, it goes through QA, UAT normally and then at the end, you have high confidence of what it was. So that ends the presentation. So if you go through those links, there are other papers that I have written you can look at. If you go to that link that you will see this whole presentation as well as the code I saw. So you can go back and download and play with it and all that kind of stuff. If you break your production database, don't blame me because of whatever reasons, right? So don't point it to a production database. That's all I'm trying to say. So yeah, if you follow me, that's Twitter, that's my website where I blog. The databaserefactoring.com is where I have all the patterns that I've talked about in the database refactoring book are on there. So you can just look at diagrams and figure out what it is known. And this database refactoring book is actually available on Flipkart. I checked this morning. So it's 372 bucks or rupees, I think. That's the other book I was mentioning about recipes for continuous database integration. So if you have any questions, fire away. They are mostly unit tests on the model layer but you can also use JBehave if you are interested in that. But it's any testing framework that you are comfortable with. Exercise the model so that it persists to the database. Yes, writing business logic in stored products itself is a big mess. So because you can test it properly, you don't have tools that will tell you coverage. You don't have tools that will do like static analysis of the code. You don't have tools that will do like online refactoring. I know VSTS for database professionals does some refactoring, but still you run in this problem of like not finding what the code is exactly doing, right? So one way of doing that is to write the tests for the stored products using the code that was used to write the stored product itself. Like I know SQL Server, I bet you are using SQL Server. In SQL Server, there's a unit testing framework for TSQL itself. So you can use that to unit test the stored products if you want. Yes, it is. Yes, yes. So that's where you can apply Martin's refactoring on top of your stored products, right? Try to separate them out. If one stored product is doing too many things, try to separate them into multiple stored products and then call them from like a controlling stored product or something like that, then you can test the individual stored products through your unit testing framework, which you are comfortable with, like any unit or something like that, right? So that's why like, if you are doing database refactoring or code refactoring in the database, Martin Fowler's book still applies, right? Even if it's database-based refactoring like stored products, because it's still code that you want to be properly designed, properly maintainable, and all those other things, because that language is not Turing complete. There are no so many tools that you cannot really apply against those, right? Because static analysis, dynamic analysis, or branch coverage, and none of that stuff can be applied against that, unless the database vendors themselves say, okay, I am gonna provide this tool and come up with it. If rollback is a big concern to you, right? Then make sure like all the refactoring that destroy existing data, actually keep a copy of the existing data somewhere. It gets really complicated if you want to do it over a large database, and also really care about rollback, right? So for example, if you are splitting a table, right? So when you split and the system goes live, one hours of transactions come in, you wanna rollback now, like what do I do, right? That becomes a very expensive thing you have to do, which is like keep track of new transactions, keep track of old transactions, keep track of new data, keep track of old data, all of that you have to actually program it, right? In some like really 24-7 availability kind of projects, what we have done is we used to continuously upgrade, continuously downgrade, and continuously run unit tests and all that kind of tests against our code base as well as the data. Like it did that, someone checked in, it upgraded, ran the test, it downgraded, ran the test, and made sure that all of that was still available. So you can put it in a loop and do that. Thank you very much, guys.