 Everyone welcome back. I hope you enjoyed your lunch. I know I did Next up we have Ben Shaw joining us all the way from Auckland who is here to talk about development Thank you Yeah, hi everyone My name is Ben. I'm an amateur Python developer and this is middle-out Python development Just some background or stuff about the talk to start with You should have some experience with Python development This is sort of an intermediate level talk, but hopefully even senior developers will be able to take something away from this In all my examples, I'm using Django but just due to the nature of what I'm talking about this will apply to other frameworks as well as Developers We're often told to do it fast and that usually means that we end up doing everything at once What do I mean by this? Well, let's say we're starting a new Django project and Our process might be something like define our models write our views turn on admin build some templates put in the test data and then see if it all works and It's only when we get up to this stage here that we actually start testing what we end up with is a Cludge of stuff with our html controllers Datas templates views and models and then when it comes time to test we test here or here or here or somewhere here What we really should be doing is decomposing our application into separate components and then Just test at the boundaries boundaries between them. So that's here and here So that when it comes time to bring everything together it all fits nicely And at any time we can swap out a component with another one and we know everything's going to fit together to demonstrate this I'm going to need an example application and I've decided to make a cat database because Internet What's a cat database? Well It stores cats and breeds Each breed has a name and each cat has a name and a breed Now if there's any solution architects in the audience, that's the URL diagram for it and Just to illustrate a bit further This is what it looks like on the main menu. We can choose to look at cats or breeds if I look at the If I click on cats I get a list of cats because there's nothing there at the moment We just have an option to add a new one If we're going to that I get a form for adding the cat's name and selecting its breed and Then when I save it I see the cat and the in the list and I can do the same thing with the breeds as well So it's a just a simple Django crud application So I'll start off by looking at the bad way of doing it when I say the bad way I might also refer to this as the old way, but I just mean the traditional Django way of using everything That Django provides you so this is a code for the view to just Look at or edit a breed and it's just basic Django stuff We get the object the breed out of the RM We pass it through to a model form and then save it if it's valid and We pass that form through to render and just render the template How would I go about testing this? Well, I probably mock my breed dot objects dot get Probably mock the breed form probably mock render and then just assert that all of these have been called correctly It's a little bit tricky to mock this. Well, we're mocking it itself was easy But how do I know what parameters I should be passing to get to get the right data out and Am I even sure that the data source is returning the right thing? Should I write a test to test the data source? Is that overkill since it's just the standard Django data store? Maybe using the standard Django RM itself is overkill So I have a lot of assumptions and questions And it all sort of leads to me asking what other Implications does locking myself into the Django RM mean? Have I committed myself to Django when something simpler would do? Have I committed myself to a database when something simpler would do? Basically, have I committed myself to blank when blank would be better? So looking better back at the sort of architecture what I want to do is start developing here Which is why I call this middle out development So first I'll talk about testing building and testing this boundary here And this is all about providing the data to the view So I need to just make something that provides me the data I got to just write this quickly Quickly for testing but it should have a consistent interface And it needs to just be of good enough quality to do whatever I need to do at the stage For the first take I'm just going to do it really simply and use a Python dictionary in memory to store these the objects So I've written a data facade class which does this This isn't all the code but I'll just highlight some of these important ones Firstly I can put a cat into here I first set the object the ID of the cat I then also put the cat's breed and Then I store the cat into my storage dictionary and return its ID The set object ID just checks if my object already has an ID and if it doesn't It gets the maximum data maximum ID out of the data store and adds one to it So this is basically just an auto incrementing key Then when I want to get the cat back out I just provide the ID and we get it out of the dictionary and There's similar methods for Getting and putting the breed and getting a list of cats or a list of breeds So this was quick to write it's very simple and it's in the memory so it's fast But unfortunately it's volatile so every time Django restarts we lose it to test this it's basically just Storing stuff, so we just need to test that our storage behaves as we expect i.e. stuff goes in and stuff comes out The interface is whatever we've defined And we just need to test it as appropriate So because I know that I'm going to be writing some more data facades The way I've decided to test this is just to build a base Test class that implements all my test methods This is just an example of some of them there I'm testing that I can save and retrieve a breed I'm testing that when I save a cat it also saves its breed I'm testing I can get a list of all the breeds and I'm testing if I save something multiple times It will override and not keep creating new objects and of course. I have similar tests for saving saving retrieving a cat and getting a list of cats So to look at that first method the save and retrieve breed method. It's fairly simple We just create an expected Breed that we want we put it into our facade We pull it back out again using the ID that was provided and we check that what comes out is what we put on Now to implement this test. I'm subclassing my test base and just writing a setup method and This is the entire code for this particular memory facade test So what I'm doing is instantiating a facade for the test to use And then looking back at the implementation I'm using the facade here Which is the one provided by the subclass So I've got a test for that How do I use it in my view? Well, this is a different view. This is one for getting the list of cats but all I'm doing is instantiating the facade and then calling my get all cats method which I've previously tested and Rendering that template using the standard Django renderer So this is good, but looking back in the pros and cons It's volatile. It's lost in every Django restart That means if you're running the debug server every time you save a file, you're gonna lose everything so it's not Quite not that good if you want to have a consistent view that you come into and you you see a list of things that you expect to Be there So I'll improve it slightly with a take 1.5 which is still using an in-memory database but I'll populate it with some test data and That's simple. I just subclass my memory facade and in the clear method. I'm inserting a bunch of breeds and cats and Then when I look at my view I get my pre-populated list of cats there So how do I use this new? Data facade in my view Well, the view code doesn't change. I'm just changing What in the background I'm changing what facade is getting loaded So again, this was quick to write simple in-memory fast and also it's preceded with test data So every time we go in we can see the data that's in there That it's instantiated with But again volatile every time I save a file. I'm gonna lose everything So we should try to take care of this So the next implementation, I'm just gonna be storing this On disk instead of in-memory So it's slightly different, but it has the same interface Now if I want to get an object, I call this load object method and pass it the generated object path If I want to put an object into it again, it sets the object ID and then writes it to some generated object path And if I want to get all the breeds to get a list I'm listing the directory containing whatever my objects are and just mapping that to my getBread method So it gets called for every item in that directory And the store object and load object are very simple They're just using pickle to either dump to disk or load from the disk Respectively now to use this in my view again view code hasn't changed at all So how do we change out the facade that we're using? Well in my case, I've decided just to do it really simply Just in the settings pie. I'm importing all my facades and commenting out the ones that I'm not using But it's up to you how you want to do this You could use an environment variable to switch so that perhaps in your CI smoke test You're just using a pre-seted memory facade to get a clear consistent look of some data maybe as you move through your Delivery pipeline you change to using real a real data store So this on-disk storage again was quick to write simple and it's persistent between Django restarts The cons are that it's on the disk so it's slower And how do I test this? Well, just to remind you of my memory facade test look like This is the disk facade test So the only thing that's changed is I'm now loading her a disk facade instead of a memory facade all the test Methods are still the same as they were before so this is the power of developing in this way You can have a consistent interface change everything and you don't have to change your tests So I've looked at a view which already has been Which is already using the facade and we can see as we change the facade back ends out. We don't have to change the view But how do you convert from the old way and to this style? So this is the new The new single breed view and I'll go through and just compare it To the old way so the old way when we're getting the the breed out. We're just using the Django RM in the new way we create a facade and call our facades get breed method in The old way we were using a model form so we can pass our breed instance straight into the model form and In the new way we have to Create some it's not a model form anymore. It's a standard Django form So we're creating some initial data which we're then sending through in the old way to save data We just check if the form is valid and then save it The new ways are a little bit longer. We need to instantiate the breed object From the form data and then put it into the facade So that's developing and testing the sort of back end. What about testing this boundary here? Well, we've got a standardized interface so we can mock it and we can test it our view Causal the methods that we expect on our data facade. So we're just mocking the facades in The setup method I'm just patching my facade loader patching Django's render method and patching the breed the form Django form So that I have mocks for all of those and Into test it. This is just one of the tests I have I'm testing That I'm getting a list of breeds. I have an expected context which I'm Sending through to the render method and I'm just asserting that the render method has been called with that context So this context should be built with the list of objects That come out of my facade from the get all breeds method And I assert that it goes through to my render function Testing a getting a single breed is kind of similar We again haven't expected initial value which is going through to our form We have an expected context which is going through to render and then I'm just asserting that I'm calling the form with the right data and Calling render with the with the right context which contains that form So the expected initial values that I have and the name should come out of the data facades Get breed return value and The context the form should be whatever was rendered with that breed form method Testing getting a new breed. So this is one that doesn't exist yet It's fairly similar. I've just changed the Expected initial value that I should be getting so that the name Should now be blank if we don't actually have breed But again the context is just containing that form. So what about testing here? This is Sort of the layer between what you're presenting. This is your html that you're presenting to the browser So we can test this in the standard way like with selenium But now if you want to you can switch out the back end facades to get known data to render So as I mentioned maybe in your CI system, you you want a consistent view in one In one of your dev environments so that you know That your renderings all working properly And if you wanted to you can switch out your template generator i.e. Django and you already have tests in place at both ends to help you know that you've done that correctly So what if we want to write another back end? Well first we need to ask Does the on-disk storage actually fulfill all our needs and it might do So why not put that into production if it works? On the other hand it might not in which case we have to code another data source Now this is simple because we already have an interface that we have to code to and we've got tests for it So all we need to do now is just go through and fill in all those blanks So let's say we wanted to store in a sequel database We change our get single breed method to do select star from breed where whatever Let's say we wanted to store in a Mongo database We change our view to do collections dot breeds dot find whatever Let's say we wanted to use Neo4j Some crazy Neo4j syntax What about when cheeseburger DB comes out maybe we just write I can has breed So Let's ask why we might actually want why we might actually need a database Well, one reason might be searching because with our memory and on-disk Facades, we don't really have an easy way just to search for objects so when we want to Write a search method add a search method to our real facade i.e. the database facade We'll just implement a we'll write a new method called breed name search and implement it with our select star from breed We're named like whatever We need to also add that same interface to all our mock facades so that They all work in the same way But you don't have to do it properly For example, you could just pass in the name and say if the name is whatever we're looking for return some set data Otherwise return an empty list You could even if you wanted to write a method that would go through your dictionary or look through all your files To find out that you're looking for It's up to you as to what you actually want to provide for whatever level of testing that you're at So another advantage of doing it like this is that it's easy to create libraries from your code Because everything's already separate So let's say that another team in our organization wants to access our data Well, yeah, your library is already separate. So it's trivial to provide that without giving your whole presentation layer What if another organization wants to access our data? Well now it's trivial to instead build a REST API using a lightweight framework on top of your library Just on the subject of deployment you should treat your library as a separate package So it will have its own version and you can refer to that version and your main applications requirements.txt and Then just host your library on dev file whatever you use So there are some downsides to the style of development a lot of Django things won't work so because you're not using the standard Django models your admin interface won't work model forms don't work and You have to well your migrations don't work It may be possible to code an intermediary layer that will take data out of a facade and translate it to a Django model in which case that would solve the first three things there But I haven't I haven't tried to do that, but it could be possible You have to manage everything yourself. So as I mentioned you've got to do all your own database migrations, but There are third-party libraries That already do that And you have to manage database connection sessions in your web layer yourself And there's of course some overhead to it As we saw The view methods are a little bit longer. You're writing mock facades for stuff and of course it makes you test which is good, but it takes time And so for small projects It might not be worth it So just to conclude This talk was somewhat inspired by uncle Bob's clean architecture and design talks There's heaps of them on youtube and he goes into the theory behind this in a lot more detail So really good to watch If for some reason you wanted to make your own cat database you can fork that on get hub see the full code there And what can I say, but try it you might like it Thanks. All right. Do we have any questions? Django has Model managers for models So you've got a model data thing data thing object dot call to the gives you a model manager method That could be a nice place to Those could be nice things to chuck behind the data facade Yeah So yeah, so you could use your data facade as a front to the Django RM If you wanted, yeah And then you get everything for free that Django gives you Yeah, and it's tidy it's hidden away. Yeah, good idea so I don't know if this is on Okay, I don't know if maybe you just touched on that. I'm quite sure but I I tried something like this in the past. So I had my it wasn't a class It was a module But anyway, so it was like my database library my access to everything because I thought one day I might have to switch out the back end who knows, right? Yeah, so And then I tried to abstract it out and I found myself Transcribing whatever the it was in the query set that Django gives me back in the orm to some dictionary That got really tedious and then you end up just passing the query set out And then you realize that query sets allow you to build more queries on there And if I don't want my other parts my viewcode or whatnot to be Too tied down to the orm to the Django orm I can't run Do this nice thing that the query set allows me to do and build more queries there So I need to go back to my database access little layer library And it becomes this this huge pile of random queries Which I just need now some sort of abstracted away into this library And it gets from all these different parts of my application has these these random queries Which is basically encoded in there Do you have like a little function call that does that query for you? And you just end up having this this knowledge leak from everywhere into this one place So what is the what is the strategy around that? Write your facades better Yeah, have a try to have a consistent interface Decide what the data that you're actually getting out is maybe your facade needs to do some transformations as well so It sounds like You have a bunch of of different queries that you're that are all kind of slightly Different and they should be their own methods. It's hard to say Yeah, it's just it's more about being able to get the data Provided to you in a useful way and the the point of this was about not Coupling yourself to anything so having control so I guess Uh, yeah, if you're putting a Django or a rim into stuff, you're not really decoupling yourself From that you're still tied to it in some way and as I said, this is not going to work in every situation But um, it's just useful to have in your toolbox and have that way of thinking as well Alrighty, thank you very much. Um, and thank you to Ben. Cheers