 So my name is just Nisheli. I work at Doctor on Demand. So what is Doctor on Demand, the rest of the talk will not make a heck of a lot of sense if you don't know what the product actually does. I could explain it to you myself. But shortly after we launched, we got a bit of free publicity that we could never have hoped to have asked for in a million years. We got picked up by the Colbert show. So before I start, I will let him explain exactly what we do. It was like a free advertisement. Oh. The side wasn't working. And you can kind of tell when this was released because of the reference to a not working. No, it wasn't working. Jim, this consultation with a real board certified licensed physician who can diagnose you and prescribe medication. The cancer talk is one of about 1,000 doctors nationwide treating vacations via app. This is a major breakthrough, medical expertise with the convenience of a smartphone. Why waste time getting an exam? We can just shoot your doctor an emoji of your shattered fever. Got a little sunglasses, isn't that cool? You face time with a doctor from your computer, smartphone, or tablet, which is great news for anyone with an iPhone, an Android, or a public library where they can press their junk up against a web fan. I'm not safe from this. This app is going to revolutionize medicine. It's designed to modernize the health care system by using mobile technology to make a simple doctor's visit stress-free for patients and doctors alike. Yes, it's less stressful for everyone. You really think your doctor enjoys cupping your balls? Now he can watch you doing yourself. If you get really good at it, I know a sign that'll pay you $40 to do it on camera. You can even upload photos for them to examine. In fact, the only thing you can do is set a stool sample. Learn that one the hard way. Rune a perfectly good headphone jack. There is the future of modern medicine. And we'll just cut that off right now, because it automatically goes to the next one. On our website, this is what it actually looks like. You go in, put in symptoms, fill out a few basic pieces of information, like you were at a doctor's office, and the system connects you with a doctor in minutes if everything is working correctly, which it normally does. So that clip was actually a little bit scary, because we're on the West Coast. And all of a sudden, we had friends on the East Coast calling us up and being like, hey, did you see we're on the Colbert show, and we're like, what? What happened there? But it turned out to be a really good thing. And it's a lot easier than me trying to explain it, because he does it much better than I ever could. Doctors get paid per call. So lots of stuff with the website is the same, as you might expect. We have all of the same issues that any web app has. You have data that people need to access. Lots of it is static or custom. People, after they have visits, they have charts associated with those visits. We have access controls. We have all of the standard stuff. But when I was looking at DjangoCon, I was like, oh, hey, they're still accepting proposals. One of my coworkers was like, oh, you know that thing that you built that does the matching? You should talk about that. And I was like, oh, well, what's the worst they could do? Just not accept me? No, the worst that they could do is accept me. So there's lots of cool stuff. There are lots of other talks about how to make that stuff work well, and frameworks, and APIs. That is not what I'm talking about today. Not everything is the same. The biggest thing that is different is the fact that we have to keep track of where people are in the call process. So people want to initiate this matching process. And you don't know exactly how long it's going to take, because you don't know exactly which doctor is going to pick up, and you don't know if one of the doctors is asleep at the wheel, and you have to call other doctors to make sure that it connects appropriately. So we also have to deal with the fact that regulations say that I can't just route you to any licensed physician in the US. Oh, no. No. My life would be much simpler if we had one indivisible country that did not involve states. Instead, what I have to deal with is the fact that Dr. A has licenses in these five states. Dr. B has licenses in these seven states. And I can only connect you with a doctor that is licensed in the state that you are currently located in, assuming you have never seen that doctor before. There are other rules if you have seen that doctor before. But if this is your first time, I have to connect you to a doctor that is licensed in your state. There are 52 cues in that case, because there's all of the US states. There's DC. There's international calls that can actually go to any doctor. And there are also some other territories. So there are over 52 different cues that you have to deal with. So one question you might ask is, this sounds like it's been solved before, right? Does this matching people who need help to people who can help them sound at all like an application anyone else has ever dealt with in the past? Maybe customer support? So why not just use an existing customer support system and do the routing that way? There are a number of reasons. Partly why I'm here is if anyone knows something that would mean that I built this system in vain and I don't have to maintain it or make it work better, please tell me. That would be awesome. But assuming that that is not the case, we looked at several different systems and none of them could meet the requirement that we had, where when patients are calling in, it actually has to be a video call for most purposes to satisfy state requirements for a legitimate telemedicine visit where a doctor could prescribe medication if appropriate. Actually, a lot of people ask if we're a pharmacy pill shop. We're not. There are a significant percentage of our calls that don't involve prescriptions. That's a question I get asked all the time. But in this case, it has to be a video call. So we would need something that can support iOS, Android, and web clients in terms of connecting their customer support routing system with the video technology that we use. And there was nothing that was flexible enough to drop in in that manner that I found. There seem to be mostly monolithic solutions for call routing where if you have a customer support system and you have a voice over IP or traditional PBX that they support, then they will do everything for you. They will handle all the scheduling. There are massive amounts of ways that you can configure it. But if you just need something simple and you just want to plug it in, not so easy. So I got the opportunity to look at a bunch of different things. And again, I may not be the unique snowflake that I think. And I would love for someone to tell me that I can just throw all this code out and never have to worry about it again. So without something off the shelf, what did I get to do next? I got the chance to use a lot of technologies that might be common in more Python back-ends development, but that you often see as kind of side notes in Django projects. So the first option that we needed to look at was, well, before I get into the first option of what I considered, the data that you need to keep track of for this is pretty straightforward. You have all of your patients that are calling in and what state they're located in. And then you have all of your doctors that are currently online and available and what their current status is. So to simplify things a bit, the current status is there are more than these in reality, but current statuses can be considered available for a call, currently being paged for a call, or actively in a call, and what states their license in. So that's all the data that you need to keep track of. And first option is to just use long-running HTTP requests. Now this has a number of issues. So one thing that it does work for is a very quick proof of concept. If you just need to get something running and show people how this is supposed to work, so this is actually the first thing that I did. You set the time out to be really, really long on your web server. You allow the request to go on for minutes at a time and assuming that you know that your doctors are generally behaving correctly, which they are in a proof of concept setting because it's all staged, then everything is fine and dandy. It's simple to implement. It's good for prototyping. You just use a database and check what's going on in terms of, so you have a visit object, and that has a status of whether or not it's been connected yet. You can use models for everything, and it's all fine and dandy when you have a few demo patients and a demo doctor. But it does not scale. And worse than that, even if it did, it's flaky and unreliable because if a client loses their connection, they don't necessarily have a way to get that status back. If they did, how would they find out the result? So you end up starting with something more like option two, where if the client's flaky anyway, you do need a way for them to recover from that and see where they are in the current match process and get reconnected to a doctor. So you might as well start having the ability for the patient's client to ask what its current status is and whether you've found a doctor yet. So Ajax polling. And then on the back end, because you don't have this long-running request for the match, what I did is use Celery, which is an asynchronous task queue written in Python. It integrates really well with Django. Another talk would be on all of the configuration options for it. I will not go into that. There are many. It works well out of the box, but there's lots of stuff you can do to tune it. And that fell naturally into this use case because I could have Celery, lots of workers running in the background. It scales really well as we have more patients coming in. They're just able to check the status of the doctors and page a doctor one at a time. We have a built-in business requirement that if the first doctor that's called doesn't answer within a minute, then you start paging another one and another one and another one. And these can just go on for as long as they need to with some arbitrary timeout. And so that's all great. So the pros for that asynchronous task, you don't have to worry about request timeouts. Cons for polling. It's no longer real time in the way that your long-running HTTP request is. So with the long-running request, you just respond as soon as you have a match. And you're great. With polling, to simulate real time, you have to keep making this request to the server over and over and over again. There's got to be a better way. And there is. So you can still use polling as a backup. But the optimization that I went to was using WebSocket, which is an HTML5 technology that allows clients to receive pushes from the server so they open up a channel. It's lightweight. It's perfect for this. And then on our mobile devices, you also add in push notifications. So if the user has enabled push notifications, you can use that to, as soon as a doctor accepts the call, immediately push that information to the clients. And then even if you have polling as a backup, it doesn't have to be as frequent for it to appear real time to the majority of the patients calling it. The cons, it has a few extra moving parts. You have to worry about setting up this extra service. If you don't want to set up your own, there are now SAS providers who do WebSocket for you. I think the biggest one is Pusher, and they'll help you with that. So stage two in making this thing work well. Tracking doctor status. We have a lot of doctors that can be online at any time. They all have licenses in a variety of different states, like I've said before. The first option would just be to have all of this be in the models, and then every time someone makes a call and make the appropriate queries to figure out who's online, who's not, that's slow. It could be out of date. The pros of this approach of just using the database are that there are no external dependencies. And I think that is really the only pro, because it's slow. The data can get stale really fast. You have to make all of these extra reads and writes to your database. It's inefficient. It's not good. Depending on how your models are set up and the way that ours are set up involves quite a few filters and joins to do this just using the database. So I found that Redis worked as a really good solution to this problem. Because I needed the data to be persistent. Did I spell persistent? Right. No, I will fix that later. And that's going to be on video forever. All right. So Redis is really good for persistent data structures. And the data structures that I used are the obvious ones in this case. We have a data structure that has the doctor's ID and their current status. And that's what we update. So it's used as a cache for this ephemeral data of what status the doctor currently has, because we only care about it while they're on shift. It's not something that we need to store long term. And the other thing that is useful in Redis that I found is queues. So the first link there is just the official Redis client. Well, not official, but the biggest Redis client available in Python. The second one is a queue package that I use. So what you can do in this case is instead of storing everything in the database, have queues in Redis of the doctors that are licensed in each state. When they become available, push them into these queues. And then as patients want to see doctors in these states, pop off the next doctor, check their status. And then if they're available, match them. If they're stale, just discard them at that point. And you can keep using queues this way. One disadvantage of this. So if you start with just a FIFO queue and you have to have a separate queue for every state, you can end up with duplicates, depending on how you do it. Where the doctors that have licenses in more states will get richer and see more patients because they'll end up in more queues more frequently. So one of the optimizations you can do instead is to use a priority queue, which won't have duplicates the way that a FIFO queue would if you just have IDs. And it's kind of depending to the end. And it has a score. So whoever's seen patients the least recently can get the next patient available in that state. The last thing that I came across that I thought was interesting in this process of developing this architecture was atomic call acceptance in February. The last half of February, we decided to do this wonderful thing where we made all calls free for two weeks. That was painful for me. Everything worked. So it was actually really great. But one corner case I discovered at that time was that it was possible to have multiple doctors accept a call simultaneously. I thought I had protected against it using pipelines, but I hadn't done it correctly. So I actually switched to something that was a lot easier to read. And that was just to use a simple lock. And this is just kind of a way that you can do distributed locks and resets in their official documentation. What you do is you set a unique ID. That second part that's the value is it doesn't matter what that is if you're just using it as a lock. If you're doing something else with it, this is the standard key value store. And then the two options are that it expires after a certain amount of time. And then you only set this if it's not already set. So what this does is say I set call attempt 2. So this patient is calling in. Their call is number 2. And both doctors try and accept at the same time. I mark call 2. I set the value as some random value. And only one of them, only the one that gets the lock, gets to accept the call. So you can't end up with two doctors accepting the call at the same time. That's just a neat trick if you're using Redis already and you come across something that would use this. So in the end, we have an improved system where the database does what it's good at. We have celery taking care of asynchronous processes that can end whenever they need to. Specialization improves our efficiency. We can, at some point in the future, offload all of the celery tasks to other servers. It clusters well. It's very scalable. It's nice. And Redis keeps track of our ephemeral data and lightweight persistent cues. And that is the end of my talk. So there's now some time for questions. Thank you for listening to me. This was the first talk I've done at DjangoCon. And I'm really glad to be here.