 Thank you. So, with apologies to people who are at the Te Pūdaha Mātatini whānau day on Monday, the first three minutes, five minutes will be all that you've seen before. So, that picture will look familiar for one. But for most people who weren't there, let me go over this. So, this is an emergency department or an A&E department in New Zealand would be called. We have patients who are in beds and doctors and nurses that are treating them. We have a waiting room where people come in. They're triaged, which means they're assigned a priority and they wait until a bed gets free. Usually that's the bottleneck is the bed. Once one's free another one goes in and again they're chosen by priority with the most severe patients being seen first. And then some fraction will actually leave without being seen because the wait was so long. So, the original motivation for this work that we were doing, it started quite a while ago when I was at Washington University in St. Louis and one of the physicians at the hospital there who worked in the emergency department came to see me and said, we've got this problem of boarding where patients are in beds, they're being seen, we know what's wrong with them, they need to go to the main hospital but there are no beds in the hospital. So, instead they're occupying beds in the emergency department and what you might imagine is if they're occupying beds in the emergency department then they're backing up the whole system. So, what this physician wanted was a queuing model that would predict the effect of boarding on the patients that were leaving without being seen. So, and because this was the US that would then show lost revenue and then they could actually use that lost revenue number to persuade the hospital administration to do something about getting patients out of the emergency department and into the hospital faster, more efficiently, more beds in the hospital. So, we did that, we built a queuing model based on some fear number of approximations in there as you might imagine because we've got priority queues here and we have people leaving without being seen. But what it actually made us realise is we don't actually understand the behaviour of patients who leave without being seen. So, most of this talk is actually going to be talking about understanding the drivers behind patients leaving without being seen so that we can actually have a good model of this abandonment, this leaving without being seen and with a good model of that piece. If we have a good model of the other pieces then we have a much better model for our decision making in the emergency department. And so, you know, this, one of the reasons this is a big problem particularly in the US is there's increase in emergency department visits because if you don't have health insurance the emergency department has to see you whereas doctors don't. And a decrease in emergency department capacity because some hospitals can't even afford to keep them open because of all these patients that don't have insurance to the US. So, there's some high crisis and here's some lovely pictures of waiting rooms in the US where you can now start to understand why people are leaving without being seen. So, do we know enough about patients' behaviour? I would say we don't and that's what one piece that we're trying to answer in this work. So, up to 15% of patients in some hospitals leave without being seen. In our data it was 7% to 8% of patients are leaving without being seen so very significant numbers. And of course, huge health concerns, you know, basically they probably come back, they probably were sick and as I mentioned in the US they care about revenue. So, related literature to this work, you know, we are related to the literature on abandonment. There has been a little bit of empirical work and so most of what I'm presenting today is actually going to be empirical work and also in the emergency medicine literature there's been some work. So, as I just said, what I want to present is some empirical studies that we've been doing just trying to understand that leaving without being seen behaviour. So, we want to identify, you know, the drivers of leaving without being seen. That leads to some managerial insights for how we can actually manage our emergency departments. We want to take a look at the hazard rate of leaving without being seen to lead to sort of a discussion in terms of how would we actually model it and then broadening it out again if we want to model the ED. What do we actually do? So, you know, because this is an empirical work we're going to come up with some hypotheses. Most of these are not going to make you fall off your chair. So, the first one which should be blindingly obvious is that we would expect the leaving without being seen probability to increase in waiting time. And one thing I want to point out with some of these statistics is why are people leaving? Well, some of them have waited so long that they got better. But there's actually quite a lot of patients that leave because they feel too ill to wait, right? And those are the ones you're really worrying about. They just feel so uncomfortable, so ill that they actually leave and maybe they can go find another doctor somewhere else but maybe they just try again tomorrow in that their health outcomes are going to be significantly negatively affected by that leaving, okay? So, yes, we would expect leave without being seen probability to increase in waiting time. We would also expect it to increase in ED crowding. So, that's how crowded the waiting room is. Part of that comes to psychology of queuing basically the more anxiety I have, the less tolerable my weight, and also we have seen some work in the medical literature that shows this as well. The last of what I'm calling a baseline hypothesis in other words what anyone who's familiar with queuing systems would actually hypothesise is that we would expect the probability to decrease in observed surface rate, right? So, you're sitting there and you're watching all these people conserved, it's going to make you less likely to leave than if you were just sitting there and nobody seems to be going into the waiting rooms. So, one thing for the people who are really paying attention, you might say, well, if I know waiting time and I know service rate, don't I know queue length, little's law, yes, except that because we've got a priority queuing system, what we're going to be looking at in terms of queue length, crowding, it's going to be the total number what we're looking at is waiting within a class and so, you know, we did do some tests and we don't have to worry too much about colonialities. So, here's our hypothesis diagram. We leave without being seen. Depends on wait time, crowding and service rate, but more interestingly we also want to know are there some interaction effects? Is there an interaction between waiting time and crowding? Is there an interaction between waiting time and service rate? And in particular, we would hypothesise that indeed this should be, so waiting time should depend on the observed service rate and in particular we would expect that if you're observing lower service rates you'd be more likely to leave for the same waiting time and for crowding the same thing we would expect that if you've waited the same length of time those who see a more crowded waiting room are probably more likely to leave. So, how are we going to examine these? As I said, this is an empirical piece of work so we have all this data from the emergency department. We actually have more data than we're going to use in this study. We have the moment that the person walked in, the moment they were triage, the moment they got a bed, the length of time they were in a bed, the number of tests that were done on them. But for this, basically, we'll actually just use when they arrived, when they departed, we'll do some control variables on their demographics and then their acuity. That's basically their priority for service. This was an urban adult-only USMRED. So there's what we're going to control on. Gender, ethnicity, acuity, day of the week and hour of the day because there's obviously big non-stationaries in any emergency department. And most of our patients are acuity classes C and D. E are the people who really shouldn't be there because they just, you know, they didn't need to go to an emergency department because they, you know, whatever thing. But they probably don't have health insurance. They don't have a doctor to go see so they went to the emergency department. They're your resuscitation patients so they won't leave without being seen. So they're not included in this. That's very good. Yeah, the mean is biased. I think we've got more women than... No, I think we've got more men than women in this. Yes. What you will notice is that 68% are African-American because it's an urban US emergency department. So that would also explain why we see more men. We've got more of the gunshot wounds, et cetera. It's being the US. So what are our operational variables that we're going to look at? We're looking at the leave without being seen. Did they or not? Their waiting time, the crowding which we're going to measure upon arrival and the service rate which we're going to measure over their waiting time. Now, in our data set, for about half our patients we actually know when they left without being seen, which is great. So basically in this emergency department there was a nurse sitting close enough to the front door that about half the patients were kind enough to say they were leaving when they left. Half of them didn't. So half of them, we know how long they would have waited because we know when they were called. So what happens is the patient is called, they're not there. They're called again after a little while and a third time and if they're still not there after three calls they're assumed to be gone. Well, the first time stamp of when they're called we know they've left before then, right? And not only that, for that data we actually know how long they would have waited had they stayed, which is going to be useful. So what we have to do for our patients when we know they live, we have to estimate how long they would have waited had they stayed. So yes, that's our, as I say, about half of them we know when they left and about half of them we don't know when they left but we know how long they would have waited. So yes, so for the ones that left and didn't get called we have to estimate how long they would have waited. We actually tried two different ways. They basically give the same result. It's not that important. Why we need it is because we're going to feed in waiting time into our probate model. So we do a probate model to try and estimate how this probability of leaving without being seen depends on waiting time, crowding and service rate. Into the same queue assuming that they're actually the same acuity, the next person of that acuity. You maybe could have actually, that's an excellent way to think about it. Why that wouldn't work actually. Okay, good. Thank you. And then we also, as I mentioned, we want to test for these interaction effects. So here's our probate model for testing for the interaction effects. And these are just all our controls buried in there. So here's the one that doesn't test for interaction effects and everything is significant. Good thing, in fact highly significant. So indeed, as you would expect, waiting time, you leave more likely to leave, crowding, you're more likely to leave, high service rate, you're less likely to leave. So what sort of effects are these? So remember the probability of leaving without being seen was 7. something percent. That goes up to 11 percent if you're a type E patient and the waiting time increases by one standard deviation. So those are absolute how much that 7 percent changes by. It goes up by 0.8, not 0.8 of 7 percent. So then in one standard deviation increase in ED crowding, you can have a very significant expect. You've got about 15 percent of the people leaving without being seen again for type E. And service rate, you'll see those changes are less significant as you might, not significant, less magnitude as you might expect. Crowding is how many people are in the waiting room when you arrive and it's both your own priority and everybody else. So it's total number of people in the waiting room when you arrive. So here's our model with interaction effects. Again, we see significance, but more importantly, let's actually look at the marginal effects. So what we can see is as service rate goes, gets faster, the leave without being seen, sorry, your probability of leaving without being seen, given waiting times that it's mean, so fixed waiting time that's mean, you're looking at the probability you're leaving without being seen and indeed, as you would expect, it confirms H4 that you're more likely to wait when it's faster, even conditioned on waiting time. Okay. And that sort of flips it around. So now we're looking at service rate at its 10th, 50th and 90th percentiles and looking at waiting time. So you'll see these graphs kind of separate in terms of sensitivity. And over here we're looking at the probability of leaving without being seen. And you'll see that our C and D patients kind of look the same in this case, which is not true in the more interesting case, which is looking at the effect of waiting time at different levels of ED crowding. So if our hypothesis was correct, we would see this as a monotone curve. So we only get partial support. So basically we would see more crowding means you're more likely to leave for the same waiting time. Until some point and then actually you're less likely to leave. So what do we think is going on there? We think what's going on there is it's an expectations thing. So there's a fair bit of psychology literature that talks about how people actually make expectations in terms of weights based on information. So if you show up to a really crowded waiting room, you kind of know you're going to wait a long time anyway. So you're kind of expecting this long wait and therefore because the wait is long, just like you expected, you're less likely to leave. Interesting. And also we see a bit of a difference. Some emergency rooms are more crowded than others. Yes. Yeah. Maybe it's really crowded because there's no alternative hospital within 100 feet. Right. So in this case there are other hospitals. So it's only data from one particular hospital. Yeah, it's one hospital over time. And there are some I mean there are some good hospitals that get very few patients leaving without being seen. That's kind of the idea is if you can actually prevent patients leaving without being seen. So it's really a resource and how variable is your demand. Somewhat the demographics of your patients. What other choices do they have? Can they actually go somewhere else? Did they drive there? That sort of thing. So yes, there's all those things influence the decision to leave without being seen. No, but it doesn't change. It doesn't have dramatic changes. You tend to see these once they're crowded they kind of stay crowded for a while. Yeah. Yeah. Right. Well it may be unless you're lower acuity in which case there'll be ahead of you in the queue and it could get quite complex to think through those. So the other thing is for this case we actually see a difference between type C and D. So D a lower priority than C. And if we see the actual probability, so these graphs are on the same thing. So if we look at the actual probability of leaving without being seen and what are my coloured curves there are different crowding levels. So this is for crowding at its 95th percentile. You'll see that C is much more likely initially but then D has this quite sharp curve as waiting time gets longer. So your C has kind of formed their expectations more than your D which again might speak to what you were just saying D on right. It's sort of you see all these people. Yeah. So and you know it's sort of interesting the different shapes on these depending on so this is crowding is very little. Right. So crowding when there's almost nobody in the room you're not very sensitive not very sensitive until weights get very long and then it's like what are they doing? I'm never going to know whether they must all be out to lunch because nobody's nobody's here and I'm still not getting served right. Yeah. Because they have not this was back to them because they feel too ill to wait. Right. They've come this is hugely crowded crowded waiting room and they just like I can't stand this. You know it's not good. Right. So so what can we what can we pull out of this in terms of managing the emergency department? So initially it increases in waiting time but then as it gets congested the sensitivity decreases which means you need to set expectations. Right. So people kind of can judge but on the other hand there's also you don't really want the visual cues associated with an overcrowded waiting room and so as far as we can see this reduction in crowding is the first order effect that's kind of what you want but the sensitivity to weight sensitivity to crowding I should say is the second order effect that needs to be considered and there's a very interesting research being done right now by someone I know but not someone I'm working with that is actually trying up estimated waiting times in the emergency department and just seeing what effect just predicted waiting times actually have on people's leaving without being seen. Are they more likely to leave less likely to leave you can kind of argue at both ways you want to you obviously want pretty good estimates of those waiting times yes yes so people are yeah so it's all there's a lot of noise in this right because you don't actually know I mean most of us are not doctors and we don't really know well first of all we don't even know where a C or a D they don't tell you that they don't say oh we've diagnosed you as a C and then diagnosing what the other people there are there for I mean you can get some sense of who's really urgent but I think it's hard to tell who's ahead of you in the queue which is why we just use that total crowding as our statistic we looked at because we you know we think it would be hard for you to actually figure out exactly who's here even though in some sense it makes a difference right you're going to be doing some estimation of oh they look fine they're not going to oh how much waiting time that's true so they're just I think they're just doing aggregate numbers though right because again you're not telling people are you a C or a D so they're just putting up screens and I think again it's not my work so I think all the screens are doing is saying average weights of that people sort of predictive models so I think it's just you know here's the waiting times so again you don't know if that's going to make people wait longer more likely to wait or not they're also trying so they've got quite a nice controlled experiment though so they have the screens blank for some of their days they have the screens showing something entertaining for other the days and then they have these statistics on waiting times for the third ones so it'll be very interesting to see you know what actually affects because there's all this psychology of queuing that also says that if people are entertained they're more likely to wait so that's why they have mirrors outside of elevators because it makes the wait seem less long because people are busy looking in the mirrors there was also an airport that had all these complaints about the baggage taking too long and what they did was throw them away from the gates and the complaints went away so all of these things are important particularly in the psychological things which are hard to model but they are actually important to understand from a system design standpoint especially in an emergency department setting we were actually talking about people's lives so this idea that these higher observed service rates giving patients a sense of progress is a good idea so the company certainly wants that it makes them think they'll have to wait long the other is just less comfortable yes yes is there any way to disentangle that or to not with our data but I think it would be important to disentangle because I think you're right that those are actually two effects that are both important yeah yeah especially this is adult only but you can imagine some waiting rooms you've got screaming kids and yeah yeah which is an objective function mm-hmm I mean it seems that if they've got people waiting and some people leaving they'd rather have the less urgent people being waiting yes and that's not necessarily what's happening right it's those I mean the ease who we don't have a huge amount of data set they have fairly high leave without being seen rates but this could be just they want to adjust the queuing as well as the how they import people right so well they do give C's higher priority so in few maybe they're not giving them high enough high enough yeah but then they've got their B's in their A's right so yeah I mean they just need more of these horses ideally and less boarding which is where all is started right if they can get those people out of beds that are supposed to be in the main hospital they can move people into beds which you know that was the original motivation of of the study was you know what effect is that boarding having on these patients leaving without being seen yes it is which is another interesting another interesting question yeah yeah yeah yeah yeah yeah right yeah yeah like his you yeah yeah yeah yeah well yeah except it's non-stationary so eventually to calm down and things but yeah yeah no some of the weights I don't know if you saw the I didn't really pointed out the mean waiting time is 97 minutes but there are weights in that data set of a day that was the extreme but yes so some of these weights are very long um so so that's that's kind of some insights in terms of modeling this in terms of what we can do managing this but in terms of modeling a lot of our queuing models do actually just assume on how long you're going to wait and then you leave so we can we can look at some hazard rates for our different security classes there I wouldn't spend too much worrying about some of these bumps in terms of the data gets pretty thin out there um so so this is basically your likelihood of leaving as waiting time increases and the first thing you'll notice is it's not flat so unfortunately assuming an exponential time until people leave as most of our models do it's not great it's really not even close um the the I'm I'm not going to put too much significance to this kind of up and down but it is actually consistent with some other work that's looked at abandonment from a service system not an ED um and they also saw this unimodal up and then down in terms of kind of once you've sunk that much time and that somehow you become less um and that's somewhat the story we saw who is just going to wait and actually we are trying to test for that and nothing showed up in terms of that was my original hypothesis we've got two types of people here the people who are just going to wait forever and then the people who are who are not and we did try to tease that out and we couldn't see it but doesn't mean it doesn't exist um yeah so that's one of our control variables in terms of the original model but yes in terms of the hazard rates we have actually they look pretty similar we did the with the control variables and without and they look pretty similar so you know so how are we actually going to model this leaving without being seen um well if we can only choose one then actually you're better off choosing crowding than waiting time so actually if you build your model based on crowding slightly more accurate is what the data would say then if you build it based on on just waiting time um it's not necessarily monotone um and then we see this varying sensitivity so that's all going to be a little hard to model unless of course you go to simulation so if you've got a lot of decisions and emergency departments are made via simulation what I would point out is we can now use our um I can come out this way okay um you can we can now use our empirical model to plug into a simulation right so so you're simulating your patients in your emergency department the patient gets called at that time you know their observed wait time you might have to kind of estimate the service rate that they observe because that's keeping all that data is going to be a little hard um and you know the crowding when they arrived so we could them leaving then in the simulation you do a coin flip and if it's one then you root them to exit and if it's two you root them to a bet um the only disadvantage is you're going to overestimate the number in your waiting room because you're not taking them out of the waiting room until they get called but other than that that should be a fairly good model of leaving without being seen right um so yeah so that's one so that's if you want to simulate it so I think what this work has shown is if you want to simulate your emergency department I'm pretty comfortable with this as a way to do it if you want a queuing model which of course queuing models have advantages because they're explicit I think um if you have to use a patient's distribution Wyville seems promising and notice you could actually parameterise that based on crowding level which will arrive to another one could have another one um and as I just mentioned it looks like predicting leaving without being seen based on crowding is slightly more accurate than waiting time well that's actually good news because there's an approach for queuing systems so this is a plus on arrivals general service multiple servers finite waiting room and abandonment um that uses the number of customers in queue so what Witt's approach does is he estimates these abandonment rates depending on number of queue from the original patient's abandonment time distribution right so he's assuming in this model there's some general distribution until people leave well you don't have to assume that you could just use the rates at which people leave a different number of customers directly from the data so basically if you've got this many customers behind you those are your rates of leaving and there has been some work unpublished undergraduate thesis however to adapt it for priority queuing systems so there's an open problem as you could actually see if this works but I think this would be if you're going to do a queuing model approach would be based on numbers and then adapt it that this work make sure it actually works one of the things problems with using a queuing model is of course it's not stationary but people who do this stuff queuing models and emergency departments usually assume it's stationary enough within an hour or whatever your period is and then approximate it that way so in addition to the offered waiting time it does depend on these other factors I've already took that through and we kind of know how now how to simulate leaving without being seen and therefore we can simulate our emergency department queuing models might be a bit challenging if we only take one variable I think there's real potential there for how to do it and that's just the piece on leaving without being seen what are we actually going to do about open questions there one of them easily solved with simulation really hard and queuing is we actually have competing resources we've got nurses, beds, doctors all of which you need to actually get your service time that's hard priority service always throws a wrinkle into it basically their priority is absolute but it's not clear to me that's how they should be doing it and I don't know if you've seen Ilza who's here in Auckland present but she also has been thinking a bit about can we actually wait the time that you've waited with some priority and have a better policy and other people have shown that service rate depends on workload so most queuing models will assume that service rate is just from a general distribution but actually it's not doctors are people and they work differently depending on the backlog so it would be good to have effective models of that and of course the thing is these are actually life and death decisions which also makes this very interesting work because here you could actually save lives if you do it well enough so there's huge amounts of work to do I would say with modelling these emergency departments that's it