 Mike working All right Okay, so today I'm going to talk about some work that my research group is doing so I want to start off with a series of caveats here actually no, I want to start off by thanking the organizers at this event I know that it's kind of small and scrappy this year But I think it's really cool that this is going on I think these guys and women did a great job of pulling this together on a pretty short timeline And I anticipate this being Bigger and better in the future. So are you guys having fun? How many people are having fun? How many people are doing cool hacks? Building cool things. How many people are building something on mobile? All right, so My let me now let me get to the disclaimer. So Unfortunately, this is not a tech talk I was just peeking into the other room 113a and This guy's got code up on the board and he's talking about what this function And I said it looked really confusing I was like if I can't walk into the room and figure out what your framework does I don't know if I want to use it, but anyway, I'm not going to do that There's going to be no or very little code That's part of this talk and the code that is part of this talk doesn't work yet So this is not, you know, I'm not going to teach you something that's going to be useful to you in the next 24 hours. This is a research talk about stuff that we're doing that might work someday in the future, okay? This also is not about a Google product service framework or whatever Although we just got some support from Google to work on this project So maybe someday and they're not so distant future Google will actually support something like this This is a talk about the future. It's also designed to be interactive So if you guys have questions or are angry about something Please stop me and or if you would prefer that I talk about something else like graduate school or How to use the Google search engine better or something like that that you guys might care about Please just stop me. I will talk point out. There's a talk about entrepreneurship in the other room So if you're less interested in the future and more interested in making piles of money somehow That guy in there is really tan and he has a really nice tie So those are both signs of a successful entrepreneur. I have a t-shirt on and So anyway, you can you can use that as your way to judge our respective fates On the other hand, how many you guys have heard about mobile apps before? How many you guys are building a mobile app as part of this hackathon? okay, and I'm sure the reason you're doing this is is somewhat related to this idea that Mobile apps are going to be your ticket to the big time Somehow by writing a mobile app. They're going to make a lot of money I was peeking my head into another talk the other day and there was this guy who had a Slide up actually he was the other thing about this guy was he was speaking really loud I don't know if I'm speaking that loud, but I could hear him all the way down the hall Right as I was about to leave the building. I don't think I'm speaking that loud But he was he was talking and he had this slide up and he was saying this is what a billion dollars looks like and it was this Huge is anyone was anyone at that talk? I don't know what was going on But you heard this guy screaming about what a billion dollars look like so I had to peek my head in and he had this picture of this You know cube of money right and it was supposed to be a billion dollars But then it got got me thinking because I'm an analytical person like maybe you are maybe you're already thinking this Doesn't it matter what size bill you're using right was that a billion dollars in ones or was that a billion dollars in like $1,000 bills which might look smaller right so anyway, but this was a big Cube of money right like and you can imagine having that at your house and being worried about To having a big cube of money like that, but maybe it was all one so someone grabbed someone ran off They'd be like too bad. You only got a hundred bucks Okay, so those look like fairly big bills. So this guy's pretty happy But my goal today, I always tell my students when you give a talk you have to have a goal in mind You should have an objective. What are you trying to do other than fill up the time? So you can stop talking My goal today is to first of all to convince you that you are not capable of writing mobile apps You are not going to be this guy with the dollar bills You're more likely in your present state with the present tools that you have available to you to be like this Sad cardboard thing that has been left out in the rain. I don't know what that is or why It's out in the rain. It's clearly not cardboard because it would probably be disintegrated by now, but it doesn't look happy and However, I'm going to leave I'm gonna you know This is like it's sort of like the rising action part of the talk where you get sad and confused You're not sure exactly what's going to happen to you and then I'm going to provide you a pathway to the future I'm going to give you a way out of this predicament that I'm going to show you that you are actually in whether you want to be or Not and that way out is this new system that we've created, which is awesome And I'll describe that system to you and also this awesome features All right, so to some degree while the title of my talk might actually be building less certain mobile apps The actual title of my sock cock or the objectives of my talk are to make you less certain that you're capable of writing Mobile apps and more certain that you need my help to do this, right? And of course this is again I I do not have a tan or a purple tie yet, right? But I do have a money-making plan, which is that I'm going to cash in on your future success, right? So if that's you see there's me right in the middle of that pile of money that you've somehow made, okay? So this is my plan Okay, so let's talk about the app store So 15 years ago, it was really not possible or extremely difficult for a single programmer to quickly deploy software Across the entire world You couldn't wake up in the morning, you know, at your normal early hour of like 1045, right? I mean you're out of bed before 11, so that's a win You know, brew some coffee, go over to Starbucks, pay $5 for your latte And then around lunchtime decide, hey, I want to write software and distribute it to a billion people today And then you couldn't do that by the time you usually wrap up for the day like around 3.30 or 4 You know, because that's when, you know, Sports Center comes on again or whatever So you couldn't do this 15 years ago, right? And I'm going to argue that was good, right? This was a good thing So the software marketplaces that we all know and love like the Google Play Store They have exposed people all over the world to code written by people that wouldn't have been able to do this 15 years ago 15 years ago, there were processes for distributing software that you had to follow You couldn't just throw something onto the app store and hope that people would download it And now you can do that Um, why is this a problem, right? So here is a graph showing the distribution of programming talent, programming skill, okay? So I want everyone to perform this little mental exercise Look at this graph and decide where you are on this graph, right? Are you better than the average programmer or worse than the average program? So the problem is if you actually, if you guys actually told us what is in your heart of hearts This is how the distribution would look, right? Everybody out there, how many times have you ever met someone who writes software who says I'm a pretty crappy programmer, right? I actually think I'm a terrible programmer, I don't know why I'm doing this But I continue to write code even though it's terrible Most of us think, well, okay, I might not be an awesome programmer I might not be one of these guys who's like super, super productive But I'm pretty good, you know, there's some terrible programmers out there, they're not me, right? Of course, this is what the graph actually looks like, right? And it's a sad fact that 50% of the programmers out there are actually worse than the other 50%, right? I can say this without most certainty Um, when I went to, I graduated from Harvard and there was a joke there that Every year, Harvard's goal was to get more students into the top half of the class, right? That was sort of the goal behind grade inflation Somehow it didn't seem to work out that way So, now let's say that you are different, right? Let's say that you are half, again, so half the people in this room, give or take Are above-average programmers and the other half of you I should say half of the room, half of us are above-average programmers And half of you are below-average programmers So, even if you're in that above-average group, it's a good group to be in, I know It's extremely difficult to anticipate what goes on after your software is deployed And it's become more and more difficult as we've made it easier for you to deploy software So now that you can easily write an app and push it out on the app store And have thousands of people using it tonight and tens of thousands next week And a million in a month, it's extremely hard for you To anticipate everything that can happen out there, okay? And so I kind of think this is like the Pandora's box of software deployment, right? I'm going to allow you to program one billion devices, but there's also this caveat, right? The ways that we've allowed you to deploy software Deploy software onto some of the most complex and challenging devices out there, right? Mobile smartphones are super exciting as a development platform This is how we are bringing the web and computing to people The people that we're bringing online on the internet are using these mobile devices They don't have desktops, they don't have laptops Everybody with the desktop or laptop is already on the internet The people that are new, the way that we're expanding the internet All on mobile devices So this is incredibly cool, incredibly exciting But also incredibly difficult as a programming environment Okay, so now let me try to convince you That if this is how you feel about, I love using the laser pointer to poke this person in the nose If that's how you feel, if you feel confident about your ability to write mobile apps You should not, you should feel uncertainty and despair, okay? And I'm going to do this using only 16 bullet points, okay? So, okay, ready? Here we go Here are the things that you're not thinking about, that you should be thinking about That can affect how your apps perform in the wild How about mobility? Mobility is the characteristic of these devices It means that things change really quickly One minute I'm standing inside this room And I've got a great Wi-Fi connection And then I make the mistake of wandering outside Which is a completely reasonable and normal thing to do During three months of the year in Buffalo And then suddenly your app, network connections, ruined Right? Same for the accuracy of location services So maybe it's a trade-off I was in here and you had no idea where I was Because UB hasn't put its access points on these mapping services yet But then I walk outside and suddenly GPS comes up And suddenly you know where I am What other devices or resources do I have access to? This changes as I move from network to network I might be on a local network at home Or I have other computers nearby That I can contact, I can use them to offload computation To do other interesting things And suddenly again I walk outside, I'm on a mobile data network And this is over Those are not publicly accessible endpoints How about on a geographic scale? Mobility, differences in the networks that we've deployed across the world The networks that we have here are different in quality and speed In the aggregate as networks that you find in developing countries And sadly because you guys happen to live in the United States Is really awesome networks that you find in other developed countries That have put up much more kick-ass mobile data networks than we have What about the power grid? Mobile devices, these are energy constrained You guys are used to always having an outlet nearby Unless you're in an airport of course Because airports have all been built with no outlets in them But normally you guys have outlets everywhere You go overseas, you may not be able to find an outlet If you find an outlet, there may be no power coming to the outlet for a period of time So particularly the new people that we're bringing online onto the internet Suffer from a lot more variability when it comes to power availability than we do What about other devices? This affects how people use their smartphones You guys have desktops and laptops The new people that are coming online The new customers that we're bringing onto the internet Just so they can download your apps A lot of them don't So my advisor was recently overseas in Indonesia He works at Google now And what he discovered is Everybody there is a smartphone That smartphone has a feature that is probably someone unusual in the states Which is they are all out of storage capacity Why are they out of storage capacity? Because these people don't have a laptop with a terabyte of storage That they can offload their photos to This is the only device they have And so once they've taken photos of everybody in the family There's no storage left And this affects the ability of the device to install software updates And things like this Energy constrained So here's another thing that fluctuates throughout the day How much energy is left in the battery? How does that affect whether or not the user is actually going to be able to make it To the point they want to be able to get to without their device running out of energy How much does this device last? There's huge differences between devices When it comes to processing power When it comes to energy consumption When it comes to storage capacity, network usage All of these features So the hardware has a big impact on this And even if you can know these things How much should your app actually use in certain states? Is it okay if your app uses up 50% of the user's battery? Maybe if your app is awesome and the user loves it But maybe if your app hasn't been used in weeks or months It shouldn't be using any energy at all Maybe at that point any energy consumption is too much So users are a great source of variation, sadly How often does a user use the app? What percentage of their total usage time does this represent? So this could be a very popular app to them Or it could be an app that they've actually forgotten that they installed Or it might be an app that they only get out once in a while to do one very specific thing So this affects how your app should behave How important is your app to the user? What other apps do they use in an Iraq with? An app that's on a device with only four other apps Is probably can behave a little bit differently Than if you're on a device that has 50 other apps How often do users charge? Even when power is available You have users that charge all the time In the car, at their desks And then you have people who only seem to plug in their phone When that little warning comes on And both of those people exist Sadly, despite my attempts to move people from the latter category to the former category Do users even know how to connect their device to a Wi-Fi network? So we run a smartphone testbed here And one thing that we've always been surprised at Is we have users that never use Wi-Fi They don't seem to be able to figure out how to connect to the UB secure network Right, which admittedly is harder than it should be But also there are very very good guides available for how to do this Do users have a metered or unmetered data plan? This affects how much network you use And how that network consumption affects the user So, oh, I'm not even done I'm not even counting my own bullet points, right Devices, does this device have a particular feature? And what are the power performance trade-offs inherent to the device? So this is, if you combine all these things together There are decisions that your app has to make That are very very hard to make confidently And so you should be uncertain And my advice is don't try to solve this problem yourself And I'll show you how you're going to try to do that in a minute And then I'll show you a different way that we have To give you guys new tools To build more adaptive apps that can respond to some of these challenges Okay, but this is sort of the mantra of the rest of the talk, right Which is that in general programmers are dumb, right Especially and including you And it's not really even fair to say you're dumb You guys are smart people But you're not going to be able to figure out how to adapt all that variation that I just showed you You just can't do it Partly, you don't live in those other countries You don't experience those networks You don't use those devices How many people here have a device that they purchased in the last year or two? A smartphone Okay, you guys have no experience of what it's like To use most of the devices that people out there in the real world actually use, right Your devices are fast and they've got components that are really energy efficient and they've got lots of storage Try going and finding a nexus one Right and see if your app still runs on that device, right There are something like 20,000 different devices Supported by android at this point Which is a pretty stunning number And as the android ecosystem continues to age that number goes up Okay, so Let's look at how you might try to do this today So let's say you're doing something simple in your app You have some sort of background task Maybe you're checking for some sort of update from a web endpoint And you're trying to figure out how often should I check If I check Frequently it means that the app has fresher information And I might be able to deliver notifications more quickly If I check infrequently it means that the app might have stale data But it also reduces the amount of energy consumed by the app and the network usage Okay So now hopefully I've got you thinking about all the different things that could affect this otherwise very simple decision And you might end up thinking I don't know what the right rate is to use at this point Okay, so here are some of your options today This is the most common thing that we've seen people do Which is they guess Right how many people have written code like this where you've had to pick some sort of timer interval And you think oh, I don't know five minutes is a nice number Right, you know, I like multiples of two 32 seconds kind of weird But whatever it's multiple two it makes me happy because I'm a computer scientist So you know 1024 I don't know how long that is but whatever again another multiple two So yeah, I mean you you come up with little you know little guesses right So I'll just say okay every two minutes. I'm going to check for updates. That seems reasonable. Maybe that's a little fast Maybe I would say 10 right So that's one way you can do right But here's the thing if you were not sure about how to pick this value There's a probability that you're wrong That in certain cases this is too fast and also in certain cases that this rate is too slow So the other thing you could do is You know, let's say your app is out there and you're bored and all the money's rolling in and You know in between all the fancy dinners that you're eating and the trips to Tahiti and stuff like that You know, you want to make a couple updates to your app So you might sit down and think okay. Well, you know originally I used this static Timer interval and I don't know. I can do better now now that I'm a billionaire now They hit this huge block of money at my house Um, I can actually do better than this. So let me let me write up some code that does this Okay, so here's what I'm going to do right You know clearly I care about energy consumption and there's a few comments on the play store Despite the fact that my app has like a 4.8 rating. There's a couple users that are complaining because they say it drains their battery So here's what I'll do I'll set some sort of threshold and I'll say if the battery level is below that threshold that I'm going to slow down the rate at which I Check for updates. Otherwise, I'll actually, you know run it at even a faster rate So this seems completely reasonable. Has anybody actually written code like this before? Yeah, most people just guess yee-hong you're not you're not a lot to answer Um, yeah, most people don't even bother to do this right because maybe again Once the billions start rolling in you just forget about it But here's a way to here's an attempt at adaptation and this seems completely reasonable right Okay, so what's wrong with this? Well, there's a lot of things that are wrong with this right I just told you that these devices have different battery capacities They consume energy differently your app may be the user's favorite app or an app that the user rarely uses And so can you really pick a single threshold here? Maybe for some users you should never check for updates or you should check for updates once an hour And for other users you should always be using the fast rate because they Um, they always use the app. It's the favorite app. It's open all the time Okay And then what threshold is that going to be? How are you going to choose this? I deliberately left this off the slide because who knows 20 percent 10 percent 50 percent Really depends on how energy constrained this particular user is So, you know do these branches even do anything? Right Depending on what kind of network you're using it may be that the cost to perform this check is actually really low And you may not need to care it may work to just set one one static thing and just move on Right if most of the time there aren't any updates What's the overhead of just sending a few packets over the network? It's not that bad Right or you might find out that when I'm on 3g this is really critical Whereas when I'm on a 4g or an lte network, it doesn't matter as much And So you want to know not only is there a difference, but how much of a difference is there? If there's a big difference that I might want to slow down more aggressively If there's a small difference, I might not even bother Okay And by the time we get done sort of starting to try to answer some of these other questions You're you've gone back to doing what I told you not to do which is to be a superhero Right and most of us when we try to be superheroes end up like this guy Right who's like an overweight superhero with a duffel bag, right? Uh, most of us just aren't cut out to do this. Okay So what's the root of the problem here? The root of the problem is that you know You are being forced at the time that you develop your code to Collapse uncertainty you have about how the code should work Into the certain code that you have to deploy into place Right even if you try to do this case-based adaptation that I just You know just destroyed right here That's still completely deterministic And there's no way for this code to adapt itself at runtime So our solution is simple right stop trying to be so certain right, you know, this is a this is a message of hope Don't program this way When you are uncertain Find a way to tell the system And you guys don't have that way yet, but let me show you how to do it So here's what you would do today. You try to do some sort of adaptation that has to be baked into your code right up front and um, it is the the How well this works is completely dependent on a number of different things that you don't know yet So let me show you another way to write this code Like this So this is the system that we're building Is designed to support entirely this syntax So You'll see the difference here right What this does right Is rather than forcing you to write deterministic code and collapse uncertainty in your system It allows you to identify parts of your system where there is flexibility What you are telling our system is that It is okay to do the thing either the one way or the other way. That's all right That's okay with me I don't know which one is best And so rather than trying to figure it out when I wrote the code I just decided I would tell you about it and let you figure it out So Now what this requires now is an entire system for getting from here To the ability to make this choice at runtime So the goal is when I hit the top of this new code block The system should know what to do It needs to do either the first thing or the second thing and it should be able to figure out which one is better So we do this in three And there's three parts to the system as any good system There must be three parts right if you have a system with two parts or four parts You know either add a part or combine two parts together right because you'll find a three part system is much more satisfying Um, so the first part is you express Uncertainty or flexibility is another way of thinking about this when you write your code And we give you a new language tool for doing this The second thing is the second step is we do some form of testing Because we the system our system doesn't know anything other than the fact that there is some flexibility here That I could potentially use to adapt the system later So what I need to learn what the system needs to learn is all those things that might matter to how this decision is made Right all those things that we discussed before How do they impact the performance of the different options that you've provided? And finally our goal is to get to the point where we can build some sort of decision logic That allows me to make this decision accurately at runtime So remember the system can't do both things it has to do one or You know one out of the end that you've provided And so figuring out which one to do is the end goal Okay, the goal is to be certain at runtime Let me pause here and ask if there are any questions About this okay So let me show you some of this just to give you a flavor of the syntax here So here's an example of using a maybe block statement to Express uncertainty or flexibility in code code blocks, right? So this syntax is familiar to you if you've used a you know a conditional statement before Except for the fact that there's no condition Because by you know by definition you are not providing a way to make this decision Just a indication that there is flexibility here that could potentially be useful Okay, we have a label That's part of the syntax that identifies the statement and it's used When you use other api calls that are part of our system And then you provide multiple alternatives We also provide a keyword that allows you to use this as part of assignment So on some level this is so just as they say syntactic sugar, but it's nice At the first statement says that the perf string can take on three different values Low, medium, or high the second statement says the timer variable can take on values between 1 and 60 So this is a way and again you could write this using a maybe Block syntax, but it would be kind of ugly and and verbose. So this is a little more compact Okay So the other thing you have to clearly have to be able to do here Is to tell the system What happened and whether or not it was good or not There is a notion of goodness that's built into the system There is algorithm one or algorithm two is better on this device with this pattern of inputs produced by this User given this relationship between CPU and memory speed blah blah blah So there's a lot of uncertainty, but there has to be some way of measuring this right you have to be able to tell the system Algorithm one was better or here's a score for algorithm one So what we do is we provide a score api the score api Allows you to programmatically associate a score with a labeled maybe statement and this should say algorithm not label Sorry, it's a bug in the slide Here what i'm doing is i'm telling the system what i care about is the runtime of these two algorithms I don't know which one is better, but i can tell you that Here is the thing that i care about by using this api call As you might guess i can also do this automatically So for on if you don't if you're lazy like most programmers are you don't want to actually do this We can infer certain things about the flexibility that you've provided That are common so one common thing would be runtime How long did it take to execute another common thing could be energy consumption on mobile devices? But here's sort of even if you hadn't provided this here's The equivalent of what's happened. I'm measuring something about the alternative and i'm associated the score with the label So this is a measure of goodness All right, so the the other thing the other part of the maybe api is a log command So our system will automatically Uh record a lot of different information that could be potentially helpful or potentially useful When trying to figure out how to make this decision things like What kind of network am i on how much battery is left? Um what other apps are running information about the device that doesn't change very often, you know things like this But if you have some sort of specific information that we wouldn't be able to normally Normally infer that you think might have an impact On how well these algorithms perform you can tell us so here again same thing that should say algorithm not label What you're doing is you're saying I think that it's possible that the performance of these two algorithms depends on the length of their inputs And so i'm going to tell you that So that when you do some machine learning later, this can be a part of the inputs you're machine learning out Because if you didn't maybe you wouldn't it be able to figure out how to choose these things correctly Okay One way that I like to think about this is this syntax allows you to separate your app into two parts So rather than having to build a single app that runs the same everywhere What our syntax allows you to do is produce a portion of the app That you now write that does run the same everywhere. This is sort of standard stuff that's required for correctness And then there's the part of the app that is tailored to me to a specific user By creating flexibility you allow this division to happen So if there are decisions in your app that are required for correctness They have to go in the part of the app that everybody gets The idea behind this is not to allow you to write buggy code It's to allow you to expose flexibility so that I can take a portion of the app and customize it very closely to each user Okay, so let's move on to the next step, which is testing So once I've expressed the flexibility in my app Next thing I'm going to do is perform some kind of testing now Here's one easy kind of testing for you guys to think about which is Post-deployment split testing. So you send out your app to a thousand users 500 of them run algorithm one 500 of them run algorithm two Or maybe they bounce back and forth between those Over some period of time I record information about what happened And I use that as inputs to the next step of the process In certain cases we're also looking at ways to run this beforehand. So if I can Pull out good automated testing suites for your app And run them in an environment that has realistic performance I might be able to get a sense a little bit of a hint About what works and what doesn't even without having to bother real users Because clearly this is a bit intrusive. It means that there's some portion of your user population That's actually might end up running code that doesn't work very well for them But it's required in order to so we can learn about what works well for everybody So, you know, here's my little Graphic that says what I just said, okay So yeah, so we're looking into ways to do this through simulation, which is better because we don't have to bother real users Various forms of monkey testing On real devices that again don't have to bother real users or they have to bother the poor monkeys And so so here's a fun idea that we want to try out which is this idea of simultaneous split testing So I told you that that phone can't do two things At the same time it can't actually choose two of the alternatives But maybe it can with enough compiler support We might be able to execute all of the different alternatives one by one And pick one of them The reason to do this is simply because it means that all of those different statements were executed in roughly the same environment So as long as the network conditions aren't changing extremely rapidly I might be able to run all of them under essentially identical conditions Which gives me a really nice sort of apples to apples comparison between the different pieces of flexibility that you've exposed Okay Finally, I want to be able to remember the goal here is to be able to make a decision at runtime I have the top of this maybe statement. I want to know what to do And depending on the type of flexibility that you've exposed and how it varies Based on all these conditions. There's a couple of different ways that this can work out The the simplest one is it doesn't matter So you may have exposed a difference between two algorithms or between two strategies that essentially is irrelevant They just end up producing identical performance Another simple outcome is that there is a single winner So we find out that it turns out that you know and you didn't know this So you found out something new but one alternative that you provided just outperforms everything else everywhere that we can test it So this is essentially what's where current sort of split testing approaches Stop, right? So up to here What we're doing is split testing which some of you guys may be familiar with And I like to think of this as split testing with better syntax because I really like the syntax that we have I think it's very elegant So at this point depending on you know, how you feel You may choose to just remove this particular bit of uncertainty or flexibility from your code You may find out oh man that thing I tried just didn't work. I'll just get rid of it. I'm a little embarrassed by it I don't want anyone to see that crappy algorithm I tried I'll just leave that out because I don't anticipate it'll ever be useful But you know, it's also worth pointing out that is devices change If there are small differences between things you might want to leave that in there Just in case we find out a year later when we test it again That oh, okay, this new device with this particular You know a set of components actually does better with this flexibility that's still in there, right? Okay, so now now is where it gets fun because that's sort of again That's that's where split testing gets right split testing leaves you in this position where you have to pick one One thing to run everywhere and of course the nice thing about split testing is that You do only get to pick one right because it's easy So I do some testing on two different devices And somehow I have to pick one because if I don't pick one it becomes a lot more complicated But it also becomes a lot more interesting um So here's another pattern that the adaptation could take which is that it might depend On things about the device or environment that don't change very quickly So in this case what that would mean is that if I took all you guys in the room About half of you would do better with one algorithm the other half would do better with two algorithm And that would be and I could just tell you guys that I could tell your the apps running on your phone that and I'd be done No, I need to maybe test again in a year when networks have changed or whatever But but pretty much I don't need to do any dynamic adaptation What's best for you now is also best for you now five minutes from now in an hour from now Um And this is probably because the best alternative depends on something that doesn't change very quickly Right like for example the model of your device You're later. You're buying a device. I have to start over but as long as you continue to use the same device. I'm good, right In this case what we can do is we can do the learning on the back end And we can propagate decisions to devices and just let them let them hold on to those decisions forever The second case which is really interesting is when The adaptation actually depends on time So even for a single user the best choice depends on something about the environment that changes quickly In that case. I actually need to be able to propagate strategies for adaptation to the device So your device actually when it hits the top of that statement has to do a little bit of extra work to figure out Um, you know, okay. Well, I might I might have slow network or a fast network How does that intersect with other things that I need to compute and make the decision? But essentially what I'm doing is propagating Custom decision-making logic for each statement that's based on learn performance attributes for your device Finally if everything if all of this fails I can always fall back to doing something extremely simple, which is every time Somebody installs the app. I do a little bit of testing And I probably even if I can make good decisions about what to do I'm still going to introduce a little bit of uncertainty in the system so that I continue to learn over time Right, um, but if worse comes to worse It may be that there is no algorithm that can take the attributes from your device and predict what to do And in that case I can Either do one of two things if it's a static adaptation case I can just test both and see which performs better If it's a dynamic adaptation case, I may be able to automatically create a strategy just for you Right, which is kind of cool Um, and finally and and I really hate this option, but it's in here for correctness sort of completeness I could also just make you make the decision Right. I have two alternatives. You know, I don't know maybe this is because they have trade-offs Maybe this alternative always performs better, but also consumes a lot more energy So what I might do is put it in the settings automatically promote it into the settings dialogue for your app and let you make the decision The nice thing about this is that I can tell you Exactly what impact that setting has on the app So right now when you choose settings in the app, it's like well, I want to I want to you know On my music player it might be like well, I want to sync music on 3g on mobile data networks and on wi-fi But I don't know when I choose that setting how that's going to impact me Right, how much data extra data usage is that going to cause right? So this we can do because we've done this testing and we know more about Okay, and you know, obviously as things change over time Uh, this is not the end of the road We get into a loop here where we're testing and resolving uncertainty and building new strategies dynamically as things change All right, so currently We're working on a Java-based prototype for that's targeted at android apps because we think android's a fun place to do this given that there's a lot of Uncertainty and mobile app development We are If you want to talk to the person who's building this primarily it's yee-hong back there raise your hand Yeah, so yee-hong is the one who's and this is a little out of date. Actually yee-hong should get this part of the talk Um So Let's see. How am I doing that time? Okay, I'm almost done. So I'm going to skip through the results because they're not super interesting And I want to talk about a couple other things so So we're also other than than mobile apps. We're also looking at other places where we can expose uncertainty and where it could be useful So one place that I'm excited about are libraries that run everywhere So sql light is an example of something that has become a really Critical part of a lot of standalone apps, right? There are a lot of apps that use sql light Most android apps use sql light for a variety of reasons. It's used for or m. It's used for simple object persistence In a lot of cases it's used in ways that it really kind of shouldn't be It's basically, you know, we've seen a lot of apps that use it essentially as a key value store There are better ways to do that, but hey You know everybody who writes android apparently knows sql. So that's what they want to do And in this case the uncertainty comes from just the environment i'm in right an embedded device with, you know 4k of memory is a very different beast than Sort of a slice in one of google's data centers that might have an enormous amount of memory Right in a really fast disk So here the uncertainty comes from the fact that these ubiquitous libraries end up in lots of different places, right? Harnessing parallel development. So here's a way to write better apps Farm out parts of the app that seem like they're difficult or complicated and have three people on your team write that function right Three people write it it passes all the unit tests And how do I choose it? Which one to use don't Just wrap them all maybe statements push it out and we can figure out what to do for you All right, so this is a nice way of You know allowing teams produce to produce robust code for mobile and other types of uncertain environments Encouraging more brave development Right, so you know, let's imagine you're hacking on linux, right? And you find some old crufty code that has a linus torvold's comment on it, right? Are you going to rip into that and you know remove it and put in something better? I don't know I wouldn't right But I but here's here's what you can do, right? So here's a little bit of a friendlier way to contribute to projects rather than saying yeah linus that code sucks It's terrible Just say okay linus like we'll run your code if it's the best, right? Otherwise we'll run like him, right and let the system figure this out on its own Again, it's possible that your code is terrible, right? But it's also possible that linus wrote that code in 1982 when he was really tired It's still probably pretty awesome. But anyway, so there's another thing The other thing that that my group is looking at that I hope that That there are people who are involved in this community that will Get behind this this idea of expressing uncertainty in the classroom so I'm next year. I'm teaching a new course on how the internet works And when you take a class at university essentially what happens is I show up and I say Hi, my name is Jeff and I am here to tell you about all this stuff and unfortunately and you know, it depends on who you're learning from You may be more or less unfortunate, but unfortunately, I'm all you've got, right? So if I don't understand something Then you're going to learn my misunderstanding of that thing, right? Alternatively, if I don't explain the thing in the best way Maybe I'm not the expert in the area. Maybe I know something about the mobile web But I mainly know about it because I'm teaching a course on the internet I've learned about a lot of this stuff wouldn't it be great if I could go find somebody else to help me Right. So this is my former advisor matt welch He leads a group at google that focuses on mobile web performance So matt knows a lot about the mobile web map thinks about the mobile web all day long And I don't so maybe it would be better if matt could explain that particular concept as part of the course And so what we're doing Now let but you know, let's imagine that both matt and I have taken our stab at explaining this idea And along comes the confused student and I want to point something out to the women in the audience Which is I think alarming which is that if you google confuse student about 80 of the photos are female Okay I don't know you guys should get on google about that because it's a problem. All right I had to I had to look hard to find this confused man Okay, so anyway confused student Is not sure and and the point is who should get a chance to explain the mobile web to this confused student You know, I'm teaching the course and maybe I'm a little better at relating to students Or maybe I know a little bit about how to uh, you know explain things in simple terms to people who are beginners But again My former advisor studies the mobile web. He thinks about it all the time. So maybe he's better. How do I know? Maybe if I had a lot of confused students What I would be able to do is figure out that it turns out that that some confused students do better with Me explaining it to them the ones that are wearing red and glasses And the other confused students do better with matt And of course the nice thing about this is that if those confused students remain confused After either I or matt explain this particular idea to them As long as we're explaining the same idea We still can rely on each other's explanations for backup. So if the one of the Red-shirted glasses wearing confused students still doesn't understand the mobile web after I explain it to them for five minutes Then I can say hey Why don't you listen to matt? Because matt will explain the same idea and there's a little bit more reinforcement And unlike a lot of online courses the reinforcement is done by somebody else So rather than forcing them to say hey, why don't you listen to me explain the same idea over and over again until you get it? You stupid dolt. Um, I can say maybe I just don't match up with your learning style Here's somebody else who has a different take on who explains it differently uses different metaphors has different sort of a mental structure for understanding So we're rolling this out next year. It's going to be used in The first class on how the internet works in fall 2016 I'm creating sort of a library of online content for this class And i'm hoping that we can have contributions from other people In the ubcse community and in the broader tech community So if you think that you can explain some of these things and you want to give it a try The videos are at most five minutes. So it's not like you have to record an hour-long lecture with powerpoint or whatever Um, it's just if you have a very clear way of explaining something like a web protocol Please email me and we'll get you involved and this Site internet class door to earth should go live around the end of the year And at that point we'll be able to solicit contributions from other people in the tech community Okay, so in summary, you know embrace uncertainty Please be willing to try out our new system. This system will also go live Soon at maybe dot cse dot buffalo dot edu Um, if you want to do something that's like split testing, but a lot more awesome and also free Um, I hope you'll think about using our tool jay because we will You know have the whole thing set up where you can write this into your code and the rest of it just sort of happens automatically automatically Um, finally just a little plug for my group if you like this kind of thinking and you I mean clearly you like to hack because you're here at a hackathon But if you also like thinking about building experimental Next generation computer systems, uh, you know, please consider doing a phd If you're an undergraduate talk to me about other opportunities in my group I like this stuff and I like working with students. So That is other Is that where I ended? That's where the producer gave out. Well, it's a good slide to finish on So thanks guys for your attention and I will take questions Questions Performing it's a great question, right? So overhead of making this decision Depends on the decision itself is not going to be non-zero Right, so we will definitely have to weigh that against the potential benefit Right, so if we know after testing for example that there's only a small difference between two choices And yet we have to run a very complicated decision-making algorithm. We might say forget it Right, just pick one of them doesn't matter, right, but yeah, definitely definitely the case Right, so if the decision can be made in a in a static way for your device Then we push it to your device and there's no runtime logic, right? So at that point the overhead is basically zero The only time there's an overhead is when there's more dynamic Decision-making that has to be done in that case we push a strategy down to your device and that has to be executed Maybe every time the decision is made but maybe not Right, for example, if we can figure out how You know and more of a reactive programming model how it changes based on its inputs Then we can figure out when it needs to be recalculated So there's some ways to improve the performance of this But we are yeah, we're certainly cognizant of the fact that if there are small performance differences The amount of machinery that's required here is probably not worth it. Good question. Any other questions? Yeah Right so yeah, so the the testing our goal is to do as little on-device testing as possible Right because that creates situations in which users are exposed to Decisions that might not be the right one, right So what we do is that the testing process is something that you can control through the web api once you use our tool Um, but ideally we can figure this out before we actually even get it out to users, right? So if we can do some simulation if we can do some traces that we collect from users to figure it out, then great But Yeah, I know. Yeah, the simulations are going to be that accurate So probably in most cases we're going to have to push it out to a to a small group of people And run it for a short period of time But we can also we can also do sort of continuous testing, right? So this is common in all of these frameworks where you know One percent of the time when I make a particular decision I'm going to choose a random value, right? And what that allows this system to do is continue to generate new data about how well things perform So that I don't have to do like focus testing where I make a lot of wrong decisions all at once, right? So if I make a wrong decision every once in a while Not the end of the world if I can learn, right? Yeah, good question other questions Sorry, I've been favoring the side of the room, but there's more people over there You mean further once it's once it's a settings Yeah, so so so we really we really want to make these choices without punting them to users I mean to me that's the ultimate last resort. I mean apps have too many settings anyway, right Yeah, but again the nice thing about it though is once it becomes a setting We actually have a lot of information about it, right a lot more than developers normally do Right because we've tested it and we understand the trade-offs on that device So rather than saying You know again if you look at the settings on your apps right now There are settings that say this may increase energy consumption. Well, how much right like if it's by 50 percent Then i'm probably not going to check it if it's by like 1 percent Then I might not care Yeah, exactly so this but this could be done in our system for every user, right So for you, it might say this increases energy consumption by 3 percent for me It might say 30 right and you and I might make different decisions at that point Yeah, of course. Well, that's the thing depends on the device battery capacity You know for for example something like caching music over the network It really depends on when I add music to my library Right if I add it when I'm off on a lte network then the energy overhead might be quite large Right, but I know that right I can know all this stuff because I've done the testing on your device And I know things about how it works But yeah, no, I like the idea of doing these things, but I don't like settings, right? I don't want more settings. I want fewer But at least if we have to do settings we can do them with a lot of extra data that is helpful to a user Any other question? Yeah Yeah, so so it's a great question If the score is something that the that the system can compute Then that's id so if it's performance related If it's energy consumption and there's not much of a performance difference My group is also involved in studying like a bunch of other different aspects of these types of systems So if I can measure for example the responsiveness of the ui That I could use that as a part of the score But there are other cases where it kind of depends on what the app ends up doing, right? So Maybe your goal is to Maybe your goal is to engage users And have them spend a lot of time sort of inputting data into the app, right? So if you can measure that you try two different layouts, right? You measure that Then that's the score for the layout Right in that case the score is not something that the system would have been able to compute for you Right So ideally you know in the type of things we think about because my group is sort of focused on building mobile systems Rather than higher level app type stuff. We think about Performance, you know energy consumption and things like that, but I think there are softer aspects of app Performance that you might be able to use the scoring interface to express Right, but there will also but that also ends up being things that you have to measure as well Yeah, yeah, so the the trade-offs is a really good example of a place where the system Uh would would either need to do one of two things, right? Either we need to Make either the trade-off creates a new decision, right? Which is Is it okay for this app to waste a lot of energy to improve its performance a little bit, right? Because for example, if I have a setting that improves performance by 10x But only increases energy consumption by five percent And that's a win, right? Whereas if I have another option then increases battery consumption by a factor of two And only performs produces a small change in performance Then again, so the case is where there's a real trade-off Might end up being things that you either have to promote as a setting Or have You know a a sort of a system-wide setting for different apps, right? So for example, the system might be able to figure out Okay, these are the apps that use the most often for those apps It's okay to trade off energy for performance Whereas if I have an app that you haven't used in a month That's just running in the background that app can never trade off energy performance, right? It always has to use the option that reduces energy consumption Even if it would increase performance because who cares about performance, right? Like you don't use it. So Right, so yeah, the trade-off case is interesting, right? That's something we need to continue to study We need to see how often that actually happens These are very good questions impressed Any other questions General life skills questions that I can fail to answer All right. Thanks. You guys have been a great audience. So enjoy the rest of the hackathon I wish you the best of luck in creating awesome stuff