 All right. Thanks, Dave. I hope you can all see and hear and you just Tell me if you can't but yeah, well, let's save the questions until on the very end All right, so welcome to my Licentiate defense It's in kind of weird circumstances, but you know, that's that's just the times we live in So title of my thesis is the whole story type-driven synthesis and repair and You know, it's a bit of a pun, but it's also because you know, it's all about these typed holes and And and I can a lot of the things I work with it are based on how how we can use these type holes To yeah, dude to do what we want to do. So let's just start with that, right? So what is the motivation right? What are we trying to do as a motivation behind this essentially, right? and so the big picture is that You know when we program right and we write programs we write a lot of code, right? We write a lot of text and then we we just compile that text into source programs or programs that we can run right and And in some languages, you know like like Haskell You have type systems and what type systems allow you to do is that they allow you to kind of give a specification small specification of what you want to do to the to the Compiler and then the compiler can actually check that your code Let me just start a timer Yeah, then the compiler can actually check that your code matches the the spec spec that you make right and And some languages even kind of go up it further than that and they allow you to and they they can they can infer The type of the code, right? So you don't have to specify what all the code does you can just kind of partially specify it And then the rest is inferred and Some just infer all of it, right? So and we have these these types and these small specs available, right? But we also have tests right people write tests for like how they expect their code to behave so In particular in Haskell there's a this Notion of these property-based testing, right? So it's a very popular property-based test-based framework in Haskell called QuickCheck and there's a kind of a tradition in the community that you You write these properties along and you ship them along with your code, right? So and the bottom line is that We have these code bases and we have a lot of information in these code bases, right? and what I want to do in this thesis and And and further on my PhD is to show how can we kind of make better use of all this information that we already have around To help with development of code, right? And then not just you know afterwards like I don't want to just compile the code and then run the properties I want to be able to use it during development, right? Because so while you're writing the code I want you to be able to kind of make use of all the information that you Provided to the compiler so far, right? So in Haskell you have these these type systems, right and the one way that that They that ghc the Haskell compiler that we mostly use Allows you to kind of interact with the the type environment is to have these type holds So what is the type told? Well, you you write an underscore somewhere in the code and then you run the compiler and you get an error, right? And the message looks something like this. It says that it found a hole. It tells you where the hole is, right? So this is in the interactive context, so it doesn't tell you much It just says that it's in the interactive context and then it tells you What the type of the whole is like what is the inferred type of the whole? So I'll show you better in a demo later on like how we can interact with it, right? But so my first my first paper that came out in 2018 at the Haskell symposium in St. Louis Was this paper called suggesting valid hole fits for typed holes Okay, and what do we do in this paper? Well in this paper we we kind of take these holes and instead of just displaying the type the inferred type and the environment We do some additional work and we try to figure out based on the type and the environment What can we put in place of the whole? Right, so let me just show you a demo right away, right? So here so first of all I'm going to turn off the type told just to show you how it is originally, right? So here we give it a context. We say this is a hole and it should have the type That's what these double colon means and it says it should take a list of integers and give you one inch, right? And this is the original thing that comes that I showed you earlier, and if we have some context Let's say if I have say let a equals a One two, so let me make sure there's an end one two three in I can give a context to this And you can see that this this a that's now in the context of the whole it actually appears in the relevant bindings here Okay, so let's see if we turn on the valid hole fits Show valid hole fits now, so they're on by default So that's why I had to turn them off in the beginning, but now we run the code again and instead of just giving the environment here We also get the valid hole fits now. What are these telling us? Well, they're saying okay, if you have something of type List of ints to int you can apply the head function to it, right? And the head function just kind of extracts the first element of the list So it's telling and then you can apply the length function which gives you the length of the list You can find the maximum element you can take the product as a list and so on right So it's telling you here are the things in scope That you could use in place of the whole and they would work, right? So if we say here, maybe we say head Then well, it doesn't it doesn't compile because it's not applied to anything So we use head and we use it on the list we had right it works So we can say here that okay if I say this is a function that takes in the list and returns a number It says we can use these functions and we can also use the maximum, right? So we replace the hole with maximum and then it finds the maximum element in the list, right? So these are pretty useful in their own right but we wanted to take it a bit further and Because sometimes, you know, you're trying to write a program and it's not just the single Identifier that you need right you need more complex expressions like You know, you need expressions that are more than one just one entire fire. So for that purpose I added these refinement levels, right? So what they do is that If you set the refinement level to two it allows up to two additional holes in the fit, right? Let's just see how that works. We run this again Well, it takes slightly longer. It's looking at more things But now we have the original violent hole fits But now we also have this refinement hole fits right and they're telling us okay If you give me so if you replace the hole with the fold L1 function Apply to something of the right type then the whole type will actually match the type of the hole, right? And and because we set the run level to two it's saying okay If you give me the fault L if you give me the fault L function and you apply it to two things of the right type So the first thing element has to the first argument has to be a function from into into int and the second one has to be an Int then it will match the the type of the hole, right? And we can see that okay We do that and and the cool thing here is that we can kind of use this recursively, right? We can say okay fold L and then We can give it two numbers and it will look again and now it's looking for two holes and then it's saying okay You can replace this first hole with something of the type B. Okay. Now the type of the hole is a bit too general So let's actually just copy paste exactly what it says, right? We have to give the type as well So we have to say okay, I'm gonna give you something of type int to int to Int and I'm gonna give you a Something of type int Let's see Right here in okay, and then it finds things that work for that right so it says for the second hole Well, it doesn't know what to give you but you could say something like max bound or min bound But you can on the first hole you can replace with something like const or minus or something like that, right? So we pick something the right types. Okay, and we say Let's give it plus and let's give it a zero here and then you know it works, right? It does the sum of the list so that's pretty useful and Yeah, so and the main thought about this was yeah, so now we we were kind of just querying the compiler We're asking it. Okay You so you have already how all the information, right? You know what's in scope and you know the type of the hole just give me things that could work, right? And it allows you to kind of explore more and be informed about the environment, right? And when one interesting problem here is How do we how do we pick what we what we look at right? So let me let me go back to to know To know valid hole fits and not the refinement level and so the idea here is okay So how do we how do we pick one of these right? So one thing that we did is that we try to sort them in a nice way And the way we we sort them here by default is the size of the type application as we call it And it's just saying okay like how how how so it is looking at these things like yeah, how how many Types do you have to pick to make this work, right? so that's why head and last are first here because you just have to pick one type of You just have to say that a is an int But product it's kind of farther away because you have to say okay This t here is going to be a list and this a here is going to be an int so then we we sort by the size of the of the Of that type application as we call it So let me show you another example, which is it takes in a string and returns a list of strings And there we have these you know This is like a function takes one string it breaks it up into more strings, right? And and the functions in scope are lines and words which is kind of pick the lines in the string and the words in the string But we could also kind of repeat the string again and again, right? And then we just have to set repeat at string, right? But for these two we don't have to pick anything so they're closer But the other way we can also do it is like sorting by subsumption. What is that? Well, it kind of means a You know try to sort them so that like the more specialized ones are first and the more generalized ones come after, right? And the idea here is okay repeat is is a More general than lines in some sense, right? if you set this a to be a string well then then the type of repeat matches the type of lines, right and then return is even more general than repeat because Because if you pick the monad m here to be a list and then the a here to be a string It's it's like a more general version of repeat which is again is a more general version of lines But this is a bit slower because you have to pair wise comparison to make this graph, right? okay, so but So okay, so we now we we kind of we have this list here and now we we want to figure out, okay But how can we you know filter it, right? We ordered it somehow, but how can we filter it more, right? Let's go back to the presentation, right? So This is what I said in the paper, right? Can hard be hard to choose which fit to use But multiple fits with the right type, but different behaviors are suggested, right? And then the important part is being able to hint to GSE how the function should behave would allow us to discard wrong whole fits and now at the time the the idea that I had was That we use something like the refinement types from from liquid Haskell where you can actually in the type specimens type systems specify these Sat clauses you can say, you know that the type of the function is not just Into bull it's actually in to Bull and you know that the bull is going to be true whenever x is positive And then this gets sent through constraint solver and the constraint solver kind of make sure that these programs don't type check unless the constraints are satisfied, right? But you know this this is not in the in the sense You know like in the original motivation, right? We want to make use of the things that are already there, right? And here we're asking We're asking the user to provide us with more specific types that they'd have to do more work to get some use out of this, right? But what we already have are these properties, right? So as I said, we have these property based tests in the Haskell community, right? And and usually you have them already available So so you might you might not specify this type, right? But you might have the The property is positive available, right? So you can kind of specify this sad thing as a property So that leads me on to the the next paper in this presentation, right? So that's called a proper property based automatic program repair And we submitted this last September and it got accepted to to XE as hell this year And it takes this idea right using Properties to specify how the programs should work, right? But you know because we also like we said before we want to make use of what's already there, right? and if you just give a bunch of properties and then try to synthesize the whole program from just the properties, you know You're not making use of a lot of the the information that's already there So so we're basing this on automatic program repair, right? so the idea is that you you have a program that's Almost correct, right? And you have some properties about the program that you want to hold and The idea is you want to repair the program so that it satisfies the properties, right? So to do this we we use the type tolls and I'll show you in a bit But yeah, there was joint work with with between me and Leonard And we yeah, I had a lot of help from from co-authors. It's not just my work And I want to thank them as well. Okay, but let's look at like how does proper do it like what do we do to? use this technique of Trying to repair programs from the information that's already there. So here's an overview This is a little loop that that the repair process goes through and It starts off with the source, right? We start up up here and this is the source code, right? That we have for the program and this is like the GCD The greatest common divisor algorithm But this program has a bug in it, right? So the bug is I'll just tell you right away. Is that this first line here? Okay instead of returning the B as it should It actually loops it just loops and there's no progress being made here This is just gonna loop over and over again And we chose this example because this is from another kind of genetic program repair paper This is the example they use there, right? So this is a very common example in the program repair community Okay, so how do we do this program repair now? Let me walk through you through the the algorithm So first we we find what parts of the program are the properties, right? And how do we do that? Well, first of all, we match the the we look at the types of the functions in scope So if they have like a the quick check property type, they're chosen as properties But we also look at the names because a Quick check has this thing where you can say quick check all and it just picks out all the functions that are called prop something, right? So we also just look at the names and we see okay if it's called a prop something Then we're going to assume it's a property, right? We're going to try and use it as a property The second okay now we found the properties, right? But what is the next part? Well We have to figure out. Okay. What what parts of the program are we going to try and repair, right? Because we can't just try and change the whole program, right? So we we we inspect the bindings of the properties and we find the targets, right? So we look at these props here and we notice that we're using this gcd prime function, right? Which is a function from the same module as the properties are in like they're in one of the The modules that are we were eligible for repair, right? So then we know, okay We're going to be looking at this gcd prime function and changing that function in order to do this repair, right? That's how we're going to kind of Shrink the search space, right? Okay, so then what would we do? Well, we we we find the failing properties Okay, and in this case the first property here Uh, it actually it's it's more like a unit test, right? It's just taking taking some input examples and checking what the expected output should be And this first property actually succeeds, right? Because it never hits the zero case It ends up in the second case here and finishes But the second property here well, it tries gcd prime with zero and some x And uh, you know it tries gcd prime zero of zero or whatever x and it's just going to loop again and again so, uh We we try to so so how do we what do we do next? Well, we try to find the the fault involved expressions, right? So we're not going to just look at the whole program right away We're actually going to try and figure out. Okay. What you know, what should we be focusing on right? What what are the parts of the program that can be wrong here? And you know the the idea there is We use this thing called haskell program coverage, which allows you to instrument programs And determine what parts of the program are being evaluated And then by taking this counter example generated by quick check And then running the property we can actually find what parts of the functions are being Run, right? What what what expressions in this function are being evaluated, right? And that allows us to determine the fault involved Right, and the idea is you know if if it's contributing to a property property being False, right? It has to be evaluated at some point, right? And those are the things that we can limit it to There's some future work that we have like to do this even better right and false localization in general Is just a very hard problem to do but but this is our kind of First first attempt to do it, right and it works quite well We actually in this case we locate the the faulty expression, right? So the next thing we do is that we perforate the expressions, right? Where does perforate mean? Well, that just means adding holes, right? So we take the the gcd prime 0b here and we replace it with a home, okay? Now in this example, I'm just showing you the the one that kind of works But of course, you know we we we we replace the whole expression with a whole And then we also have other versions where we replace different things with holes, right? So we kind of make different versions of the code that that's filled with different holes Okay, and then we run ghc and a plugin To generate the candidate fixes, right? So here this is where the valid hole fits from the first paper comment, right? So we we we run the valid hole fits Synthesis and for that we get the min bound But we also would do a little more work where we do things that they generally do in genetic program repair Where we kind of mine the code base for expressions And then we check, you know, are there expressions in scope of the right type? And this allows us to pick up like magic constants and literals and stuff like that Which makes it a lot more useful, right? So we generate these candidate fixes, right? So we get the min bound from just from the type, right? If you set this in a to int then that this matches the constraint it works We get all these ints from scope But crucially in this hole B is in scope, right? It's it's an argument to the function And because it's this this hole is located at the right hand side of this definition This B is an int that is in scope. So that will be a a valid whole fit, right? So that's suggested Okay, and then we do a candidate evaluation So then we take all the candidates, okay? We replace the holes with those candidates, right? And then we run the evaluation again, which means we run the tests again, right? And we check, okay, did this candidate fix the tests or not? And so we replace this hole here with one of these candidates, a B in this case And of course, this is the right solution So it works and then it satisfies all the properties And then in the end, we generate this fix, right? We generate a kind of a diff saying, okay, you should remove this line And replace the line with just the B, right? So I'm going to show you quickly how it works So let's see, so here we have the broken GCD function, right? We have the properties that we looked at And we just have some unrelated function here And now we're going to run this, I prepared a config for this to make it quick And we say examples, a broken GCD And now first it starts describing the problem That it's like pre-computing some valid wholefits And it doesn't work in describing the problem And this takes about five seconds And then it starts running, right? And it runs a genetic search But in this case, it doesn't actually have to do any genetic search Because the repair is actually just there, right? You can just find it immediately saying If you replace this with a B, then it works And then we get this diff that we showed you, right? And it happens quite quickly And what it's doing here is that it's really Because it doesn't just fail the property, right? What happens is that it goes into an infinite loop And of course, that's treated as a failure, right? You don't satisfy the property if you don't finish, right? So then we have some instrumentation to say Okay, if you run too long with just a timeout Then we treat that as a false property And by, yeah So we managed to actually fix the code, right? And this is the right solution, that's quite nice But of course, we don't really show out the genetic repair here So we have another function problem here Which is called three fixes And here we have this broken pair So it says one, two, three But then we have a bunch of properties Which essentially boil down to saying You know, the first part of this tuple should be three The second one should be four And the third one should be five, right? So this should be three, four, five And then we just run the program again on the three fixes And, you know, it starts off by describing the problem That takes some time And then, you know, it will eventually start the genetic search And here we can see that actually It's actually doing some genetic search, right? It has multiple fixes And it's kind of braiding together the fixes that work well With some randomness as well And in the end, you know, within 102 generations We find something that actually satisfies all the properties, right? And we get the broken pair repair So it's quite nice that it just It just works by just trying all the things And the trick here is, of course, I mean We don't try every integer in scope I mean, we don't try every integer But because we kind of mined these integers from the code base already We could suggest them, right? That works quite well And now the third part is this simple refinement Because you notice in these, okay So the first one, yeah, we have a property But it might as well have been a unit test And here we're just using basically unit tests But here we actually have a property That's saying, okay, we have this f here And it's saying folder minus one But we want folder, fold l plus two Fold r, right? So what we want is that we want Running, we want to run the function on a list It should be the same as summing up the list and adding two Okay, so for that, I have a slightly different configuration Just to make it faster for the Demo And then we run this on the simple refinement And as before it starts off by describing the problem It takes a bit longer now because there are more kind of Valid whole fits in scope that we're looking at But as before it runs quite fast And it finds the right solution, right? So if you replace this expression with folder two It actually works, right? Okay, so that's the demo And now I'm going to talk a bit about the empirical study that we did Just to kind of see how well does this pan out in practice, right? So we had a couple of research questions, right? So how do we benefit? Do we benefit from using properties, right? Or and to what extent? And then, you know, with respect to that, like, you know Is it just a benefit in kind of solving more problems Or is there a benefit in, you know, do they improve the search In some other way, right? And the third one, and this is actually what we were kind of Targeting with this paper, right? Is that a do we fix overfit, right? So one thing that happens if you just have unit tests Is that you can kind of come up with a fix that solves all the unit tests But it's not it's very specific to those tests, right? And they don't it doesn't try to quite hold up to the general problem So so we wanted to ask the question, okay Does using properties in automatic programming pair Does that actually help with the let me just check the type Doesn't actually help, you know, with this overfitting problem, right? Okay, so for the first question, we we we ran a bunch of Program repairs on we got data from from a functional programming course at Chalmers We ran a bunch of repairs on them We got some solutions, right? Proper data find things that that satisfied all the texts tests And then we checked. Okay. How many like in different configurations, right? So we have both unit tests and properties. We have just properties and we have just unit tests, right? And here's our result, right? So we have three different algorithms that we're using We have the exhaustive search, which just tries kind of everything In scope and then tries two of everything in scope And it's quite slow We have the genetic search and then we have a random search which just picks a valid overfit and random Uh, and we can see that a random one didn't really work. Didn't find any solutions. So Uh, the exhaustive ones did find some solutions, but the genetic one found a lot of solutions Uh, and we can see that, you know, genetic search paired with properties worked quite well Uh, but You know, but in general, uh You know, it didn't it didn't matter too much, right? We can see that for both, right? We we didn't do as well as with just the properties and the caveat here is of course Um That for both, I mean we have more tests We also have to satisfy the unit tests that were maybe not covered by the properties And then we do quite well in the units, but we also have more overfits, right? And we address that in the third question So so the end result was yeah, we didn't didn't really help with producing patches, right? But let's look at the other benefits, right? So for example, uh, the speed, right? So here's a graph that describes the the time to first result to kind of, you know, we ran the program We saw like how long did it take to find a repair, right? And it was we can see here that we have the genetic search in in in terms, uh, using the properties And that was very fast at finding something that that worked, right of the first repair Um, so I think that's quite, uh, that's quite nice Actually, uh, and then, you know for the yeah, uh, yeah, exactly All right. And then the third one is a Do we address the overfitting problem? Um, and then, uh, to to to look at this we we had, uh, me me and my Me and miller we we kind of we generated a bunch of these fits and then we we each looked at the patches that were generated like the fixes And we had to say, you know, is this an overfit that's or is it is it a good fit, right? And then, uh, by by so we we use this kind of that And yeah that methodology and we found that We we actually we did reduce the overfit ratio by adding properties, right? Uh, so that that was quite nice Okay, but Yeah, so that was uh, that was the the proper paper essentially, right? Um, and in the proper paper I talk about this Haskell program coverage thing, right? Uh, so we count how many times these expression is elevated during execution, right? And and then by applying Property to a counter example and in strumming the resulting program We can exactly see which expression in the module are evaluated in a failing execution, right? So the keyword here is execution Like we have to actually Run the program in order to be able to fix it, right? We have to be able to to see what happens and what what is being evaluated when we we run it, right? And for that to be possible, we have to be able to compile the program, right? We have to be able to, uh Uh, you know compile the program and the thing is that if we have a type error in the program, uh, we can't it doesn't compile, right? Uh, even though that type error might not be relevant Like it maybe it doesn't actually matter for the code to run and it doesn't have any runtime impact, but but uh But but it still fails to compile because of the type error, right? So to address this, uh, uh Me and and augustines this is actually we wrote this paper before the other paper, but It's a better story this way, right? So we wrote this a paper on a short paper on weak runtime irrelevant typing for security So the idea here is that there are some, uh, uh, libraries in Haskell like like Russo's Mac library Which allows you to model the security security properties in the type system, right? And it allows you to assign these labels to boxes and it allows you to kind of say, okay This is public information and this this is secret information And then the whole type system thing that it does is that it checks that you're not actually leaking secret data into Public data, right? So you're not leaking passwords or anything like that, right? So to show off how how that works, I'm just going to show you a demo Right away actually So here we have a kind of a stripped down implementation of the Mac library, right? We have labels We have the low label which is public information and we have a high label for secret information And then we have this new type thing that Wraps values with the label, right? Now we have this class instance that makes sure that You can't use Public data in the computation of private data, right? And then we we have some functions that are based on this, right? We have this the labeled or function We have these zip functions that check the first one checks, uh, you know checks that the the The security level of the output is is more than more than or equal to the The security levels of the inputs, right? And this other zip function it actually checks That the security levels are the same, right? So if you you give it two lists You want to be able to check that the The security level of them both are of the same and then you can return that same zip list at the end And then we try to use them in these bad functions But what the trick is that we're I mean we're breaking the law, right? Where we're not using them correctly We're trying to give we're trying to compute like public data using private data, right? And so if we run this now, uh, let me So so now we don't have any plugin. We don't have this writ plugin that we wrote Enable so we we try to run this a And compile it right and of course what happens is that you know We were modeling these security types and then we're just violating the security And we get a lot of these these type errors, right? Because I mean that's what we want right? We don't want to be able to compile the code with security in them But the motivation here is you know, we want to be able to run the code We want to be able to see what's happening even if we haven't quite figured out the security yet We want to be able to look at it and say, okay, this works We want to be able to play around with these values, right? So what i'm saying here is that We have we want to be able to print out what the results are And then kind of look at the functional correctness, right? to see, you know Does the code work and we want to be able to you know, like in the context of program repair We want to be able to run the program and see if it works So what you can do now with this plugin is that you can You enable it, right? And then we have a bunch of these rules that we write to say, okay, we can Discharge we can make it make things equal we can ignore some some constraints And we can make unlabeled things labeled and stuff like that But I mean the the main point in the context of this list is that, you know, we have this code we can run it And now it turns all the errors into warnings, right? So they're not Type errors anymore. They're just warnings They have these the main specific error messages that are relevant, right? And it's yeah, I mean the same that there's a there's a forbidden flow, right? You're using secret information in a public context But the trick is that we can still run the program. It still compiles And crucially because the the changes or the type errors will be made were runtime irrelevant You know the the program still works. It's still it's still it doesn't crash, right? And that is what we were kind of mainly looking for in these cases, right? And then the idea, of course is okay. We should be able to use this To repair these type errors, right? And then kind of have a correct program That's maybe incorrect security wise and then we can kind of use that as an oracle to repair To repair the security of our program, right? But we haven't done that yet, but So this is just a building block. So but the idea is okay Sometimes, you know, we want to be able to make use of the information that we already have But sometimes the information that you gave is kind of Places too much constraints On the compiler and it can't satisfy them And then we want to be able to kind of keep that information around, right? We want to be we want to have the security information here But we still want the developer to be able to run the program and use the information Even if even if they provided too much information, right? So this is still in the line of right, like let's make use of the information that we have, right? And also let's make sure that You know, you don't have to erase things just to make it compile is then you know that that information is gone Right? We still want it around but we want to allow running things like that All right, uh, that's uh, yeah, and that's the future plans, right? We want to We want to use this and marry this with the security fixes and then Uh, I mean we have some more ideas about it like proper and we want to where we want to take that and then With a valid whole face like how do we want to extend those? But I think I think I've covered most of of what I What I wanted to talk about in this list All right, so, uh, that's all for now and And thank you for listening to the list All right