 What I want to talk today about today is graphics and Maybe that's that's a good topic after lunch something fun I also want to thank and I rest would just run away very much for inviting me to this great conference But let's talk about graphics. So Actually when it comes to graphics animations game engines this sort of thing I'm functional programming language is really got a problem. I mean most functional programming language is at least and definitely has So the problem is that on one hand They are state-of-the-art Systems libraries frameworks which look great have a lot of features But they're typically implementing languages like C++. They're very imperative working on some mutable object graph and Here this example is from Sprite kid with some objective C code. So great output, but Maybe not the best programming experience well You could bind to that just using former functional interface from Haskell then you get the same great output But you're writing kind of very imperative low-level code in Haskell so we don't win that or you take one of the Graphical systems built up first right from scratch in Haskell nice and pure on the visual side Maybe not that great, but it's a nice purely functional data structure. You do all your stuff on so Maybe also not the optimal solution So can we have both? Can we take a state-of-the-art? framework and somehow Instead of just writing imperative code in Haskell Making such that we can have really nice functional code and and use that state-of-the-art library so in a way that this library which is kind of the rapper the The intermediary kind of takes these operations on the purely functional structure and somehow try scribes them to state manipulations on the object graph so What I want to do today. I want in three parts I want to tell you a story of how to do that So first of all, we are going to look at what's the state-of-the-art how do these libraries work? How do we use them the good ones the ones which look nice and then What's the kind of Haskell API a nice pure function purely functional API that we would like to use with them? And then the tricky bit. How do we get the two together? Okay, that's the plan and because To make it fun and concrete I'm going to talk about how I've done this in the context of Apple sprite kit 2d game engine using a little sample game as a running sample, which I'm sure you've seen before and Nevertheless the the concepts what I'm going to talk about. I mean this could apply equally well to other Graphics libraries as well and I invite you to kind of write a binding to you to your favorite one It's just when it goes to these kind of high-end graphic things then typically you have to pick something platform specific So, okay, let's write right into it and let's look at What we want from the visual from the Game engine site and how that looks like so This example program I Told you about so the running sample just so you there are some components that these pipes moving There's a background. There's some physics which handles collision stuff like that so we want to cover all this right we want all that and So how's this represent a scene like that a game scene like that is typically a represent in a hierarchical manner So we have to see in itself and then in it in a tree Arrangement we have all the different components of the scene and Then there are visual things like the sprites But they're also components like the dotted boxes here Which are just they have no visuals, but they group together other parts to provide structure And then there are some nodes in that tree which attached to them. We have actions to implement things like animations moving something Over a period of time from a to b stuff like that and then finally some of those nodes We can associate with what's called a physics body and that's all the objects with physics bodies they will be Used by the physics engine to calculate the physics of the game and they will be influenced by the physics So that's the different bits and pieces we have have to Worry about now actually it's not really a tree. It was a tree actually These scene descriptions are graphs because of the back edges, but for the remainder of the talk. I will ignore that complication All right, but because of this is based on object oriented framework Of course all the different nodes in that graph in that tree. They're all different classes of sprite kit and Well, it's object orientation. So we have Inheritance and the class hierarchy doesn't sound very hastily at this point, right? We somehow we have to do some impedance matching here and the way it's organized in in sprite kit is there's this abstract class called SK node which is Providing some properties and some methods Here I'm using swift code instead of objective seed to make it a little bit more readable And those properties and and methods they inherit by all the other classes because they're needed for any kind of Game object or graphics object here and then we have the various classes for different types of sprites and shapes and lights and whatever there is in those graphical scenes and and then finally there there's a Strange one the overall scene is actually a subclass of one of these and it does make a lot of sense I think that's bad design in that library but nevertheless if you want to represent this stuff in Haskell we somehow have to deal with it and We can't ignore it because what you have to do in sprite kit The class which you write your overall game in has to be a subclass of that scene class Because you need some game specific state you need state which is specific to the type of game you're implementing or animation you're implementing and That's not all there's also in that class hierarchy above SK node is another one It's called ns. Responder which is a cocoa thing and it's about getting events all the input key presses mouse movements all the stuff the react to We have to get at by having by overriding these methods In those that subclass now that sounds bad now Do I have to write Haskell class with subclasses and Objective C or Swift class in order to get at input That wouldn't be a very nice Haskell program I guess So we'll have to come with up with some better way of doing that. So this is the first problem the first OO imperative versus functional mismatch Now what about the second one? Well, it's a mutable object graph So let's look at the specific part of that particular game So if you look at the bird you'll see that depending on whether it falls or goes up. It changes. It's tilt, right? So how is that achieved? Well in in objective C or Swift the SK node class has a property called set rotation and The way we can implement that is that depending on the vertical velocity We change the set rotation and then the graphic library will take care of the rest well if you look at this overall scene graph then there's this sprite in here in the graph and There's a property on that one object and I have to mutate it in place Who here is programming Haskell? Does that send cleavours down your spine? Everybody else it would send cleavours down your spine to you But but where does the code come from which has performed that mutation? Well There on the scene class There's actually a method called update and that update function is called by the framework once per frame So 60 times a second this update function is called and that update function because it has to see in as it's class Why a self can access the entire scene graph can change whatever it wants called object oriented programming, right? and So it just mutates stuff. I Mean if you paid attention to keynote this morning shared mutable data structures and probably is also going to be mitre-threaded. No fun Okay, so we will have to come up with something better But it's going to get worse Because look at this I mean there these pipes and they new pipes created old ones flying out. So the scene changes over time So how's that implemented in sprite kit? Well that SK node class also has a few methods to manipulate that object graph You can't just change the references because then the graph structure would break So you have to do everything why are those provided methods? And for example, that's one to add a new trial to a node So when these pair of pipes come into the scene this method has to be called on Some sub tree which represents both pipes alright, so So what that means is in the overall object graph This pipe pair is kind of there's one node which has all the pipes underneath it And then the update function is called and it decides well We need the new one and sticks it into the graph and Another one and then at some point one of them will move up the screen has to be dropped Okay, so we are kind of in place mutating the graph while the whole thing runs That sounds even worse So we have three problems number one Somehow we have to handle this class hierarchy and this stuff in this base class and some event handlers Then we have mutating properties which affect visuals and then we have graph edits happening while everything animates So So let's look at what we want. This is that's how the Existing libraries work, but let's look at what would we want if we could just have a wish our wish API and We will keep it simple. I mean that we have because it will be very simple We won't use any complicated like state transformer higher order GA all this stuff You don't have to know about it. We just use pure functions simple algebraic data types That's it whatever functional language you use these things you have in every single functional Okay So in one way or another So how do we handle this subclassing well? Specifically in Haskell the way we are going to do it is let's look at this class hierarchy again So I said this is casein class. That's a bit strange. It's wrong. Anyway, so let's just strike that We'll find the solution, but let's look at the rest So now we've got this base class and then we got all these different types of visuals And that's really alternatives So it screams some type Okay, so in Haskell what we are going to do with define an algebraic data type saying well A note is either a plain note no visuals or a label or a sprite or a shape or whatever right and then Remember that base class that introduced some properties and methods which I use on all of them And so this is a little bit Haskell specific now in Haskell if you use record syntax to specify algebraic data types You can have some fields which occur in all of them or a subset and So what we are going to do or the stuff is in the SK note class This is going to be in every single alternative You can use it uniformly regardless of which war variant we've got and then Each except now all the other alternatives. They have some specific fields which are specific to that particular visual So for example, the label label variant has a text which is label text The sprite variant has the texture for the sprite and so on so That's our representation nice simple algebraic data type. So how about scenes? well scenes is another data type and that data type has A set of children so square brackets in Haskell's lists So it's a list of notes as children on the scene and Then other information like background color and so on and the the the last thing to note is up here These are parameters to that data type So it's a parameterized data type and those type Parameters whenever we use a particular scene for a particular application We instantiate them to a type which contains all the state which is particular to that game or whatever we are implementing So we have one for see for the scene and then the notes can have some other local information and So basically all this kind of subclassing business we resolve by using Parametric polymorphism That's much nicer Well, at least for a real-function program all right so So let's let's do that. Let's see what happens. So In this program So we We've got some state for the notes That's quite simple in these applications usually we don't have any specific state only for the pipes We maintain some random numbers in order to have them different types and then for the overall scene We've got different states the game can be in can be running or the bird can be crashing into something or game can be over and Then there are some other fields like was a key pressed What's the current score all this kind of information goes into the scene state and then we instantiate? our data this is types in aluminum has go right like and We have types in in which instantiate the scene data type with those types. We define specifically for this game and Then we Define the bird that's important as a sprite. So that's a sprite note And it's got a name. It's got a starting position and we can attach actions to notes It's one of the thing I said in the beginning. So the action be attached to the bird is Cycling through a set of textures in order to get the flapping of the wings, right? So that action is running forever during the whole game and then We put all this into an overall scene which has a particular size the background color and as it's only component for now the bird and If we if we run this then looks like that bird flies in the sky All right, so that's good start So, okay, let's look at actually putting some movement on the screen. Okay, so we have to mutate stuff and So I Said in the objective C side There's a field in the SK not class, which is the set rotation of a sprite or any of you and Well, this is a break data type. This is immutable. We can't just change it We can make a new version with a different value, but we can't change it. We can't update it. No side effects So we account for that by changing the type of the update What was the update method into a function which I call seen update function and it's defined as a pure pure? Functional type so what does that mean? It means it gets a scene as its first argument if you're not used to reading Haskell code Haskell has this weird thing called curing where if you have a binary function You put arrows between the function arguments as well. No brackets So this is a binary function gets a scene gets how much time passed since the last time update was called and returns a new scene Okay, but that's just a pure transformer. No side effects gets a scene returns a new scene purely function All right, so we can just return a new scene where the rotation has been changed and Events we handle same way we have a callback handler function and then a pure function which gets an event Description gets the current scene information of the game and produces a new one for example with the score changed or whatever and Now if you put that into action in our Example then that's the current situation. So there's gravity already. We want the bird to go up so we add some code and In particular, we add an event handler the event handler inspects the current event it got if it's a key press Well then change the state it returns a new state doesn't mutate it where the key pressed flag is set and then In the update function our update function we pattern match You don't have to read this is just weird has the pattern matching It means I take the scene state and I look at is key press true. Yes or no if it's true well, then I call this function which adds an impulse upwards velocity to the bird which I won't show you If there's no key pressed I call this other function which computes the tilt of the bird which I will show you Now that function is again a pure function It gets the sprite note as its first argument There's some pattern matching on that to extract the physics body Why the physics body because the physics body includes the velocity of the sprite and then it does some mass to Compute a new tilt a new angle from the current vertical velocity and then it doesn't in place Modify anything it returns a new note with a new tilt Okay, so all pure transformation functions and if we let that run then we see we get the Event handling and we get the tilting Right, so how about graph edits that was our third problem. Well actually same deal Let's not change from a functional program perspective All these the scene and all various notes they have Children fields which are just lists of other notes and by changing those lists by adding something to a list like that You get a new element in the scene So for example In order to implement the pipes, so we are here with the whole game, but no pipes it So how do we add the pipes? Well we extend the update function, so By adding new be new note into the graph, which has an action a custom action What does this custom action do well? It spawns pipe pair then it waits 1.5 second seconds, and then it repeats this forever So every 1.5 seconds we get new pipes and this custom action Well, that gets the pipe the parent pipe note and What it does it takes its children and colon is least Concing in Haskell it puts One new element into that list of children and that new element is a new subtree with that new pair of pipes And that's it So again, we take a note and we return a new note which is due to be different so if we look at What that looks like when we run it again, then suddenly we've got these pipes appearing. All right, so too easy It's bit too easy isn't it? It's like I've got all this mess here, and then I make a wish like it my nice interface and done So I don't know if you if you watch a movie like a fantasy movie or so and there's some Magic in it usually whenever you use magic or the protagonists in that movie use magic They have to pay a price a horrible price, right? Well, this is magic. So how about payment? Well, let's look at this in detail. So this is the theme graph Now what do I have to do to implement this update function? Well, I have to take this whole objective C structure I have to make a Haskell data structure out of it using this algebraic data type then I apply my Haskell function I Get a new version of that Haskell data structure. I have to take the whole data structure and convert it into a scene graph again Only to get one new subtree What about the old pipes? And all the rest of the tree Are we going to really copy all this from objective C land in the Haskell land and then back again? Does that sound like a good plan something you want to do 60 times a second to get a smooth animation? actually, it's worse it's worse because I Said these notes they can have animation action stuff attached to it Like if I have an animation action which moves something from here to there over say a second Then I Mean the framework has to store that information about where is the animation? How far see all this information has to be stored somewhere, right? And I told you it's implemented in C plus plus is objective C API. How do you think these people have implemented? They have just some pointer to some hidden state. Nobody can get it. It's all mutated in place. That's of course So if I take this the from the public API the information about the data structure Converted into Haskell and back again I get something which looks the same but loses all that internal state all the animations are going to start from scratch It is going to be a huge mess So what I call eager Marshall and taking the whole scene graph marching over is this is a good kind of like This is how it works in principle, but we can't implement it like that Okay, so with this hidden state. It's impossible and would be way too inefficient anyway So now we need need a good idea So what's our good idea? You've got this graph here This is our Haskell data structure a scene and that update function as a pure function So I don't marshal the whole thing. I do something which I call lazy marshaling I'll explain it in a second to to get I've seen a representation Haskell Then I apply the Haskell function to get a new scene representation and then more magic I have two scene values which I compare and compute the difference and Then only that difference is applied to the scene graph Okay, we don't make a new scene graph we figure out what changed and changed the old thing in place But this is all done in the library. The user doesn't have to know the Haskell user doesn't have to know anything But how do we do that we pay three times in verse and verse amount So how does lazy marshaling work? The idea in lazy marshaling is I take the root of the tree. I Make a Haskell representation for that. That's my record of the various fields Now I don't marshal the rest instead what I put in here is what Haskell program was called a thunk You all I assume know Haskell is a lazy language things are only evaluated if they need to be evaluated So there's a way to represent at runtime a computation which we know how to do but we don't do it yet This is called a thunk. So we put thunks in here and Thunks have pointed us to the other parts of the graph structures Which they can use to marshal the rest if ever needed if we really traverse the whole scene or would ever path We traverse on demand marshal and the same with the other bits and pieces in the stroke structure So how can we implement that in Haskell? If you believe in purely functional programming, I think now is the time to close your eyes Don't tell your kids about it So what we are going to do so there are two things here We creating these thunks using something which is called unsafe performer or so unsafe all functions in the Haskell library Which is called unsafe is basically don't use them. It's not quite true What does unsafe mean unsafe doesn't mean my program is going to crash unsafe means the Haskell compiler Doesn't know whether what you're doing is okay or not so you the programmer Has to decide You have additional obligations of reasoning here to make sure this. Okay. This is all okay. I assure you and Then there's something other weird like what's that so? I've written a library To using something which called template Haskell which the Haskell metaprogramming system to have inline Objective C code in Haskell so you can have a Haskell program with inline objective C and that's called from inside the program If you're interested in that how that works I give a talk about that at the Haskell symposium a few years back. It's on YouTube. You can look that up So I'm calling inline objective C code to do the marshalling Using this unsafe perform IO which is suspended into a song and only really execute it if you need it That's number one So now we've done this we've got our lazily marshaled structure and now we We apply the Haskell seen update function and Now the you in this particular Functional implementation only one thing changed the name of the scene was seen was changed for whatever reason Everything else stayed the same so now my library has to figure that out it has to check what changed and Then notice by the comparison only that name changed So I only update that name in place in the objective C representation nothing else is touched So how can we do that? Keep in mind These things here are fun If we evaluate them we perform the work. We want didn't want to do Any hardcore Haskell programmers here who know that you see a runtime system really well So there's another unsafe function. It's not just unsafe. It's really unsafe Pointer equality So that means like really don't use it. Why did they put that in the library if they don't want anybody to use it? Right that doesn't make any sense and it's got a strange hash mark that if you if you've done any Haskell You know when you see the hash mark you go Basically means internal low-level runtime system stuff again So what does that do it actually takes the pointers? I mean says it does what it says it does it takes the pointers the literal pointers into the Haskell heap and Compares them. Is it the same address? Yes or no Well, we had thunks in this record right a thunk in a represent heap representation is a pointer to whatever Internal representation we take that pointer if that pointer didn't change it's still the same thunk because if that thunk gets evaluated laziness doesn't just mean Only evaluated if needed it means the second thing it means if you need it twice you still only allowed to evaluate it once That's the difference between laziness and non-strictness in case you were wondering So what happens in the runtime implementation is when a thunk is evaluated it gets updated overwritten By column pilot generated code with the thing it evaluated to meaning the address is different Okay, that's how we can figure out whether the update function touch that stuff and then if it's Still the same while we don't do anything if it's not the same Then we have some inline objectives code objective C code to do the update. Okay, so now the third price we pay looks Harmless in comparison, but let me assure you it's worse So what's that? well We want to make sure remember what the goal here is we have this scene graph and whenever something changes on the Haskell site We want to just update the existing objects, but to do that on the Haskell site We have to know what those are So all note representations which haven't been created in Haskell, but which have been marshaled from objective C They contain a pointer to the objective C representation that's in this SK node type and That pointer serves as an identity as a name if you like for the thing on the objective C side So if there's a change in any of those fields, then the object pointed to by this pointer is being manipulated Sounds nice. I mean maybe SK node harmless. So what's the problem here? any ideas? Why is it really evil? No really unsafe something. Just maybe The problem is what if somebody duplicates those pointers on the Haskell site? So now we have two Haskell objects Representing same objective C thing they change in different way. So the library is still safe it Just if you do that if you duplicate it's only one of them is going to be taken the other one is regarded to be as if there's nothing Making fresh object, which may or may not be what you want, but doesn't crush your program at least But you may know that The company I work for tweak. I always working on an extension to this type system for linear types and and That would help here if we would indicate in the type for the update function that it has to be linear in the scene Then these things can only be used once all right, so that's it The Haskell sprite kit is open source and GitHub if you want to look at all the gory Haskell Coating really unsafe stuff and you want to show if you're not a Haskell person and you want to go to your Haskell colleagues and really annoy them say has this unsafe language What are you talking about? Just use my library and you'll be good But maybe you're a Haskell person or you want to be a Haskell person and you want to write some cool graphic stuff then well use that library as well and generally these ideas I think so using this lazy marshaling and Diffing is something this more general idea you may All know probably react me react and react native the frameworks from Facebook for JavaScript the similar idea there instead of manipulating the browser don't directly Manipulate a pure structure, and then there's some automatic translation of changes So this really general approach to turning messy imperative Graph mutating stuff into purely functional API Which I think is good thing to do So thank you very much questions like Yeah, so Infect the way so the question was how about other languages than objective C Infect so Haskell has a very good foreign function interface for interfacing with C and The objective C interface actually goes via the C interface and you you can use in people have used it for other languages as well They are bindings to C++ libraries, and they also go via C interface. They are bindings to Java Our other languages so all these things can kind of see API is the lingua franca for interfacing languages in a sense, so That's how it's usually done in the Haskell ecosystem Directly interfacing with C++ is much more tricky because of all the name-langling There are a lot of specific things if you go via C. It's all fairly standardized by the architecture there With respect to the inline code there are two libraries actually which do inline C for Haskell And so my objective C one does see because he's Objective C is a superset of C. There's a second one called inline C, which is also by Twigio which is similar and focused on C and There's no library for inline C++, but There's no conceptual reason why there couldn't be so I think this is again a really interesting way of doing generally Interesting way of integrating languages by having inline code of one language in the other I found that a lot better nicer to work with than other approaches. I've tried in the past Yeah, so if you go to this github repository in the read me there's a link to a paper Which describes these techniques Pardon Okay, and well, there's all kinds of libraries for game programming on a package package is the Haskell package system Most of them are in a state, which is not really very usable They're all very expert most of them try to do the what I said kind of do it from the ground up and It's not that you can't write a game engine from the ground up in Haskell or bind to something or so But it's just a lot of work and needs a lot of specialized expertise So this stuff usually doesn't go very far because people do it for a few months. They lose interest to something else The this one you can use GitHub on github. They are also two samples. So this one and another sample game small one we did with it Just as a demo So that's perfectly usable you can compile it in a standalone Mac application and nobody has to know that Yeah, but that's what I'm saying is basically Really in this space where there's so much domain expertise in implementing these frameworks doing everything from scratch In your favorite language may not be the best option because it's just too much work to to implement something like that