 Bon dia. Just let me maximize so you can see all of my slides. All right, so I'm here today to talk to you about flying spaghetti monster. But before that, I wanted to just comment on how much I'm enjoying the conference so far. So one of the things that I think I've learned to appreciate over the past day and a half is how much care has gone into how the conference is put together. It's not just that the small pieces make sense, although every talk has been excellent. It's about how everything adds up to a big coherent story. And as I've been watching the talks, I've started to get a little bit obsessed with almost trying to work out which character each speaker is playing. Because I think that the organizers have been very careful to select different kinds of talks. So we've had the captain of Chaos Engineering on before. We've had Master of Monitoring, the serverless hero, the king of blockchain. And I guess in that vein, I'm here as the hipster of functional programming. So that is the angle I'm taking because I passionately believe that of the many problems that we have in software today, better languages and better techniques for modelling our problems is a really important tool that we have to solve our problems. Not the only one, but a really important and a really lovely one. So who is the flying spaghetti monster? Well, this is a meme from 2005. So there was some controversy in the United States education system about the teaching of intelligent design, which is the concept that nature, animals' plants as we see them, were designed by a single kind of omnipotent intelligence. And that's the way that we can account for, explain and understand the enormous complexity that we observe today. However, the person who came up with the flying spaghetti monster didn't quite believe that was the right way to go things. And so he created this meme and circulated it. So this particular picture is an adaptation of a very famous painting in Rome, the ceiling of the Sistine Chapel. So the place of the flying spaghetti monster, the deity, was originally taken by an old white man in a beard, the Christian god. I think it's a really excellent picture and I'm going to come back to it later on because I think it has a lot to actually tell us about managing of distributed systems. So the language that my talk depends on is called Idris, named after a cartoon dragon if anyone is into origin stories. It's developed in St Andrews, a university in the United Kingdom by a guy called Edwin Brady. I think Idris is a really elegant and beautiful programming language. So it's in the vein of Haskell, Scala and F-Sharp. So it's a statically typed functional programming language. So some of its characteristics are that the compiler checks the types. So that's not unique to Idris, but obviously if you try to put a string in a function that expects an integer, then Idris will tell you no. And it will tell you no at the point you compile the program, not later on. So that's a statically typed bit. But also what makes a functional language is that functions are values. They're first class citizens in our programming. We can do things with them. We can calculate with them. Functions can be passed to other functions. They can be returned from functions. So functions is a first class citizen that explains the functional programming language bit. But the bit that makes Idris, I think, very interesting is the third point. And that's not very common in programming languages and certainly not very common as a first class built-in feature of a programming language, which is that types themselves a value. So not just an instance of a string or an instance of a number, but a concept of a string, a concept of a number. Our first class citizens in Idris. So just like functions can be passed around in many languages like JavaScript, types can be passed around and calculated and used in expressions in Idris. And that gives us some power that I don't think our existing generation of languages gives us. So if you want to get into Idris, please consider buying this book. It's by Edwin. He's the creator of Idris and also an exceptionally talented educator of programming. So like all great programming books, this manages to capture the detail. So it manages to give you all you need to know about Idris, how it works. It also manages to tell you about the practical elements of the language. So it helps you set up your editor and give you a workflow for actually working with Idris. But the thing that I really enjoy about the book is that it manages to go at that higher conceptual level. It manages to explain why development with types is an interesting way of doing things and move beyond the kind of guardrail concept we might originally have taught, where the compiler is just an annoying person who yells at us every now and again and actually becomes our friend and helps us express the things that we need to express. So please consider getting this book and quite a few of the examples in this talk have been adapted from Edwin's examples. So this is a really simple Idris function. Just to give you a little bit of the flavor. So it looks a lot like Haskell, F-sharp, Scala. So if you see this is defining a concatenation function. So the thing you need to know to understand this code is that Vector is the name for an Idris type that's like a list except that the length of the list is part of the type itself. I'll explain how that's important. So concatenate colon is of type. Vector of length N and type A. And it also takes another vector of length M and type A and returns a vector of length M plus N and type A. So it's that final bit that's quite interesting. So in Haskell you would be able to say, you know what, you're only allowed to add two lists that have the same things in them. So in Haskell you're not allowed to put together a list of strings and a list of integers because it's statically typed and it says that won't work. It has to be the same type. But in Idris it can also infer a lot more about the combination of those two lists so it can tell you that they're the right length. And that means that if you were to mess up the implementation of concatenate, so for example if you were to accidentally throw away one list altogether and just return the other list, this would not compile, which is quite interesting. That's a compile time guarantee. And Idris can do this because under the hood it's quite similar to a theorem proven. So it knows how to inductively go through recursive calls, work out how things join together in your code, and tell you whether or not you've got this right. So the fact that we've got first-class types, i.e. types that we can do calculations with, because you've got that expression, that plus in the length, that lets us do extra things and types tell us more about our program than they could in a conventional, statically typed language. So I think I should stress though, as I'm a functional programming hipster, Idris is a wonder from the future. So it exists now, it's stable, there are great docs, you can use it, there's a book on it, but it's not as far as I know used in production. It's a research language at the moment. So an appropriate response to this talk is to say you might want to learn more about Idris. You want to go and research it or join the community, join IRC. It might be that you want to contribute to the language or the ecosystem. There's a lot of space for people to get involved. It might be that you take your own programming language and look how the ideas that Idris has can be applied to that programming language. But an appropriate response is not to go in on Monday and rewrite your banking application in Idris, unless your aim is to bring down capitalism from the inside or something like that. But if your aim is to actually be a responsible chooser of technology, you need to think about when to apply it. So Idris doesn't really have the ecosystem to be widely deployed yet. But I think it already has a lot to learn, has a lot to teach us. And the reason for that is that there is a big problem today that Idris I think is the solution to. So there is actually a use case already. It's just that we don't yet have really good answers. So what is the problem? Well, microservices. You could say, so I'm a big advocate of microservices in so far as it's an architectural trade-off that you are deliberately making and that serves your needs. I'm a terrible opponent of microservices for its own sake as a buzzword because someone at a conference said it was, I don't know, good. So microservices are neither good nor bad. I'm very clear on that. What microservices are is a trade-off that may or may not be appropriate. So I know we've heard a lot of great talk about microservices from other speakers and I won't be disagreeing. I'll be building on top of that. So microservices, characterized really simply, are that they do things in small pieces. So I'm sorry I can't give them a formal definition. Unlike, say, REST, there's no thesis on microservices that tells us the exact definition. So I'm going to try to sketch out what I think the characteristics are of a microservice architecture. So it's all about small pieces. That's the micro obvious, right? Data can be processed a little bit at a time. So if you saw the talk yesterday where Kafka streams to broadcast events and then there are multiple listeners, well, that's kind of an example of how microservices can help because the processing of those events was broken down into individual different parts of the system that needed to do different things with it. That's great. Different things live apart. That's great design. Microservices also mean that code can be changed a little bit at a time. So we can independently deploy code. We can upgrade a version of our programming language or library on one service without having upgraded it on all services together. So we also can change code and functionality a little bit at a time. And we can also understand the system a little bit at a time. And this has a caveat, as the talk before is very clearly demonstrated. But theoretically, if you have a well-designed microservice with good boundaries, you know exactly what it does really quickly and that's great for you to get started. But microservices are a trade-off, not just a collection of good stuff. So we gain freedom at the cost of eventual consistency. So no longer can we guarantee that either the data was processed or it wasn't. No longer can we guarantee that all our system uses the same version of Java. No longer can we guarantee that everyone understands everything, although you couldn't ever guarantee that, but compared to a monolithic architecture. So we've got a price to pay. It's just like, I guess, if you have a data-intensive application and in order to get better throughput, you update your database with different parts of the process at different times outside of a transaction. That's great. It gives you freedom. It lets you maybe scale better. It lets you do all sorts of different things with your architecture that you wouldn't otherwise be able to do. But it also puts the burden on you to handle the fact that you're doing things a little bit at a time. Because what happens if things go wrong in the middle? So this is a quote that I think is great programming quote in general. I really like our programming quote culture. So Greenspun said about programming in general, that any sufficiently complicated C or Fortran program contains an ad hoc, informally specified, bug-ridden, slow implementation of common list. So you could just see this as bike shedding. You could be just, oh, list is great, C is bad. That's not the point of this quote. The point is that if you have a system that starts out simple and that you keep adding little bits of complexity at a time, if you don't find a way to break through that and think about the system at a new level of abstraction that works for you, then it's going to get out of hand. So this is a kind of a tragedy that Greenspan is observing, where we've added more and more complexity, but we've missed out on some kind of unifying abstraction. And I think this characteristic that Greenspan sees in programming is, of course, present in microservices. So my tenth rule of microservices is that any sufficiently complicated microservice architecture contains an ad hoc, informally specified, bug-ridden, slow implementation of a distributed protocol. And distributed protocols are hard. People study these things in universities for years. There's all sorts of difficulty understanding distributed protocols. If you've tried to kind of read papers on PAXOS, you'll see it's very hard, and it's so hard to understand them that writing raft, which is a new distributed protocol that is easier to understand, was a very worthwhile intellectual achievement. So I think that in microservices, we also have the problem that Greenspan recognized that we build up little bits of complexity, but don't necessarily have good ways of thinking about the whole that we develop for ourselves. And in the case of microservices, it's particularly tricky because that distributed protocol is bespoke. It's for us only. That combination of how our services talk together isn't something you can find a paper on. The request comes in, you add the company to the database, and then separately you process the request to add the admin user for that company who's your new client. That protocol, that eventual consistency, it happens completely uniquely to you. So you better understand it because no one else is going to understand it for you. So how can we overcome this problem? I've set up the problem from today. What's the solution? My friend FSM and wonder if he can provide us a tool that can solve this. And there is actually a quite good conceptual tool for this kind of thing of managing protocols called finite state machines. So they even look something like the Flying Spaghetti Monster. I only noticed this later on after I composed the talk, but you've kind of got finite state machines made out of basically meatballs and spaghetti. So the meatballs are the states. So this is a DAW protocol. Very simple, for an example only. So a DAW can be either in the closed meatball state or the open meatball state. And there are fixed things you can do to get from one state to another. So you can't just change things willy-nilly. So the strength of finite state machines is that they are a deliberately limited model of computation. They are not sure and complete. And that's not a bad thing, that's a strength because it means that it's easier to understand what is going to happen with a finite state machine. You can reason about it more easily. So the more limited the model, the easier it is for someone coming to it. So finite state machines are often used to model protocols. Obviously you could come up with a protocol that was so complicated that it needed a Turing-complete description. But ideally don't do that. Usually when you're doing that, you're making mistakes. So TCP has a finite state machine description, for example. So when you've got these protocols in your system, often you'll deal with them in an implicit way. So often you will have code that is implementing transitions from state to state, but you don't necessarily have the tools or the culture within your organization to model them explicitly. It's just kind of implicit there. So this is a bit of code that implements that finite state machine, or at least a session with that finite state machine, in kind of pseudo-java script. So a door program says first of all, we'll open the door and then we'll knock on it and then we'll close it. You knock before you open, right? You don't knock after you've already opened the door. So it's not immediately obvious from looking at this code that the things are being done in the wrong order. And in an ordinary type system, or certainly in a language without a type system, you wouldn't necessarily easily find out about this, especially if we're dealing with a synchrony or timing. So maybe the fact you're doing things in the wrong order is obvious and you catch it in your testing, or maybe it happens only after sufficient load. So in Idris, we do have a way of making a concrete type that represents a finite state machine. So this code here represents the door session, but as an Idris type that can be checked by the compiler. So if you open a door, that's the same thing as a door session that starts from the closed state and ends in the open state. So that's the definition of that edge in text. If you close a door, it's the reverse. It starts in the open state and ends in the closed state. To knock on the door starts in closed and ends in closed. That final operator at the bottom there, that kind of greater than greater than equals, that's called bind, just a special name that Idris has special convenience syntax for. But that's the most interesting one of the lot because that says that if you take two sessions, one that starts in a beginning state and goes to a middle state and another one that goes from a middle state to an end state that you can join them together into one long session, and Idris will check that they join up. So you can't join together a path through the finite state machine that jumps around. So this is really handy because if I wrote this code in Idris with those types, it would not compile. So I've knocked on an open door. I guess that's a saying. It's supposed to be a good thing. But in our protocol, it's not a good thing. It's an invalid state. It's a mistake. But Idris would actually not compile if I did this because we've got our finite state machine that allows us to compile that to knock relies on the door being closed. So it can check it at compile time and only permits this code to work. So this is a kind of checking. I know this is a trivial example, but this is a kind of checking that conventional type systems make hard to do. It's not impossible, necessarily, say, in Scala. There's ways you can do it. But in Idris, it's natural and it's beautiful. And if you want to learn about the concept, I think it's best to do in Idris because that's what the language is designed to do. It's an advanced, expressive type checking where the type checker is your buddy. So we've got our finite state machine where we imagine our microservice is the flying spaghetti monster. It's got meatballs and transitions. It's got control. We know what's going on. And the compiler tells it if we've messed it up. But the really great detail of this painting and also the one that was actually on the ceiling of the Sistine Chapel is the wonderful detail that the two characters are not actually touching. There's that wonderful bit of blank space, of negative space between them. So there's not a perfect understanding. There's not a perfect communication. There's room for things to go wrong. There's room for the deity and humanity to misunderstand each other. And that's definitely true in microservices. There's all sorts of fallibility, and the speakers earlier today have shown all sorts of different things that can go wrong in a microservice architecture far more eloquently than I can. So I hope that you're already accepting that as true. But what does that mean for our finite state machines? If they're a protocol that's a conversation between two or more systems, well, what conversations have two sides, change and choice, and things can happen from more than one side. We can't assume that just because our code says that we open the door that in the real world, if it's an Internet of Things enabled door, that it will always do it. Perhaps it'll run out of batteries. There's something in front of it. Things can go wrong in all sorts of ways. And if we want our system to be truly resilient, we need to handle that. So there's a kind of a Tolstoy principle that I think is a really good description of how this works. So the first line of Anna Karenina, Tolstoy's great novel, is that happy families are all alike and every unhappy family is unhappy in its own way. So he's basically talking about error conditions here. So a happy request through your system is... there might be a couple of variations of boy meets girl or girl meets boy or boy meets boy, but it's fundamentally an easy story to understand. If things don't go wrong, there's not much dramatic tension. Things just go right through. So when you're seeing, like I guess the request pods of tracing things through many services from the previous talk, if you were allowed to assume that nothing ever fails, it would be a lot easier to understand. It can be unhappy in its own way and there can be compounding errors and all sorts of intermediate partial successes, partial failures that can happen to you. So for example, in the example I gave where you create a company record and a user for that company, well what happens if one of those succeeds and one fails? You've got a kind of a dangling record in either case. So perhaps your system says well, if that happens, I'm going to roll it back. But what if the rollback fails? So there's all sorts of compounding error conditions that can happen and it's very hard to understand but I'd say that when you're joining a team it's these partial failures that are actually often not really captured in any kind of design document or even in the culture of the team. Someone arrives at the new team, you might stand them in front of a whiteboard and tell them the happy story of what happens when Romeo and Juliet wait until they were 18, got married and he got a job in a bank and she got a job in an insurance company and eventually they retired and they had some kind of problem with pensions and that's the kind of story you get told when you join a team with a microservice architecture but especially if you've got very good automation, failures are often unique right? Failures happen because we didn't automate that situation. So it's very hard to tell the story of failures. So finite state machines are deterministic and they model things going well but how can we handle it if we have things that don't go well if we want to apply them to microservice architectures? Well you can actually in these protocols specify fallibility which is getting a little bit this is kind of one level more advanced so this is almost the same protocol so close works almost the same but you can see I've changed the second part of the state if you look at the close line for example that the opening state is open that's the initial state but the close thing is const close so what is that? So this is a functional programming age const closed is a function we always end in the close state so we're representing the end not as a constant but as a function and the really nice thing about that is if you look at open the opening state of open is in the close state but what happens after that is a function depending on the result idris syntax for lambda function if the result is okay we end up in the open state if the result is not okay we end up in the close state because the door jammed and never opened this is not a run time check this is a compile time check this is happening in the types so idris will make sure you've plugged together your operations not only in the happy path but also potentially the unhappy path so this is a door programming idris with enforced error checking so when I try to open if it's okay that's fine we keep flowing through on the happy story we get to close but if there's a failure we're already enclosed and that's how idris can promise that we start in a closed state and end in a closed state so it's kind of a bit like checked exceptions in Java but a lot cleaner and a lot nicer so it actually tells you not only if you're going through the correct path but that you've taken into account all the wrong turnings your system might make so hooray the flying spaghetti monster has saved us right but one of the key benefits of a microservice architecture is evolution just like the flying spaghetti monster was supposed to stand for an opposition to the concept of intelligent design if the only way to have a resilient understandable microservice architecture is to have a magical architect dream up the whole thing and then hand that out to everyone on these sacred parchment well that wouldn't be very useful and in that case I'd probably recommend a monolithic architecture because when that architect is inevitably proved very wrong the system will be more easily junked and we can start again properly so we need to be able to evolve and respond to change so we can't any system of communication about how systems and microservices fit together has to take into account evolution and distribution of the protocols which will themselves evolve over time so this is where I want to rely on another cool feature of Idris that is not in many other programming languages called type providers they're originally stolen from F sharp the .NET functional programming language but I would argue that they're a little bit nicer in Idris because in Idris as we saw earlier types of values and so computing types is a more natural thing to do it's a lot easier to think about so type providers what do they do they hook into the compiler so they're like everything else we've seen today they occur at compile time they generate types from external sources so you don't have to so if for example you've got code in F sharp or Idris that is manipulating some XML if you have a schema document that describes what field is in the XML then Idris can generate the stub types for you you don't have to like other do a code generation exercise or create all these dumb types that have a field for each field in the XML Idris can understand external schemas and turn them into types for you so long as someone writes a query at type provider and that's great for reducing boilerplate and since a type provider doesn't have to be just something locally it can be a URL it can be something that you point your whole system to it gives you a really good distribution mechanism if indeed you have a way of capturing your protocols and you want multiple systems to use them so one way of doing this and this is just a proof of concept is that you could express your flying spaghetti monster using them protocol as a simple text file so this is just a text file format I made up with spaces separating the states so what it says is that the insert coin transition takes you from waiting to paid this is a vending machine you insert a coin it's not waiting you've now paid if you return you get your money back you go back into waiting if you select a chocolate it might go into selected there or it might fail if it fails you're still stuck in the paid state so that's the error condition that in this example we need to be able to handle and if we vend it just goes from selected it gives you the chocolate and there you go so if we can capture our protocol as a text file as a schema and if we can turn that into a type that makes it a lot easier to do what I was showing earlier so rather than every system having to come up with a way of writing that state machine itself especially if these cooperating systems all have to work off the same state machine the same protocol you can actually use the type provider so that first non-comment line starting with percentage provide is the type provider in action so what it says is go into vendingmachine.txt get out the contents of the file and magic up some types for me that describe that finite state machine so I don't have to so people can reuse this code potentially so now if I write my vending machine program saying it goes from waiting to waiting Idris will not only check that I'm doing the right transitions in the right order it will also check that I accounted for the failure condition so if I for example change this example and try to vend before putting any money in it would fail at compile time now I have to have slightly different syntax here you can see that I'm using strings with do because I don't have in Idris literals for specifying the transition so I'm jittering in those types but this is all happening at compile time this is all basically theorem proved to add up and if it doesn't add up you get an answer quickly so in conclusion I am making the strong argument that we are building systems that we don't fully understand I don't think I'm the only speaker today to make that argument right we've got a genuine problem with building systems that maybe literally no human on earth understands the ways in which it can go wrong and I don't think that's an edge case I think that's the common case in enterprises and I think we really need better ways to think about systems with many fallible moving parts so distributed tracing that we saw earlier that's a great example of a tool and people responding to the challenge of a big system with many moving parts but I think that that tells you kind of what happened but I think you also need a better way to represent what you wanted to happen so that you can take things into account this is a problem that I don't think we can just pretend isn't there so it's not that this complexity and this problem is staying the same over time I think this is genuinely getting worse at a rapid rate now we've got AWS and Azure and Google Cloud we're finding new ways to add more complexity to our systems we're adding more services they're interacting in ever more complex ways so actually the complexity we need to handle isn't steady it's actually increasing and if we don't up our game and how to understand our systems then we're going to get left behind expressive type systems I believe can help but as I said Idris is a wonder from the future not a recommendation from the present so that's why I've got the question mark there I think there's a great argument to say that work on using types to check protocols is going to be a really important part of distributed systems I certainly think that the idea that types do not help you once the network gets involved is false right I think that what we've seen with IO in effects is that actually types are most useful in the places where it looked like they were hardest to apply because that's where our reasoning is really challenged so I think they can help and if you want to be part of the future well you've got to take a little bit of a risk right if you want to look more into this there's all sorts of references on Idris there's Idris homepage, there's the book there's documentation on how type providers work in Idris if you want to go deeper in there's also academic work on how to handle protocols that involve many actors so multi-party asynchronous session types it's a bit of a mouthful that's the idea that you can have a single protocol that you define that spans your system and then project the roles different actors play in it so you have a single consistent protocol and then you have the concept of what each actor needs to do in it so that helps you make things add up but also if you want to know how the code that I've shown works I know I said I haven't put it in production but there is a working proof concept called Flying Spaghetti Monster on GitHub so if you want to check that out if you want to see what Idris is like that's probably a good example of a short and simple program as I say I'm not an expert I'm an enthusiast if you want to come to the the meet up that ThoughtWorks have in our office tomorrow evening then you'll be able to hang out and ask me about it if you want that's all I have so may the Flying Spaghetti Monster be with you Thanks Chris, we have a few questions coming in a lot of people are I think trying to find the line between how productionised this is so I mean starting with a simple one have you used this model to validate a real production distributed system no I haven't so I've mostly worked on proof of concept examples the actual state of productionising I think is that the language itself is quite stable so there's a version 1.0 whatever that means that goes along with the book and it's quite well tested and I think quite well engineered, good editing modes, good documentation but the ecosystem around it doesn't exist yet so if you want to contribute to that ecosystem great but otherwise you might need to hand roll a lot of stuff I know I linked that and it might be a similar answer somebody said that they see the main problem with Idris as exponential compile time and a huge memory requirement which is that's fair I think it's fair but I think that I guess like Scala is the example people probably have more experience with of a compiler that does a heck of a lot more work than an ordinary compiler and so therefore it takes a long time to do it I would still recommend given that I quite like microservices for the solution that I'm talking about still recommend making things out of small discrete applications so I think the solution is to not stress the compiler too much by focusing on smaller applications that add up to big ones okay somebody asked how does Idris compare to TLA plus which I've never heard of before but you probably know about it Idris is roughly equivalent to GADTs in Haskell if that's the thing that the question is familiar with like generalized algebraic data types but I guess the concept is that Idris is a bit like Agda in that kind of types dependent types if you will built in a question after you've compiled your vending machine protocol into several consumers how can we then ensure that the tweaks and extensions of the protocol won't break existing clients so there is a whole there's a third layer that I didn't show in the talk for brevity which is the point at which you actually execute the model against for example a network or a model of machine or whatever so that falls prey to the same problem that all formal verification falls for which is that if the semantics of your machine don't match the ones that you modeled then it's not going to go well so if in your model you assume the door could not jam then the theorem prover is going to say that's fine doors never jam, never ever so that layer relies on you mapping the individual actions to real things that your device does if you mess that up then your model won't be correct and therefore the conclusions for the model won't be correct but it's a much simpler job to model those individual actions one at a time so we'll probably reuse that than to just work out what the system as a whole might do and I guess with a slight troll face on can you not just use tests like do we need all this can we not just test our code better to avoid it so testing is a great thing and as an agile program I'm very keen on test driven development and using tests at all levels I think that types and tests in a way answer different questions but I think that complementary rather than opposing so what tests do is they provide you with examples, positive examples that your system works correctly so that's great and especially great one that you know the system works for those use cases and two it helps you map from the conceptual model of what you thought your system should do or what your customers thought your system should do to how the system actually works so these examples, those backwards E exists are really important but what types are about excluding negative conditions so types are about taking things that should never happen and trying to prove that they never happen so your positive examples and your kind of negative exclusions I think I think work together and I would not say that just because for example you have a really good type system I would not say that means you don't need any tests I think you still do because types just tell you that your system is consistent it doesn't tell you that it's the right system for what your customers expect excellent, it's good to leave it on thank you very much