 I'm sure, yeah, so we're recording progress. All right, so thank you. So I'm from Toronto, and I've been doing mostly kind of web dev and startups, and mostly it's been my company, a few other co-founders, and so we've spent the last several years just in closure, which has been nice because we get to decide what we use, and so we stick with closure. But before that, it was like Ruby JavaScript, like the very beginnings of Node, but I'm not a lot of that. Those mostly early Ruby and JavaScript, mostly closure. And so today, I'll be talking about data-oriented programming and let me kick up my slideshow here, and let's share a screen. There we go, everyone can see, yeah? And right, so data-oriented programming is a term that it gets kind of thrown about in the closure community. And as these sort of like new fangled ideas as they go, it's not very well-defined. It's something where if you asked a bunch of closure developers, they'd give you a bunch of different answers for it, but all of them would probably know it when they saw it. And so this talk today is my attempt at kind of taking the zeitgeist of the community and the practices and the way that we throw around these terms and trying to explain it to a crowd that is fortunately not very closure heavy. So this is good. I'm looking forward to feedback or questions about how can you possibly be doing this or how does this work? Or how can you not be using types or any of that kind of stuff? I definitely want some pushback because I'm gonna try to describe these various ideas in an unbiased way or at least kind of just like be just neutral about it. I'm not trying to sell you on these practices, but because I'm in a bubble. I know that I'm likely gonna come off as very biased. And so I'm looking for some discussion after and questions. And again, this will try to, this will be a bit reductionist probably. And I kind of think of it as an exercise in like programming anthropology. It's like, what are these, you know, here we see the closure programmers of the natural habitat doing these weird things that maybe no one else is doing or maybe other people are doing, we'll see. And so this term data-oriented programming and some people might say, well, is it data-oriented programming or is it data-driven programming? Cause both of those terms get mentioned quite a lot in the closure community. And it's actually today I'm gonna cover both because they used to be kind of used somewhat interchangeably, but I think, I gave a talk like two years ago when I was using them interchangeably, but there's a bit more clarity now and people kind of use them to refer to slightly different things. And so we're gonna do a talk of two parts where we're gonna talk about data-orientedness and data-drivenness. And we'll see how they're very related, but also kind of used in slightly different ways. And so the plan is gonna be that I'm gonna talk about one of them for 20 minutes and then do questions Q&A and then maybe take a break and then we'll talk about the other one, data-driven programming and then follow that up with some questions and Q&A. And so I'm gonna kind of get going by just trying to give you the two definitions so that you have something to anchor to and then we'll dive in deeper. So both of these let's talk about kind of data-oriented programming. And I would kind of in a sentence kind of explain it as the choice of using generic data structures and preferring them to objects and typed records. So using kind of the most generic plain data representations as we can in preference to objects and type records even if we have them available and using them throughout the program. So another way of maybe saying this and that might be kind of semi-universal would be imagine if we just passed JSON objects around everywhere in our programs. So we didn't use fancy classes or types. We just had kind of JSON. Now in closure, it's not actually JSON and it's not like JSON as a string but I mean like the stuff that you could put in JSON in terms of arrays, keys and values, numbers, strings, that kind of stuff and just trying to pass off like these simple objects. JavaScript calls them objects, it's terrible. They're just hash maps and arrays. So passing around just these simple things around everywhere in our programs. And then data-driven programming is the term that closure devs tend to use to refer to also what I kind of think is a choice or preference where again we prefer using kind of these generic data structures instead of writing code in situations that we can get away with that. And that might be interesting like how can we just use data structures instead of code and we'll kind of see that. But there are cases where we can basically do some metaprogramming, put stuff into just data structures and then derive functionality from it and data-driven programming is like choosing that over writing macros or functions or objects or whatever. And it's particularly used for like creating DSLs. So to mean specific like little sub languages and my kind of quick universal way of describing that is what if we used JSON as the DSL to everything? Like what if we needed to write some regex? What if it was in JSON? What if we needed to talk to and from our database? So instead of writing select statements and SQL what if we represented them as JSON? And again, not really JSON but it's kind of gets the idea across of like what if we had this we use these like very generic fundamental simple data structures as the primary interface and that it's data it doesn't have any behavior. It's just plain data as the interface of these things. So those are the two ideas we'll be going a little bit deeper into. I'll mention that there is a term that's also like in the video game C plus C community they have a term also called data oriented design or data oriented programming design and it's a little it's completely different in there and from my understanding it's more about how like they it's the architect or applications very fundamentally around like where's the data coming from? And like, you know, how do we get data from the graphics card to the memory caches? Like it's purely like, you know optimizing around the data flows of piping pixels around and so that's not what we're not what we're talking about. It's a different kind of data flow. Cool, so those are gonna be our two topics. Any questions before I dive into the first one? I do, I have a question. Sure. So when you're talking about generic data structures and you're saying, okay, it's JSON but also like it's not really JSON it's just the stuff you can put into JSON. Is that, what is the difference between that and what, you know, Ruby and JavaScript are doing? You know, just or even closure, I guess because it isn't that a dynamic language as well. Yeah, so I guess I say, you know in Ruby you have hash maps but you also have objects and you have classes and objects and 90% of Ruby programmers will create classes and objects for everything. If they need to create a user they will create a user class and put data into a user class even though they could just make a hash map with a, you know a name and then email as keys. I see. So that, so in that case, so you have the generic and I'll kind of, I'll show the closure ones but it's kind of, you know in closure it's possible to create classes and like objects proper, proper like Java objects and they also have a type called a record which isn't the Java records because it was closure had them before Java had them but they're basically implement kind of like, you know structs but it's the choice of not using those and trying to be kind of about a lower common denominator. And yeah, and so we'll go we'll go in deeper into that but it's a good question. Okay, okay, cool. Thank you. Can I ask how much of this is about sort of closure and closure culture versus LISP? Yeah, I think it could very well be a LISP thing as well. The interesting bit, I guess even in closure culture is that like I think there are some distinctions from, you know, my, you know I only see it from having really only joined the LISP community via closure but even at the beginning of closure the practices like, you know closure is like 10 or 10 to 12 years old and the way that people were writing closure code 10 to 12 years ago was very different and it was very macro heavy and or very like protocols object heavy and it kind of migrated towards these things that I'm gonna be talking about today. And so my understanding is that like, yes I guess you could be doing this in other languages and I will kind of try to kind of talk about like, okay what does a language need to be able to do this kind of stuff well? And so yeah, it doesn't necessarily have to be closure it's just something that we talk about in closure and I think there are features of closure that make this be both of these kinds of practices very easy and that's why they're done. But yeah, things like, you know I think people, you know closure programmers use the least amount of macros of LISP program is my understanding is that like common whispers just like love macros and whereas there's like a strong cultural vibe of like you know, there's the joke of like what is the first rule of macro club is don't write macros and you try to like I have over the last 10 years of full-time closure development have written maybe three macros in my work. So I have one more question. So I was wondering about when you say generic data structure is generic doing any work there? Like are you talking about, you know generics with capital G or do you just mean like the primitive types? Yeah, I mean, I think the capital generic does is kind of related, but no it is kind of more of just like the primitive types. In closure, we would just call it plain data but I just feel like it's like, what does plain data mean? But yeah, I will definitely go my first thing actually the next like as we go into trying to describe like what do I mean by generics or generic data structures? Anything else before we continue? Cool, yeah, let's dive in. I'm sure there will be lots of new things to chat about. Okay, so data oriented programming. Closure as a language, you know every language makes a lot of choices around you know, things it does and things it doesn't do things that makes it easy and things that makes hard and of all the various choices that closure made there are kind of two that are relevant to the conversation today and they're quite, you know, quite strong choices given the ecosystem. And one of them is like, you know this choice of like preferring functions to objects and being a functional programming language. And the other one that also is quite divisive is, you know, the dynamic typing versus static typing. Not having to type like, you know it's writing a whole bunch of types and typing out every type of object you create instead. Yeah, dynamic typing is a very different experience. And it's kind of particularly interesting because, you know, closure started off as being on the JVM, right? It's hosted by the JVM and Java is kind of the opposite of these. Java is one of the kind of classically heavily object oriented languages and it's strictly typed. And so, and closure can interoperate with Java. And so we can do all the like in closure you can do all those things. And I kind of, as I mentioned data-oriented programming is about, you know we can do these things, but we're choosing not to and we're choosing to kind of, yeah default to using kind of these generic data sculptures. And so kind of let's kind of clarify what I mean. And I've been using generic data structures which is kind of just my word for this right now. In closure, we would, you know a lot of people just talk about plain data, plain data but, you know, there's lots of different kinds of data. Everything is data. And so, like, you know JSON is a string that you have to parse. So I mean like what you would get if you parsed some JSON like the in memory kind of representation. Is am I okay? Cause I had like an internet warning. Is it like a connection warning? Things are okay, so. Yeah, you broke up for a second, but. Okay, I'm back. Okay. So when I talking about like generic data structures it is this idea of kind of using the very fundamental primitives and using them throughout the application. And in closure's case, if you use kind of an example, you know that's things like hash maps, lists and array and vectors kind of like arrays. Closure that also includes sets, all various kinds of numbers, you know, the integers floats with various ways of representing precision. Closure also has ratios, UUIDs, instance strings, of course. And in closure's case, there's keywords as well as namespace keywords. So you can have keywords but you can also namespace those keywords. And keywords in some languages called symbols but closure has both keywords and symbols. Symbols are kind of what are, you know the, you know, function names and references to variable names. Those are symbols and keywords are these things with columns in front of them. And so part of the idea of data oriented programming is let's just pass this kind of stuff around as we're seeing kind of just like these user record objects. They're not necessarily typed. Well, they aren't typed, but like the, we would just kind of say, well, this is just a hash map with these keys and values. And in theory, you know, we could add or remove keys and values to it willy-nilly in closure. You can put anything as a key. It doesn't have to be a keyword like I've done here. You could have strings, you can have numbers, but let's just use these fundamental kind of primitives rather than creating kind of strict type various kinds of objects and lean into kind of using these primitives and have them everywhere in our application. And what I mean everywhere, it's kind of just like default to using it everywhere. So for example, for HTTP stack, there's a spec called ring enclosure and it takes whatever HTTP request you get and represents it as an object. Well, as this hash map, which is this like plain data, then you, you know, that gets passed into whatever function that, you know, your implementation of what your web server is doing. And then you as the programmer, also then just return plain data to the, you know, the web server that's implementing the HTTP stuff and that gets converted to an actual response. Now, if you've done Ruby before, this looks kind of normal. In JavaScript, it's almost like this, except, you know, a lot of their things you have to like modify it, you receive a request object. Sometimes you have to call methods on it to extract stuff out of it. And if you wanna respond to something, you have to call methods on a response object and do some stuff, whereas in closure, they've kind of chosen to just have this kind of abstraction, you just get this object in, do some stuff and you hash map in, do some stuff and return a hash map out. And it's the same idea for like databases. If you wanna, most of the database libraries just have you pass in something like this and spit. And if you query stuff, you get an array of things like this out of it. And so it's this practice generally of just like having these kinds of primitive hash maps that aren't typed. It's not a database response object. It is just the same thing. It's just a hash map with a bunch of keys and values. Does that kind of clarify what I'm talking about when I'm talking about plain data? Do you have any more kind of questions around what that is versus like not being plain data? So is the idea something like, like we're just going DTO all the way? What's DTO? Data transfer object. So these... Yeah, I don't, I'm not familiar with that term, but I guess if... So these would be these objects that are like, I guess composed of these primitives that you put them sort of at the edges of programs. So you send them out over HTTP or send them to and from the database. And then internally you would have sort of stronger types, perhaps. So yeah, so perhaps, but in data oriented programming, we'd say you don't need the, you don't necessarily need the stronger types. You, so this would be done. Yeah, I guess in this example, I'd try to say like, this is just for communicating with outside things. No, you would pass it between functions. You would do it like everywhere throughout your application. You would basically avoid actual typed objects. Now we do have certain practices that I'll get to in terms of like, there's probably alarm bells going off of like in your heads of like, how is this kind of feasible or how do you write correct programs in a reasonable way? And I'll get to that. But it's basically the idea of like using these sorts of things just everywhere, internally as a way of communicating with outside things. I mean, depending on what you're communicating with outside, you might have to like convert it to JSON or convert it to photo buffers or whatever, but internally just like between functions, between different parts of the internal application. Yeah, just sending these sorts of things around. Okay. And this kind of given what we have to work with in enclosure, which is it is much richer than what JSON or even JavaScript object give us. Cause like, there are literals for dates or literals for UUIDs. There are much more kind of, there are more primitive types than something like JavaScript objects have. And like JavaScript, all keys or strings in enclosure, it maintains the type of the thing that's used in the key and the value. Like it's a bit better, but it is quite similar to this idea of just like passing these like very simple like JavaScript objects around or very Ruby hash maps. And I kind of think of it as another way of thinking of data oriented programming is we choose this kind of plain data to be the kind of lingua franca of information in our programs. And we're kind of, we're willing to sacrifice some of the benefits of perhaps doing things with stringer objects or typed structs and things like that. Because this is kind of a lower common denominator and then we might get some benefits by trying to lean into it this way. And just kind of how like in, the idea of a lingua franca is, you have people that are capable of talking to various languages but they kind of settle on this like pigeon language that is a lowest common denominator and there's kind of benefits that come from scale from having settled on that. And then that kind of tends to grow and there's kind of benefits from doing this wholesale and getting to those benefits really the reason for it, I think or at least the argument for why this is beneficial is that by using these kind of generic data structures, you get access to all of the functions in closure and libraries and whatever, but the closure standard library is very strong in terms of working with these kinds of data structures. And so you get access to that and you can use them anytime, anywhere you have data. You don't, so instead of having like specific access and having, you know, every type of object having to have its own functions to get stuff in and out of it. You have generic functions to work on these generic objects. I shouldn't call them generic data structures. And you also, so not only, so you have access to these, you know, this gigantic awesome code collection of a standard library, but also you then have the freedom to manipulate that data in whatever way. You wanna just like temporarily merge to, you know, these two hash maps together, that's fine. You wanna do a diff, that's fine. You wanna just extract some, just the keys, that's fine or just the values, that's fine. You wanna like, you know, intermix, like if you have a bunch of, you know, an array of numbers and array of other things and array of strings and you wanna stitch them together into a map, that's fine. Because it's all the same kind of stuff. You've chosen to kind of settle on this common soup of stuff. I kind of think of it like, if you look at, like a typed object, you know, it's hard, like how do you get the methods of it? You know, maybe there's some metaprogramming availability and but usually it's slow and not typically used very often, unless you're like, you know, in Ruby and doing kind of metaprogramming or you're writing Rails and then, you know, the internals of Rails just really rely on the Ruby metaprogramming abilities. But yeah, getting the methods of an object, getting all the values of an object or if you want to just like have an object with a subset of these methods and pass it off somewhere else. Well, if it's an object-oriented programming language, you're gonna have to probably create a new type so that you can, this subset it type and pass it off somewhere else. Merging things together, diffing, doing kind of a temporary mashup of things inside of a reduced statements. There's a freedom to the kind of, to this approach. But it does of course come with trade-offs. You know, and the trade-offs are basically everything that you would get with being able to kind of have static, strict defined types of things. And that includes things from my, at least from my perspective and I'd love to hear the point of view of people who use type languages because I think from both sides of the table, it's kind of like, how could you, you know, someone who works with types is like, how could you possibly live without types? And on my side is like, how could you possibly kind of get things done with types? Because, you know, in the closure community, there's this kind of idea that, oh, you know, with type languages, you have to spend all your time just like writing the type system. And I want to hear your thoughts on that later. But I do agree, and I definitely, as a closure developer, I know that part of the trade-off is, you know, typing helps in terms of documentation. Like it is a self-documenting in terms of what things are in the system. And I can look into a place and see, okay, this kind of object, what can I expect about it? If I'm trying to just like do validation in various places in my application, that could be useful, or just like have general correctness throughout my program and make sure that I don't accidentally, you know, pass this function a half broken object because it's missing some keys or values. And also just like the general things, like the things that an IDE, a good IDE can do, like JetBrains is pretty fantastic because of the type system. And it's able to kind of infer a lot of things, like give you auto-conclusion refactoring, support, compile time, correctness checks. Like there's a lot of awesome stuff. And some of it, you know, all of these things, all these trade-offs I'm saying, there are approaches to an enclosure. Like there are some libraries that do introduce some static type checking. They're not like as 100% comprehensive as you would get in something like Haskell because there are certain things that, you know, are ambiguous and therefore can't be determined because we have dynamic typing in the language. And for, you know, we're talking about this idea of self documentation and validation closure. There are libraries that help you write what we call specs. So you could choose effectively when you wanna check at runtime various things. And so, you know, in most of our systems, when we get data in from somewhere else, we will type check. Like we will basically validate that the shape of the objects or these primitive data objects that we're getting matches what we're expecting. But it's certainly not throughout most of the application. It's really kind of on the edges, we would checking that like, are we getting this correct stuff in and are we sending the correct stuff out? And maybe on like library, like whatever surfaces we choose, we might introduce these use these libraries to, you know, check the shapes of our data. But yeah, there are some things that doing it comprehensively might be sometimes nice. But I think, yeah, enclosure and data oriented programming is kind of saying, you know, type checking is nice, but it's also nice not to have to write types. And it's nice to just use the same functions with everything and not have to use the custom getter function for every different kind of object, et cetera. And the other thing that we kind of lose is encapsulation. So, you know, one of the big core concepts of object oriented programming is encapsulation. And by representing our data just as data and not behind inside of an object and with methods and stuff and getters and setters around it, we're basically tying the implementation of our data structures to the interface. Because it's the same thing, but there is no interface because the thing is itself. And so there is like basically no encapsulation. If something changes significantly, that change might, you know, have to be propagated throughout the application. But I kind of might say that the opinion kind of of data oriented programming is encapsulation a little overhyped, overrated. And I think, you know, as a functional programming group, we might also be a little bit of that because like we have different ways of doing encapsulation. And part of that is, you know, if you have parts of your code or parts of your application that deal with your data as generic data, like for example, if I'm writing something like a map function or a reduce function, it doesn't really care what's inside other than it implements kind of the, that it's a collection that it can, you know, iterate over or something like that. And so having generic data structures and generic functions that can operate on them is fine. And even if you were to make a change to the implementation of one of your, you know, data representations of things, the parts of the application that treat them as generic data aren't gonna really get affected. So the encapsulation isn't like the loss of encapsulation isn't that big of a problem there. And in the cases where like, you know, you're actually changing how you've implemented something but you could have kept the same interface. I just like in practice, like doing, having written stuff for 10 years in closure, I don't feel like, I think it's very rare where we've changed the internal implementation of something and it's been a giant pain everywhere else because most of the functions that deal with those kinds of objects that, and like they care about like the domain of that object. Like, you know, this is a user object and this function works on user objects and no other types of user data structures. And in those cases, like you have that collection of functions that only work with those kinds of limited types of data structures. They're all in one place anyway, and like in a namespace somewhere. So I don't, you know, we lose encapsulation but I don't think it's a big deal in practice most of the time. And then there are some other little things like, yeah, sometimes performance can be an issue and some things might be hard to model. And I'll say that like data oriented programming might say, you know, we're not saying always, you know, only ever you, you know, use these JSON like objects, it's okay if there's practical things that cause you not to. So yeah, if you're passing video streams around, you're not gonna wanna represent them as strings, like, yeah, you can still have binary data. It's not, they're no longer plain data structures. It's no longer plain data, but that's fine. Like it's more of like the idea of data oriented is let's kind of start with the assumption that by default we're gonna use these generic things except for where it can't. But, you know, choosing a different default is an important thing, like is relevant. Like, so me saying, let's default to this but not do it whenever it becomes impractical. That is a very different thing than saying, let's default to typing and not type when it's not practical. And so, you know, can this kind of stuff work outside of closure? And, you know, and I mentioned how, you know, in Ruby, there's really not much stopping you. Like there are hash maps in Ruby. And in JavaScript, you know, you have the choice of using these plain JavaScript objects or using this whole like class data structure methods and like these class type approaches. And I think there's quite a lot of languages that give you that choice of these like plain things that aren't typed and not, you know, protected by an encapsulated object class instance, whatever, you have that choice and different communities make different choices. And even in closure, we can't, we have the choice. We can do, we can create Java objects with it. We can create protocols, we can do all this stuff. And, you know, at the end of the day, most of those fundamental types of things that we use in closure, like those hashes and collections and lists, they're all objects and they're all Java objects under the hood. But in our programs, we choose not to. And so the kind of questions I was thinking about like, what makes this possible in other languages would be, you know, is there an easy way to create these things? In closure, all of that, everything kind of, that I mentioned, we could do with literals. So there is a literal for you know, hash maps of all these keywords, vectors, lists, sets, dates, UUIDs, they're all, they're literal, there's literal syntax that we can do for creating these sorts of things. Are these, the other one is, are these data structures rich enough to actually model with? And I find that like, you know, JSON, you know, we have JSON APIs that are everywhere now. And I think that's kind of an evidence of like, it's good enough because XML arguably is much, much richer in terms of the kind of things that you can model with XML. And JSON is a bit of a kind of less rich in its capabilities, but it's good enough. And it reaches this kind of interesting balance. And so in closure, it's, we have things that are slightly richer than JavaScript. I would think that like, you know, in JavaScript, I would be slightly hindered by having to do this kind of stuff, cause it's really nice to sometimes put an integer as a key and know that it's going to stay an integer. So there are certain things that are kind of frustrating in some languages. And then thirdly there, it's, you know, is it actually easy to manipulate these data structures? In closure, you have this giant standard library that assumes that like you're going to be working with these sorts of things and every single fundamental kind of primitive type in closure like implements a whole bunch of interfaces. So whether it's a hash map or various kind of alternative implementation of hash maps or an array or a string, these are, we can iterate over all of these. And so you can use map on hash maps on like the map function on hash maps, on strings, on lists, on vectors. And, you know, most of these functions can be used on a lot of different things. And there's value in that. And if you don't, and again, I'm going to pick on, pick on JavaScript here and why it's like hard to do these things and why this isn't really done, but in a hobbled way. It's partly because like the JavaScript API, the standard API doesn't really have a lot of like good functions for, you know, it has map filter reduced, but it doesn't have a ton of different things that people are used to in functional programming languages like a partition function or zip mapping. And not only is like JavaScript limited, it also like the standard kind of library is quite limited. It also like is like a mixed bag of immutable functions, which for me it drives me insane because like I have a whole bunch of stuff I want to do with an arrays and half of the functions are going to mutate my array and half of them we're going to return a new array and unless I check the documentation, I don't know. And that is painful. In closure, you have everything is immutable, everything you just kind of know by default what you're going to get and that does simplify a lot of things. And the other kind of big thing that is important I think is the value semantics. So for example, in closure, we can take, you know, we can generate two maps of things and from two different sources and we can compare them or we can compare arrays and they're actually compared based on their values, not whether they're the same object in memory and some languages you can't do that. You have to manually kind of, you want to compare these two arrays or you're going to have to manually do a deep tree traversal to figure it out. And part of that again goes back to this idea of how in closure in particular it's because closure implements all these things with immutable data structures and so it gets some of those various wins in terms of comparisons because of that because it's a lot easier to compare because two immutable things or a thing than it is immutable, like a bunch of immutable objects. So yeah. And I think, you know, I'm like, I've been talking about Ruby JavaScript as an examples in closure because those are the ones that I'm exposed with but I think it'd be interesting in the discussion to kind of see like, you know, do you think we could do this in like your language of choice or what would you have to be giving up or what would be easy and what would be hard? And so we'll get to that because I'm almost done in part one. So I wanted to kind of bring back kind of to this definition or my working definition at least is that like data oriented programming is around like choosing these like plain data these generic primitive data structures these kind of objects that aren't objects because objects kind of are types. They have, you know, specific names whether they're have methods or not. So whether they're records or not it's choosing even if it's available choosing to use these kinds of generic things because they're very flexible and by having your entire application make use of them it makes it much easier to then kind of just like mix and mash and manipulate these with the standard functions. All right, so that's my part one and so my kind of question to kick off some discussion is what language are you working with and like, is this even possible? Just out of curiosity, like can you do this in F sharp? Like can you represent things in F sharp without having to type them? And or, you know, would this be easy? Yeah, so that's my kind of pick off question but you're also welcome to kind of chat about. So if I can jump in, so I mean, I'm an F sharp user and I do come from the other side of the bench but so one of the things I like about the type system and I use the type system a lot is for example when I am refactoring code, right? So if I wanna make a change somewhere in the code base to, you know, the contract, the data contract of a particular object, you know, I change it in the type and then the compiler breaks everything and it says, okay, well, this isn't working, so fix this. So I go through and it gives me sort of a way where I can effectively go through the code base and, you know, update the code so that I have some guarantees of it, you know, running with this new data contract. So I guess my question is, so first of all like if you come into a closure code base that's maybe not written in a way like not written using sort of this data oriented programming what changes would you make? Like, how would you change? What would you expect to see in a sort of non data oriented code base and what changes would you make to get it into a data oriented code base and how do you manage things like, you know making sure that at some point you aren't trying to, you know, add two strings together. You know what I mean? Like the kinds of typical when I think of JavaScript, you think of like sometimes JavaScript forgets that something's a number and it tries to do a string addition to two numbers or something like that. And there's all these sorts of weird errors that can come up. So I mean, how do you avoid those sorts of data contract issues when you're trying to like refactor a code base? Sure, yeah. So I think there are three things there you mentioned. I'll kind of try to work backwards because I remember what they are. So, you know, I would say, you know, closure that issue of like adding strings together accidentally closure isn't like, you know, there's kind of a side of like strong and weak typing versus static and dynamic. And so most things like closure would complain. Like there aren't functions that are overloaded in that way and in closure. Like we addition is separate from is not the same function that's used for a concatenation like addition is for numbers, the stir function is for strings. But yeah, it is possible to, you know, write code that only when you run it at runtime it gives you a type error because it is kind of dynamic. And so you're not getting these compile time warnings. And so that first thing you mentioned like it's where you can like break something and then your compiler tells you every single location that is broken, that is nice. And it is something that we kind of lack. The things that make up for that is testing like having automated tests because, you know, I think there are certain things that types still can't catch, right? Like, or depending on the type system there's things that are harder to represent. Like, you know, if you were implementing, you know, this is a trivial example and no one would really do this. Like, if you were implementing your own reverse a type system might say, well, okay I know that the incoming types are going to be the same as the outgoing types but can your type system capture that it's going to have the same number of items and that they're actually going to be like reverse. And so for certain things like that like I feel like you still need to write tests and tests will catch a lot of those kind of trivial errors but the like knowing where we need to change things for certain changes certainly can be an issue. And so what we lean on I feel is having reasonable coverage in our automated tests plus also leaning on functional programming kind of makes it somewhat reasonably easy to follow through the program and know where those types of things might end up. It's not, yeah, but yeah it is a legitimate trade-off in terms of like, yeah it is certainly not as easy. Like it would take me a bit more time to identify for sure every single place that if I made a significant change to a piece of data structure that it's not going to affect something else. But I guess the other question I had was kind of if you come into a code base you want to like, you know you want to sort of push into like a more data-oriented programming paradigm. I mean, what sort of changes do you typically make? I mean, do you get from like contractors or like, what's the... The, I think partly I think part of here is that like in closure almost everybody writes stuff in this kind of way except for like you could type you could use records like closure allows you to create records and you could use what records throughout your entire application. But if you showed that to a typical closure programmer they'd be like, cool you've used records but we kind of don't do that around here. Like, and I'm not saying that like, yeah that it's necessarily better or worse I think data-oriented people who want to follow this kind of practice are making the choice of that in those trade-offs that they would prefer the benefits of using all the same functions on everything and having these kind of more generic objects and then introducing these kind of type checks but the more just like runtime shape checks wherever we're worried about that kind of stuff. And so, yeah, I don't know like it's kind of like this is me kind of just reporting on this is how we do things like this is how like people in closure land do things and it's just like it's been done for quite a while and a lot of code bases kind of look this way. So it's hard for me to specifically give a good example of like this is a case where I saw someone doing something like this and they're doing it different. It's more of like people are doing things differently in other languages. And so for example, if I was in Ruby and I saw a Ruby code base the way I would write Ruby a lot of like in the past when I was following Ruby idioms I am doing trying to do idiomatic Ruby. Yeah, I would like every single type of object that I would want in my system I would create a class for it. And then I would have getters and setters and there's a user object there's a company object and somehow the relationship between the users and companies is represented in one of those two objects maybe whereas now I would just be like now it's a user as a hash map and has an array of like company IDs and companies are also hash maps. And you could do that in Ruby, right? And so if I had a Ruby code base and I wanted for some reason to be like, okay, we're not doing Ruby idioms we're doing closure idioms in Ruby then I would be like, okay, let's not like do we need objects for all these things? Wouldn't it just be simpler than maybe just pass them around as a generic hash maps? Cool, thanks. So I have a question or observation. So one of the points that you made I think maybe not on the slide but just in passing was that you kind of find it like a pain in the butt to like write out the types it's like kind of like a lot of work. And for me, like that's definitely true of like Java and C sharp, you got to create like a whole other file and you've got like directories with like 15 types for it gets kind of crazy but then in like F sharp and Haskell certainly and OCaml and Scala it's actually really easy to just very quickly like in a single line, write out new types or new objects that are really focused. And so I just wonder, I don't know anything about closure so I don't know how hard it is to write out new classes and enclosure. Yeah, it can certainly be like part of the case maybe the languages and the tools that we use shape the way we work with them and it very well may be that like, if closure made it dead easy to type things then maybe people would be typing things more. But my return question on the trade off is do you like can you map over like what do you have to do in order to like if you're trying to make a list of like new things and like say I was creating a new struct can I ask for it's just like the keys of that struct or do you have to implement a method that implements that? Like. So I think you would have to implement your own method. Yeah, even if you're just gonna reuse like whatever is in the standard library you would still have to give a way to access it. Yeah. But it could be, I'm not saying like cause I think it might not be so bad cause like I guess it's maybe just like in whatever language they just the standard interfaces for those things don't have don't implement no keys on structs but in theory they could, I don't know, right? So it might, this might still be okay. It really depends. Yeah, and again, yeah, I think in some language like Haskell and things like the type system is quite flexible and easy and it might make that kind of trade off lesser, right? And so that's why I think in Haskell they don't do this as much because they don't have to. Right, this is sort of reminding me that like this might be where type classes provide like an alternative to what you're talking about like what you want while also giving you richer types but also like common behavior. Can you clarify what you mean by type classes? Yeah, so people are gonna have to help me out with all the more Haskellites here. So they're like they're interface-ish kinds of things for types themselves, so they're sort of like types of types and they typically have, they're modeling functionality. So if something, if a certain type is a functor then it has the map method on it. Yeah, and so that sounds like interfaces kind of. And so in closure, like all of these fundamental like hash maps and array vectors and lists, yeah, they're implemented as Java objects and they have interfaces and they implement various interfaces and so that's kind of how you get the functions being able to interoperate across so many of them is because it just so happens like the hash map implements the same interface that an array does or a vector does in terms of its ability to like iterate over it. And so I guess the real thing is that how easy would it be to kind of like if I wanted all the nice stuff of like the existing standard library of hash maps in theory enclosure, I could like public like fork or like subclass or basically take a hash map, everything that's in a hash map and just call it my own thing and maybe add some restrictions to it. Like I could, it sounds paint like not painful but it just sounds really weird for me to say that but I guess that's kind of what you're kind of proposing potentially like if you're creating your new types on the fly, you would just basically say like this is a hash map and it contains these things inside of it and then we're just calling it a new thing now, like a hash map with numbers inside of it that has a length of five. Yeah, so another follow up and this is maybe like another question that I was wondering was that you did mention that you give up kind of validation stuff and at least like maybe internally you said you like pushing it out to the edges and that was to me like one of the more compelling reasons to use like really rich domain modeling is to get like all the validation in the domain and make sure you have, I mean, you can push it really far and try to make invalid states completely unrepresentable so you have like a user status, they count status as like verified or unverified and that's it, it's not a string, it's just... Yeah, it's like an enum or whatever. Yeah, it's like an enum, right? So I guess the way that we do things in closure and therefore also in like data oriented programming is like we have kind of like optional typing like we get to kind of just at runtime, optional, stronger like because these generic objects aren't types but we can use libraries that will check for whatever we want at runtime. So I do in my programs tend to then have a way of speccing out that a user data structure has the following keys and these and their values have the following types and because it's at runtime, it's also possible to then say, well, this list has a following and it can only be three things and these numbers can only be within this range and there's some things at runtime when you're doing kind of specs that are I think very easy to do that might, I don't even know if it's possible in some type systems like whether, yeah, like I can write in my spec for a user, I might say, their email is a string and it follows this regx and there and there's also just give arbitrary runtime functions for validating whatever data I'm getting and so those we could put in wherever, like we could check and there are libraries that some people use that they kind of simulate this kind of typing everywhere where there are libraries that kind of give you your own like function, like instead of defining a function using standard closure-defin syntax, you're basically using a library that lets you put annotations on like what's coming in and what's coming out so that you get it in on that function and some people like to use that and use it throughout but in practice, my understanding is that we kind of most shops and companies will just do it where they feel it's most necessary. So for example, if I was implementing HTTP API a web server, I would definitely have all of my incoming data checked and perhaps if I was sending stuff out from that service or sending it to some other API I might do certain checks and so and then wherever I'm kind of worried about it or wherever it's particularly sensitive and it is possible in theory to take that all the way and every single function input and output is specced out and so like we call them specs they're not really types, they're runtime validations but in practice, I don't think I'm doing it everywhere. I see, okay. Cause like for some code it's like, you can kind of see you see that, it's gonna be, you see that it's correct and any changes that you make, you can trivially know that they're gonna break this or not break it and a lot of code like is generic and it doesn't care cause it can't really break or not break but the stuff that is domain specific sometimes might break or not break but I guess it is a trade-off. Again, I'm saying these are trade-offs and then just like it just so happens I think that in our community people are willing to take the trade-off. So let me jump in here real quick and I don't wanna distract too much because you have a whole nother part of your talking. And I think I might wanna come back to someone saying later at the end of the talk cause I've been thinking about this for a while but part of what I think is going on here is in this sort of constant dynamically typed versus statically typed debate is in the social sciences research community that I'm part of there's a classic, now I think it's a classic text called the tale of two cultures that talks about qualitative research versus quantitative research. And basically what they argue is there are two fundamentally different logics going on and everyone's talking past each other in the debate. And so when you're critiquing static typing I think what's going on and I hear it in what you're saying but in general when sort of dynamically oriented people are critiquing static typing what they're critiquing is sort of mediocre static typing. And I don't think so like Java or Python which isn't static typing but Python has this weird, it's dynamic but it's strongly typed and you get into these weird annoying issues whereas those critiques don't seem to resonate at all for fancy type systems like the MLs and Haskell have. And conversely I see it on the other side of sort of statically typed people or especially people from who are working with fancy type systems don't seem to appreciate what dynamic like real good dynamically typed systems can do. And part of what I hear you saying is sort of let's push away from sort of this sort of in the middle dynamically typed system more towards sort of like untyped and minimize the types as much as possible. And I think and I'm very sympathetic towards that because and I've talked about this a little bit or a fair amount in this group is the two languages that I use the most now are TCL which is untyped and OCaml which is fancy typed. And I've found sort of going to those extremes to be really, really useful. And in TCL it's not that everything's a hash map it's everything's a list. And then you're just passing lists around in the same way that you're talking about I think in the same way that you're talking about passing hash maps around. As far as I know TCL is almost a list, right? Or it has like heritage there. But yeah, it's the same like effectively a hash map is a list of keys and values. It just has that constraint that there's pairs of things. Yeah, and I think you're right. I wish and something that perhaps I'll try to write down if I ever try to give this again is there is an effort in closure to have what's called typed closure where someone has gone through and basically kind of forked the language so that you could use kind of try to introduce this slightly fancy type system. As far as I know, they were running into like some significant limitations around the way that certain functions work that would make it really hard. And it might just be a question of effort because like I've heard a story and I don't know how correct it is because like Haskell has this idea of lenses and the lenses library is amazing but apparently like 90% of the effort of that was trying to make it possible to do lenses within the type system of Haskell. Whereas this idea of like the way that lenses work would be much more trivial in a kind of dynamic system. I don't know. But yeah, for sure. Like I think having fancy type systems makes this kind of trade off less worth kind of more in the favor of, yeah, like these type systems don't get in the way so much. And we are in probably, I'm picking on the shitty ones but in practice like a lot of commonly used languages have these shitty ones. Cool. Yeah. So let me get into part two because we'll probably take another similar amount of time and then we'll chat again already. So let's move on to data driven programming. And I mentioned, let's say we go back, my quick definition, I don't have a site for this for data driven programming was using, preferring kind of these plain data. So we're still sticking with this idea of like plain data and preferring and finding ways of using that instead of writing code. So like less code and more of the static data. And we'll kind of see what that means but if you kind of know the city of like meta programming of like using metadata around the various objects that we might have in our program to then conditionally do different things, this is similar except instead of using like objects and metadata facilities, they're just data structures. And so you can just ask them what's inside. But let's kind of, I'll kind of get into it more formally here. And so modern programming I'd say is rife with this idea of like domain specific languages. And language, there are domain specific languages that exist externally. So something like SQL, right? Isn't necessarily embedded in your program. It's just a language that's really just much talking to databases. And there are a lot of these other languages that aren't general programming languages. They're probably, they might not even be Turing complete and they have usefulness. I'm gonna be talking more specifically about like embedded ones. So these are languages that are within other languages or just like libraries that are effectively their own languages within a language. So I'm calling these embedded DSLs. And they solve some specific kind of more well-defined problem that more specific than it's Turing complete and I can do anything when loops and ifs and whatever. So something more specific than that. So the domain is limited. And so they tend to be more declarative. Although, the more I think about what declarative versus non declarative, I kind of lose meaning. I did a talk last year about declarative trying to explain declarative programming. And by the end of it, I just lost all meaning and like I couldn't distinguish it like anymore. I feel like it's a kind of an arbitrary distinction of like what versus how. Anyway, but yeah, they tend to kind of be what we've called declarative, not Turing complete versus general programming languages. And so with these embedded DSLs, the most common one that exists in, I think, every languages is a RegEx system. And so within your language, you can call into a RegEx system to run a RegEx. And that was a well-designed and very common, well-designed interface, a well-designed, a common problem. And so everybody kind of has access to it and it's very common. But there's a lot of other ones that are less so. And so some examples here are like, so this idea of JSX, which was the React team, I think in trying to be like, okay, we want to represent, we want to do templating. And so we want to do like intermix HTML and JavaScript. So we're basically going to create a super set of the JavaScript language that lets us intermix HTML and JavaScript and effectively creating a whole new domain specific, like if we looked at the HTML parts of JSX versus the JavaScript parts, the HTML parts are kind of the domain specific language that's kind of embedded in JavaScript. Most of Rails and a lot of Ruby libraries could be actually domain specific languages. And in particular, like various parts of Rails, like how they do their routing system. You're basically using objects in weird ways to represent something that eventually becomes, like tells the Rails routing system what to do. In Clojure, there's a library called core async that effectively implements like go loops from go, completely changing the behavior of the, like anything that you put inside of a Clojure, a core async go block operates on a completely different runtime. And so most DSLs, like when I think of DSLs, effectively they're giving you a new interface into a different runtime than some other language. Your F-sharp, the VM has some behavior and you have some ways of interacting and controlling that behavior. And then you might have a DSL that offers you kind of a little sub universe of a slightly different VM that you can kind of program slightly differently. And I think with kind of the bargain of DSLs is that you get potentially like much more power. You can do things, you know, if you think of a regex, the amount of like, if you had to implement what a regex typically, you know, pretty simple regex is doing in regular code, it'd be quite a lot of code. And so they give you that power and often at the cost of like more complexity and the complexity primarily comes from, well, now you might have like two VMs, like there's different parts of the code that are interacting. And in the case of regex, they overcome that because there's a very clean interface. Regex, the regex system of most languages is kind of just like a function call. You know, all you say is, here's some text, here's this pattern, give me a reply back. But that's not true of a lot of other DSLs where it's kind of more like a, you know, JSX is it's all interleaved. I was talking about this core async, basically have a completely different runtime behavior that you can embed in any parts of your program. And so they, so there's a potential for introducing much more complexity unless you limit the expressiveness. And so, you know, regex is provide a clean interface, but also have a very limited set of things that you can express. And so I think part of the trick of like getting your kind of cakey eating it too in terms of DSLs is giving you more power, but limiting the expression and what you can do with it so that it's not just like an explosion of complexity of like, well, now I can just do so many things and they can interact in some of the different ways. So it's kind of just like some abstract thinking about like what, you know, DSLs are. And so in terms of like the ways that you might create a DSL and an embedded one specifically, you know, most people, if they're trying to solve a problem that they think that they could solve by creating a new language, reach to creating a new text grammar, like effectively just like straight up, let's write a new language. And then JSX example, they couldn't modify JavaScript to do what it can do because they wanted to introduce new syntax. So they had to write a new compiler that was a superset that handle the superset of the JavaScript language. And in the case of, you know, SQL, they didn't use anything else. It's just, it's its own language. Now from a, you know, how it might interoperate in like if we're writing, trying to write SQL from within, let's say JavaScript or whatever other language, one way then to interact with this is then to just kind of drop into basically text. Like this is a common way of interacting with SQL databases is that like at some point you're just like dropping strings of SQL. And occasionally there might be some pattern matching and or some way of like putting holes in that can be replaced, like in this case, but effectively it's like, yeah, like we're just going to create, either create a completely new language, well, we are going to create a completely new language. It's going to be parsed as text and the way to interact with it is through a completely separate compiler or through just passing text to some external system. Another way though, that's common in some object-oriented languages is something called like fluent APIs. And that's basically like you're creating a sub-language that's making use as much as possible of existing things in the current language. So you're using like functions or in the case of most fluent APIs, you're using various objects and methods on those objects and you can chain those objects. And at the end of the day, it creates some internal object hierarchy representation of the thing that you're doing. So this is some library called reql. I think it's JavaScript also that lets you represent that same text-based query by doing a whole bunch of like method calls with various parameters. And why is this any better? And like part of the argument for why this is better is because if you wanted to like compose and if you wanted to create this kind of query on the fly and have if statements and for loops or like basically like do anything at runtime and generate this kind of query, if you only have a text-based system, you're basically resorting to string manipulation and you can really easily shoot yourself on the foot and you basically have to understand the full semantics of the language and whatever kind of thing that you're creating. And if you have this kind of fluent API, it tends to make it easier. Like I didn't have to, I could have represented. Let's see if I can doodle here. I could have had everything up to this point saved into a variable and then later given some conditional at this part to it. Usually this fluent API is gonna work in that way. And so that composability that generateability is quite nice. But I do agree like sometimes looking at these systems it isn't like I would much prefer just like this seems much simpler to me. This seems like kind of painful, but maybe it's just the familiarity of SQL versus having to like re-implement SQL in like some language. Sometimes it's very awkward. So another way, and this is common in like in the languages that have strong macro systems is macro-based yourself. And I think these two ideas are very similar. It's just kind of like if you have objects, this is how it looks like. And if you have macros, this is how it kind of looks like. And so in a lot of lists, these kind of APIs might be or DSLs are popular. It's trying to make certain things easier and trying to make, you know, not having to resort to doing string concatenation. But the challenge with macros is that they're kind of opaque. When I get with the thing that I get from this and the stuff that I'm passing to it, it's like it's hard to kind of poke and inspect it. Like, I know that I have a select statement, but once I get this thing, is there a way for me to, you know, ask things of it? And very often it's not because like the internal representation is just some opaque thing. And so this, we're kind of now getting to this idea of data-driven DSLs. And so the big idea here is, let's just use plain data structures as our interfaces into any of these kind of DSL VMs. So rather than using text, rather than using objects and chained method calls, or rather than using macros, why not represent whatever we want using plain data structures and pass those off to the system? And the reason that this is quite nice, and like why kind of, I call the top three, that's kind of like various forms of code. Here's a completely new language that we're representing as a string because that's the only way we can do it in our language. But these are just like, the top three are like code. The bottom one, I distinguish as data because it has very different characteristics. And so it's worth kind of distinguishing it. Like, yes, this data exists within code, but if we're just talking about these like plain data structures, they have properties that code doesn't have. Like partly they're like transparent. They are just what they are. This is a hash map with keys and values. And in closure, it has no behavior because it's immutable. So this thing is just what it is. It's kind of like a number. It's just like in terms of like numbers are just values. This is just a more complex value, but it has no behavior. It is what it is. And it is trivial to manipulate. I could insert, remove things. I can generate this very, very easily if I wanted to at runtime from composing it from various kind of sub parts. Super, super easy to kind of manipulate this kind of stuff. It's also easy to kind of serialize, pull this in. You need to save these sorts of things somewhere or from a config file to the database. Trivial to inspect in terms of like, if I wanted to again ask for the keys for the values of these things or how many joins am I doing in this query? The data just is there and you can do whatever you want with it. You can ask whatever you want of it. And then it's also pretty easy to, you can write pretty complex specs. And I talk about specs in terms of versus types, but you can conditionally check whatever you want, whether like, oh, I wanna check that this join it only has two joins because for a reason or any sort of like any question that you can ask about this data structure, you could write a function to kind of ask it. And that probably would not be as trivial to do with the kind of the top three approaches. So, you know, there are some trade-offs though. And again, so this is, so the big idea here is that, yeah, like why don't we just use data structures as our interface to these external systems or even internal kind of DSL systems? Bunch of benefits to doing that. But some of the trade-offs is that occasionally it can be more verbose, right? Like usually the most terse representation would be a full custom language. And having to kind of pseudo-represent something within the constraints of an existing language is sometimes hard. If you're parsing things at runtime versus compile time, they're occasionally like performance issues, well, I'll talk more specifically about that in a moment. And, you know, and some things are Turing-complete by nature. And so like if you have a VM that is Turing-complete then you probably want a Turing-complete language to work with it instead of like something less than that. And so my key example of like a bad design choice in my opinion is Ansible. And that's why I was, I mentioned it earlier when we were chatting at the beginning because like Ansible made this choice of, okay, we're gonna have idempotent functions which I think is awesome. So, you know, once you run a function on the server if you run it multiple times you're still gonna get the same result also. But then they also conflated that with having it to use a declarative data using YAML. They used YAML as their language. And I argue that Ansible is not like it's using declarative syntax but in practice you're still writing like an imperative like do this, then do this, then do this. Cause like deploying a system involves multiple steps and they have to be in order. And so really like it deserves an actual language. And I think using YAML as the syntax to what's effectively, you know, Turing complete language is a giant pain. So that's my little rant on inappropriate uses of declarative data structures. But in many other systems it is I think a practical and a good approach. So I'm gonna go through a whole bunch of examples in like the closure ecosystem. And, you know, in closure you can represent regx is kind of using the regular PCRE text syntax but there was, there is a library called regal that lets you represent them using data structures. And if you need to dynamically generate regx is on the fly it's much easier to work with composing things with something that looks like this than this, than using string manipulation. In the problem of let's say routing. So this is like writing an HTTP web server. For many years the most popular library enclosure was called composure and it used macros as its primary kind of interface. But basically once you wrote this you composed kind of your code here it effectively just like, you know all of these things here were macros and it just gives you back a blob and you can't do anything with that blob other than just say, well, run it. It's completely opaque. You can't, and it's effective like it's very hard to like generate like if you wanted to dynamically generate a server based on some configuration somewhere it was basically impossible to do it or hard to do it with these kind of macros. And so for the last few years there's been a bunch of libraries but one of the more popular ones does this now in a data oriented way where you represent your entire HTTP API as a data structure and the underlying kind of library converts that into an HTTP server. But by being a data structure it's possible to like, you know you can put if statements or for loops or whatever to in here or you can mishmash, you know one schema with another schema and like you have this complete freedom of generating what needs to be passed to this library. You know, I'll come back to this idea of UI templating. So, you know, we talked about JSX which is how you do things and react but in closure, most templating languages just use closure data structures. So arrays of keywords and strings and hashes as a way of representing HTML. And by doing so it becomes trivial to just like introduce for loops or if statements whereas in kind of this language approach like there's a lot of weird mangling of like doing maps and having to only you can't use if statements you can only use ternary expressions. And it's just, yeah, it feels a little janky whereas here you're just like I'm just manipulating data structures or generating data structures to do what I want. And then my library is taking those in and doing what it does. So it's just like, it's a nice interface. I say the point here is like data driven is using data as an interface is awesome. And simple, stupid data is awesome because it's trivial to generate, trivial to manipulate. A few more examples, styling, CSS is its own language. And then when people tried to improve CSS they created a whole bunch of other kind of languages and one of them is SCSS and it's its own language. In closure, some people they were just like okay let's just represent CSS using closure data structures and manipulate it then into CSS. And again, in SCSS they had to introduce a compiler concept of mixins which let you like reuse styles whereas in closure by doing it this way the library author didn't have to do anything because you can just compose functions. You can just have one function return a subset of this data structure that you wanna reuse. And even like, you know, graph querying, GraphQL it's own full language. I feel like they could have done this using JavaScript objects but they chose not to for whatever reason. And in closure there's this idea of there's this library called EQL for this spec, a specification called EQL which describes effectively the same thing as GraphQL but using data structures so that you can generate it on the fly if you need to and manipulate it in whatever way you want. And, you know, I talked about specs before in closure there was a part of the closure language itself where they were introducing ideas of like how to write specs and they did it with macros and it became the main challenge I think there that some people didn't like was that one they were not very, it wasn't easy to introspect and generate but also it wasn't easy to like store like serialize in any way. And so there's a growing movement of using like there's this library that we use called Mali which just again lets you use data structures to represent the same things and then, you know, does the same stuff under the hood. Like at the end of the day, I get the same out of it in terms of I can have functions that I can use to validate data anywhere in my application but this kind of interface is much more flexible. And you know, and there are ways you don't have to go like full sale data. So here's an example of something we did in one of our applications where we wanted to define this concept of pages and so we just kind of came up with some data structures that represent our pages and then we generate the underlying kind of like HTTP and points and the various like front end infrastructure that needs to deal with different pages and link them. And it's just, you know and these aren't like pure plain data there are functions kind of sneaking in here and so this isn't like, you know this kind of taking more practical approach where we're still mixing in some stuff like I couldn't easily serialize this but it's still kind of a similar concept. We had this like routing like our own little like super simple routing system that just had, you know, an array of arrays and, you know, you describe like a get and you pass a function and then you have a list of middleware. So this is kind of somewhat data oriented apart from those kind of functions sneaking in there. If I wanted to make this purely data oriented instead of a function, I would put like a and I some keyword that references a function to find somewhere else but in our cases, it wasn't a big deal. And then we also, there's this library that I wrote called Tadah and it allows you to write these effectively super functions that include various, you know preconditions and parameter checks and separate side effects. And by splitting it out rather than having it just all within a function we can then use this data to generate the routing system and like the HTTP API but as well as potentially like we've done it where we use the same data to deploy to AWS Lambda. So by having data in a very declarative way like having these things written in a declarative way we can then manipulate it and for different purposes send it to AWS Lambda or implement it in a more traditional Java web server. And so taking this idea kind of to the extreme I kind of like I often now think of the applications as I write as a collection of sub-problems and like little sub-engines, you know there's the routing engine there's the page management engine there's like all these different like little sub-problems in my application that have interfaces to them and a lot of those interfaces are just using data and then I can have some source of truth data that I use to then I have some sources of truth maybe some schema, some data representing all the different types of things that are in my system I then manipulate that because it's data I can manipulate it trivially and massage it into what my various kind of sub-engines expect and so that's kind of what I see some of like that's what I have in my head in some of the things that I try to architect and is that basically we have some data structures that represent our data model and then we manipulate them into whatever our libraries need and hopefully our libraries are written in a except just like data structures and this is kind of like metaprogramming right it's effectively metaprogramming you're using you're manipulating some code to represent to generate other code except it's not really metaprogramming because it's not code generating code it's data being used to generate code and I'll leave with my kind of final example my favorite example of this which is AWS was in a situation where they needed to like you know release libraries for all the different kind of you know they wanted to have a Ruby library the Go library, PHP library, JavaScript library, Java library they didn't write their closure library but we'll get there and you know writing all of those libraries to interact with their AWS system could be a giant pain and so what they did was they represented every single endpoint in their AWS with JSON in a declarative way and then they just generated all those libraries or you know wrote some little library a little bit of Ruby that then generates the rest of the Ruby library or a little bit of Java that actually generates the rest of the like a full rest library and so the folks at closure but one of the teams like the core closure team or Cognitech which is the company behind closure took advantage of the same set of data to create the closure to create a closure library to interact with AWS and so I thought that was pretty cool and pretty powerful way of like not having to write 10 libraries instead represent data in one place and then generate 10 different libraries and 10 different languages. Yeah, so if I could be JSON being used to generate 10 libraries, different libraries. Okay, and so that's my bit for there and kind of bringing back, you know so data-drivenness is this kind of choice of if there are situations where we can represent things just as data or like static data structures then that's probably preferable because you know data is much simpler easier to manipulate and compose and generate than code is. And I think this is where like I would even say, you know in terms of how to apply this to other languages the generic part is not what's important here. It's using data structures that don't have behavior that are much simpler than, you know methods, functions, objects, all that kind of stuff and using those to generate the actual stuff. So introducing simplicity to the application by trying to move as much as possible into these like simpler representations that don't have behavior and then generate behavior from them. All right, time for questions? You have a few minutes. I'm willing to stick around as long as people are. Yeah, so I wanna thank you. This was really interesting. No, thank you very, very much because I've been hearing about data oriented programming and trying to understand it and now I feel like I finally have a better understanding of it and I've got a bunch of questions but I wanna open it up to other people first. So how about it? I just wanted to say really like the data driven the DSLs you were building out of just basically arrays and objects and things like that. I think that's definitely seemed like one of the powers of the non-type system where or one of the challenges I would have writing that in F sharp, it would be like I'd have to make sure that all the types are correct as you put them in and I don't have any like don't have that much flexibility. I mean, I guess I could use like a hash map or something like that, but that has its own issues. But yeah, I mean, it definitely is nice that the DSLs that you were writing were very extensible because of the fact that they weren't like any prescriptions on what types were allowed in the in this box. So I like that, really cool. And I think, you know, I think this might, you know this is why like these two are related, right? Because this data-drivenness is enabled by this like data-oriented practice, right? And so like I don't think the two necessarily exist. I mean, I think they could exist separately but the reason they do enclosure both exists is because they are so related. And I think, you know, yeah, it's so trivial to create, you know, to write these literals that it is a better way and so trivial to manipulate them and compose them that it just becomes like, yeah, the better way. But I do think that even in, yeah, I'm curious. Yeah, like so, you know, in F-sharp, like do you think you could do this? I mean, if it inspires you to think about like how could you do this, then my job here, I guess, is done. There are different ways of handling things in F-sharp. So F-sharp, you do have to kind of, you probably have to sort of map out the domain of your language. You can do a lot using like generics. So just sort of that sort of parameterization of the inputs. You don't have the sort of freedom that you do in enclosure. You know, typically when I have an issue like that, I use something called a, F-sharp has something called a computation expression, which is sugar for monads, but it's really just a nice way of kind of creating DSLs like that without all the ceremony and without making things kind of ugly as you were pointing out before. And there are certainly ways of making those classes extensible in the way that you're talking about, but I think they also run into the issues that you were talking about, especially where, well, the same kind of issues with macros is it's hard to document a fairly open language, like a fairly open DSL where it's sort of like, you can do a whole lot of things in it, but it's hard for me to say like, okay, well, you can do this, but don't do this because it'll break it or something like that. You run into a lot of frustration where since you're creating a different language inside of the one that people are presumably familiar with, it's hard for them to, they get frustrated when they try to do things that they would expect. They should be able to do, but they can't because of the strict restrictions on the macros or the computation expression. So yeah. Okay, well, I'm gonna jump in with one of my questions. So on these DSLs, I was trying to understand something. So like in the Honey SQL example where you've got a select and a from and a join is that hash map being read by a separate function that then is understanding what to do with a select and what to do with a from and what to do with a join or is it that there is a, that select is defined as a function within Honey SQL? I was muted, sorry. Right, okay, so in this case, Honey SQL is just converting this data structure to SQL and maybe under the hood, like it depends on how, like it might not be using text because like there's binary representations of these things and how to talk to Postgres, but it still has to talk to Postgres. And so perhaps this isn't, if we didn't have SQL, would we, is there a different way that we can write like a relational database DSL if we're using data structures? Probably. So this is still like confined within the world of, it has to eventually end up being SQL. But, and if I, yeah, so that's why this thing has no meaning other than the data structure and then the library converts this to something that kind of looks like this. But we could have had something, a much more different DSL here because the, if I were to compare for the routing ones, let me go to the, I somehow have done a bunch of keyboard shortcuts and now I have circles, okay. So the routing one, in this case, like rated has this nesting and destructuring and it can get quite kind of fancy. Whereas like R, the one that I wrote in an afternoon or less is linear. And so it's a different way of representing like the input into the system and this one is much less powerful and it's doing much less under the hood. And so the same thing I would say for the SQL one is like that SQL one is still just basically converting it to SQL, but maybe we could imagine if we were starting, if I was designing a new relational database system and I had to choose between an interface to it, I might not, and I was choosing data structures, I might not end up with this because there might be a nicer, even nicer way of representing relational data queries than this. And partly kind of like graph queries or kind of like that. If we go to the graph query example, this is a query that basically says, from the actor table, pull in actors, join on films, and for those films get their film title. And it's kind of like- But in these cases, there's a separate function that understands that structure. Yes, yeah, yeah. They're like, this is just like, this is an input to that function. So the reason I'm asking you this is going back to my TCL experience. So one of the things that we do in TCL, and I did this just a few weeks ago, and it was amazing, is you can, because the language is so flexible, you can essentially like on that select from join. So you have your data structure, but then you write functions named select from join that then actually do what you need to do. And then you can just shove your entire data structure into a loop and just sort of process through it and generate your output. So I had to present at a conference, and I've got this project basically on measurement. And there are two concepts that I call a concept versus a measure, right? It's all about sort of operationalization and how do you measure stuff? And the concept has certain dimensions, a measure has certain dimensions. And so I had those all sort of are written out in pseudocode in a document for this way of measuring, we've talked about this in the group before, measuring prayer style homes. And then like I basically had an afternoon to like generate this massive like visualization. And so I wrote a little bit of TCL code that basically sort of loaded up, I wrote a function for concept and I wrote a function for measure that then generated graph is code to sort of generate everything. And that sounds like another step, I guess in what you're talking about. It's all sort of linked together, but then I could, I can shove the data structure anywhere I want. So I can generate data from it, I can generate a figure from it all reading the same structure. Does that make some sense? Yeah, so the, yeah, all of these examples are just showing kind of like, yeah, the input into the, yes, into the actual thing, right? Cause like when we think of a regex, you know, we have the input to the regex function, but there's still like the machinery of the regex function. And regexes are fairly simple compared to some of these other things, but they're still fairly complex. There's still a lot of stuff going on when you tell a regex to run. And so just like you were saying, like, yeah, in this case, like you can apply this kind of David driven style if you're designing a library, but you can also just do it in your own code. It's just like, it's choosing to separate, instead of having everything just as code, I'm going to rip out a bunch of stuff into just data structures that don't have any behavior and then, you know, parse those data structures to generate behavior, kind of like you were saying, like you can reuse them in various ways. You can recombine them. Just this idea that like data is simpler. And even if it's typed, it's structs or objects that don't have methods, like all those things are kind of unambiguously simpler than, you know, a function and code and like a full like, you know, that where stuff is going on and there are opportunities in a lot of situations, kind of in your case, where you can extract stuff into data and a refactor, you can extract it into data and then have something more maybe complicated, slightly more complicated code that like loops over that data and slightly parses it and does the behavior. And any other questions before I stop the recording? Because I have things to say after we stop the recording too, that aren't fit for public consumption. Okay, I'm gonna stop the recording now then.