 Welcome to Warpforge. Welcome, Eric Meyer. Eric likes to hash things. If you can hash something and use it as indexes, then Eric wants to build software around it. And he also has two other talks here at GPN. You might want to check out. And the third talk now today is Warpforge. Which is a kind of, which wants to do reproducible builds. So if you do continuous integration and if you use containers and you ever had a problem with that or it's not as reproducible as you thought it would be, maybe Warpforge will be your solution. Perfect introduction, thank you. Hello, I am Eric. This talk is about Warpforge. It is a tool for building anything. Hopefully fast, hopefully repeatedly with hermatic environments, aiming for reliability and in contrast to some other systems, really paying attention to communicating so you can build things with friends. I also go by Warpforge on the internet. All of the slides are under that name. You can find me on GitHub and Twitter there if you want to bug me later. Hopefully this is a nice new monographic for that name. This talk is going to come in several phases. In the very beginning, I'm going to complain a lot about the state of the world as it is today because we have things that need improving. In the middle and for most of the talk, I'm going to talk about systems for building software and fixing the way we handle build descriptions. Focusing on how we can share things and how we can fix software supply chains. There will be a liberal application of cryptographic ashes throughout. In the end, I'm going to be stuck talking just a little bit about packaging and how we approach that so that it is easier to share things within the systems that I want to live with. There's going to be a tool called Warpforge introduced which is going to be a lot about fixing the bills and the supply chains. And then there's also going to be some conventions around how to package things that are more of a suggestion at the end but I hope a good and interesting suggestion. So, motivation. I use computers. This means I have to build software. I don't actually want to build software half the time, though. I would actually even just like to use software. In order to use software, I end up needing to install software. This requires the software to be distributed in some way. At some point it was built and compiled. Probably I have a problem with it someday so I need to patch it and then I need to compile it again and then I want to use it and share it. There can be a lot of steps in that process of making software available to people. And I wish that these were simple steps. I want to build all sorts of things. And once I've done so, I want to install them anywhere. And I want that to be simple. I want that to involve as few tools as possible. But I think the state of the world today is that building software is weirdly difficult. It involves a lot of manual effort. Usually if you're building some kind of open source software, you're going to go look at the project, read me. Do a bunch of stuff using your human willpower, your limited attention span, reading this and figuring out how to install the dependencies of the thing. Maybe you use some system package manager. Maybe you use some language package manager. Almost certainly you use both. The incantations might depend on what kind of machine you have. It's a fundamentally unscientific process in which you have to guess a great deal and try stuff. We manage to survive with this, but it's rough sometimes. And if you want reproducible environments and reproducible outcomes, especially if you want to try to detect bugs that only appear in some combinations or versions of libraries, good luck. Most of our tools will not help us with this today. This could be improved upon. My anecdote at a hammer this point home is I tried to build something from source the other day and it was rough. I picked a random project off of GitHub that I thought was cool. And I ended up needing to install GCC that I needed to make. A make file tried to get a different malloc library. It asked for root to do that. That was uncomfortable. I discovered I could use the system package manager but it was guesswork to discover that. By the time I got through building this one selected open source project, which is by the way really cool and I'm not picking on the one named here, but this is just like a real story that happened last week. I needed about 15 different things. I needed one language. I needed a package manager for that language. I needed a package package manager. Installing the latest package managers in Node was a journey of get Node, get NPM, use NPM to get CorePack, use CorePack to get Yarn, then use Yarn to get more packages. That was fun. None of those things were a pinned version. I ended up building stuff across multiple languages. Implicitly Bash was used as glue. GCC got involved and I definitely needed my system package manager for GCC. By the time I was done, I don't know how many other C libraries I depended on. I have no idea. It was unclear to me. I didn't S-Chase GCC so I don't know what happened. All of these things should have been versioned together. If I wanted any kind of reproducibility or like a clue what just happened in my environment and none of them were. I think the total count here is about 15. I might have forgotten some. So this is the state of the world that I want to improve upon. I want all of the supply chains, all of the things that went into this process to be identified and to be reproducible and ideally with no action for me. Like it should be a clear log. That's focusing on the build side. Even installing software though, if I've built it, can be very difficult. I tried to install Emacs the other day. I honestly failed. I found that app on my system. I'm a new boom 2 user. Make fun of me if you want. I know it's very common, very basic. It could give me an outdated version or I could install a PPA from some other person on the internet. That turned out not to work. I don't know why. I got tired of debugging it. Or I could find another version of this package in Flatpak. But then it worked strangely because it had some default sandboxes. It couldn't find my actual like .emacs in my home dirt. I don't know what was going on. Or I could use a totally different distro. You know what? I don't want to make fun of Lisp, but installing things should be easy. Installing things should be about as easy as unpacking a tar ball. And that's not the experience that I have today. I experienced complete system distributions that want to own everything about my computer and determine every part of the world in order to provide the first iota of utility. This is really frustrating. We should have more composable systems than this. Honestly, installing something should be open a tar ball. Okay, so these are the dreams that I have. And existing systems have maybe tried to improve the world. Like Linux distributions exist to try to make computers usable. But I don't think that they do. If you're going to use a Linux distribution, you have to buy into everything about that distro. It's a whole machine thing. And this means it's not really satisfactory if I want to have a dream of a universal build tool that you can use freely to do whatever you want and expect other people to come with you as well. I think Linux distros are even quite questionable as install tools because the norm in them is to make sure you only have one of a thing. And I don't want that, right? And I don't think anybody really does. I think NVM appears in order to help people get different versions of Node.js, for example. Like people want to bisect on different versions of things. This is normal. Linux distros fight you, almost all of them. It's really strange. In practice, the reason that this often comes about, I think, has something to do with ELF library headers and silly errata that are utterly fixable. The latter half of the talk will get into this. So what I want to hammer home about distros and what's not what I want about them is Linux distros try to solve problems of usable computers by making everything fit into their own universe and be compatible with themselves. I want to produce software that anyone can use in any universe. And that's a different challenge. Now maybe you've been listening to this series of complaints and think, ah, containers. Containers give us hermatic environments. That solves tons of things about building and distributing. Well, it's a start, but I'm afraid that containers are a necessary but not sufficient sort of thing. Containers are like having a clean room to do science in. And then the first thing you do is go outside, get a bucket of whatever dirt there is out on the street and you pour it into your clean room. And now you try to build. There's twigs, there's leaves. I did actually go get a bucket of mud from the street. You don't know what's in that. And that's very hard then to work with to produce clean, understandable things. Having the clean room, good. Not knowing what you put into it. And then being stuck as many container systems are with a whole, just keep a monolithic image. It doesn't help at all. It also ends up not helping with the install story. Containers have caught on very nicely in like Kubernetes and things that are microservice networking. Because there they're composable. The actual file system part of containers, like if I want to have a series of processes exacting each other, there's no usable composition story here. So this doesn't solve a lot of problems that I want solved. We need more composable systems than this. Now I'm gonna jump ahead to a couple of other comparisons with other things that are more relevant real quick here. Because I think people probably won't listen to the rest of my proposal until they know that everything else has been considered. So Docker, containers. Containment is good. But I want something way more than monolithic images. Another tool that exists already and kind of exists in this build anything domain that I want to explore is Bazel. Bazel is really close to being what I want. Warpforge, this tool that I'm going to propose in a bit is similarly a build sandbox and has like declarative languages for what we're gonna do. But there's a couple of things I want to like ratchet up from what Bazel did. Warpforge is gonna focus even more than Bazel does on being hermetic. We're gonna use containers and we're not going to let things escape them. Bazel has a very easy way of putting like container equals nah in its config. I don't like this. We're also gonna focus a lot more in Warpforge on having a way to communicate partial build instructions and snapshots of data. So my experience of Bazel and watching people use it is that it sort of assumes you have a mono repo. I want a much more decentralized system. I want to let people collaborate even across teams and like without synchronizing, this should be possible. So we're gonna work a lot on that API. Warpforge is also very likely to be compared to Nix which is another functional declarative package manager and like Bazel close, really close to what I want. If you're familiar with Nix, you're probably familiar with the Nix language. Warpforge is not going to have that. Someone in the audience just says, yay, yeah. I'm not gonna pick on the Nix language too much because I think you can just like search for that on the internet already. Warpforge, as if I say, is going to focus on APIs, message passing, like plain JSON stuff. We might introduce a templating language but it will be optional. In contrast to Nix, Warpforge is also going to be content addressed by default and that has huge ramifications that I'll get into in a Q&A section if somebody wants to go through that. So all that said, yes, I'm proposing a new system. I have seen this XKCD. I'm gonna try to make something better anyway. I hope it works. So now onto the actual attempt at solving things. I'm going to offer us the system called Warpforge which is going to be a tool for building. And a set of suggestions called Warpsis for lack of a better title which is going to be a style for packaging. Both of these are going to be fiercely independent from each other even and everything. I'm going to try to build the smallest possible reusable bits and proceed with those. Warpforge, the build tool is going to be about building anything and solving that version together problem that like list of 15 different things that I wish were connected somehow. We're going to try to connect that in Warpforge. The Warpsis recommendations are going to try to solve more of those composable systems requirements. So basically this is solving builds and packaging. And I did have to cram both of these in the same talk because they end up functionally kind of interdependent trying to minimize that but you'll see where it shows up. So first builds. First of all I just want to demonstrate a tool exists. I am offering you working software today not just a series of pipe dreams. But I'll get more into demos later on. First I want to go through the goals of this system and then I'm actually going to drag us through the APIs of the system because I think that's going to be the clearest way to see what it can really do. And then we'll have more demo screens introduced in the middle of that. So the goals. I want zero parameter total environments for my systems. I want to run things with no arguments and have them work. Once I've written some config files but then I version control those, right? I want a software bill of material for everything that ever happens in the system. And I want this to be load bearing. I want the bill of materials of what goes into this to not just be like a log that comes out to the side making vague statements about maybe what happened. I want it to be something that I execute to prepare the environment and then it is load bearing. So we know it's the truth. I want this to be done with hashes as I was introduced. I really like hashing things. I want a completely decentralized system. So every time there's data going in or out I want it to be covered by a cryptographic hash. So I know exactly what it is. No DNS lookups should be involved in determining what is going on in this system ever. I want granular inputs. I want to avoid the monolithic image trap. I want to say where my input data goes. And sometimes I want to say which output data I save. Again avoid the monolithic image trap. I don't want to have to snapshot an entire system every time I want to say, okay the build output directory, just that please. That should be natural. Now the requirements start getting more complicated. I mentioned I want people to be able to collaborate with folks without being in lockstep in a monorepo. So I'm going to need some kind of format in the system to map human readable names to these hashes that are the identity of data that I'm passing around and also to the identity of build instructions. And I need some way to share those. That's going to be fun. I want those things to work at all scales. So they have to be decentralized. I'm going to put more hashes on it. There's going to be merkle trees for days here. And because I want this to be able to work for almost any sort of process, we're going to need graphs and pipelines. And then we just got the soft goals of be usable and be optional actually. I want something that you can drop in every get repo you ever use. And it's helpful to you. Or if you don't want to use it, okay, fine, you can go back to reading the read me the hard way. So these different goals are going to be covered by several different API concepts in the system. These are all headers of chapters that are going to come up. There's also some goals that just require being careful. And there's one more huge goal that I've already alluded to several times, which is everything must be API driven in the system. I want messages. I do not want programming languages. I don't want my favorite programming language. I don't want anyone's favorite programming language. I want APIs. It should be easy to give instructions to the build system by API messages. It should be easy to get results in API messages, plain JSON. I should be able to pipe things from this build system to JQ. That should work. If that works, it should also make the system very easy to debug and easy to have audit logs of. That just seems natural, right? Okay, moving on. That was a lot of goals and credit to every other project in this space because I know that this is competitive to many other things that many other people work on. This is a very hard space, solving all of these goals, holy smokes. So thank you everyone who's ever tried to build software. We all try. Okay, so Warforge has a series of API layers. I'm going to start with the simplest ones. They're the most boring. They'll have the most hashes. Then in each consecutive layer, we're going to add a little more flexibility and utility. So in the beginning, Warforge has formulas. They mean very simply hashes go in, you do something, and hashes are going to come out. It looks kind of like this JSON object. This is an executable thing in Warforge. In the inputs map in JSON, this is a boring one by the way, it only has one input, but that slash there, that's a mount path. You can put more stuff in this map. Content comes in by hash. You can have a section where a bash script or your choice of thing, Exit, there's options here, does something. And once you're done, you can say I want an output from this path, pack it in a tarball please, and put this label on it. This is the fundamental unit of execution in Warforge. And you can run such a thing as this. Warforge run, the formula. So here's a demo that runs around with some CSV. You can see at the bottom, this also kicked out another JSON object. Yes, APIs. This JSON object that it kicks out is called a run record, and it has a section of results where the keys in this map match what you asked for. That's it, this is a simple primitive. Also, exit code, useful. This is gonna be the foundation of everything else that happens in Warforge. These things are at the bottom of everything. If you want the most granular logging imaginable, you just log these things, and you will see exactly what is happening. Because there's also hashes all over the place, it should make it super easy to check anything you're doing for determinism. If you wanted to know, I've lost the screen, how did this happen? Okay, that double time, cool. All right. I'm gonna hope that the formulas thing was on the stream already. Okay, cool, JSON everywhere, great. Moving on. By the way, I did use containers for that. Containers aren't lame, it's monolithic images that are lame. Composable input system makes them good. So, these formulas, these JSON objects, these are the bottom of our universe in Warforge. If you're thinking that they're verbose and that they're limited, yes, that is also the case. If you use just this API level, you would be copying, pasting hashes all the time, and that would obviously suck. You'd have work to get done, so do I. So, more abstractions will come. I don't know why I put this slide in this order, but by the way, there's schemas for everything. That's good, right? The next layer of API up, stepping to like layer two of API in Warforge is something called plots. These are gonna let us do multiple steps of things, whole graphs even, not just linearly. And it's going to start introducing features that will help us avoid copying and pasting hashes. So, it'll start associating human readable names with inputs and outputs, and some more stuff too. So, here's an example plot object. This looks a little bit similar to, it has things that look a bit like formulas in the middle, right? This has two formulas, step one, step two. These are obviously all dummy values, but you can get the idea. These things are wired together by pipes, if you will. So, you might have noticed that- Is my cursor show up? Yeah, great. Inputs in formulas were always the keyword where, followed by a pack type instruction, so like please use a tar ball, followed by some hash. Now, in plots we can use other things in addition to wares. For example, this pipe reference. This one has a empty in the middle here and then the word thingy. It's because it's coming up to the top level and putting thingy in here. Down below in this step two, this pipe is referring to step one, the output called stuff. All right, you see how this goes. You can build graphs. You can also run plots. So, this is a bigger example of doing some stuff with a shell, counting words. And you can run a whole plot and it will run a series of containers in sequence. You can see how the hashes are coming out in the middle and it's basically just getting stamped into the next part of the plot. So, this lets you do bigger, more interesting stuff. Here's an on moving snapshot of the same thing. You can see how this hash is coming from right up here. Cool. You could have glued this together by using the execution formulas and like JQ and some bash glue and stuff, but everybody would have to write that same glue. So, we build it into the system. Now, more interestingly, plots are gonna start using some of the communication tools that are so important to letting people collaborate without wanna repose and so on. That first example just used a where ID in the beginning. Soon, we're gonna introduce another system called catalogs which are going to map a human readable name to a hash like that. Catalogs are gonna get another heading momentarily. Other stuff that, why does this need so many transitions? Plots can also use another system called ingests. Now, this is a little bit spicy because this is going to let you do impure things and it's going to let you look at the host. This keyword at the bottom here. You can use a get ingest, for example, to look at a get repo in the current working directory on your host and get the head commit. Warp Forge knows what get is and it can do this. The key concept of using ingests is that first of all, Warp Forge is going to warn you because it's doing impure things and it's not going to let you go wild with this. And also it's going to immediately turn it into a where. So this will get sort of transformed into the string where colon get colon the get hash. So now it's time to move on to the communication parts and introduce what that catalog bit was about. Catalogs are going to be a data structure for sharing information about both data that you have produced in the system or imported into it, as well as, how's my time? How's my train of thought? Bringing data into the system, identifying it, also tracking, rebuild instructions for anything that we've built in the system. So these are just going to be a huge repository of knowledge. And they're going to be designed as a merkle tree. If you're not familiar with that, in brief, it means a series of hashes where each thing covers the thing under it. Git is a merkle tree. It's why you can trust Gary Pose, generally. So a catalog is going to start with a human readable name for some project. It's going to contain another human readable string for release versions. And then it's going to point to another document by hash. In that next document, it will contain a third tuple of the human readable name. So attributes like source and architecture tuples go in this part of the system. And then finally we get a content hash. There's also some free form metadata that can be inserted in here where this is meant to be an extensible system. In this example, there's also something called a replay shown here which will be how we do sharing rebuild instructions and more examples of that will come in a minute. Since catalogs are about sharing things with other people, it's also going to have to start talking about where to get data. This is a practical reality thing. So you can associate a mirrors document with a catalog and give URLs for fetching individual content hashes or you can say anything under this like glob of name go get from this content addressable bucket. There's a bunch of options there. And this is how we then make these names usable in plots. So when a plot uses a catalog reference, here's your three tuple. You split this on the colon character and then go look it up in a catalog. Now this seems like maybe I did something impure, right? The key element of catalogs is that we're going to make sure that every time you have a plot, that's a file that you put in version control and every time you have a catalog that needs to be in the same version control structure, meaning you're going to have one hash that covers all of these things. Assuming you're using a sensible version control system. So these catalogs can be projected onto a file system. I'm going to start going really fast because we've lost a bunch of time in the technical difficulties. And catalogs are where you can also bridge data in and out of the system. For example, there's a bunch of sub commands of Warp Forge which will let you look at a get repo, get all the tags out of it and instantaneously transform those into a catalog. That's cool. Here we're doing it on Linux. All right, recap of the API structure so far. Formulas, do one thing. Plot, do a bunch of things. Catalogs, start communicating. The other thing we're going to put in catalogs is the replay data structure. These are to communicate builds that you have done before and let you do them again. This is actually really easy. I don't even need to show a new API object for this because it was already plots. If you use any ingests, that would be unreproducible. So we're going to freeze those, but then it's already a replay. This is something that we do in practice if you want a get repo for your source on your host, you can say, yeah, use that. And when you're releasing as long as you put an output in the system that contains that same information, we can just automatically freeze that into a self-reference. This lets anybody else, whoever wants to reproduce this run, do so from the information already available in the catalog because that will then be the get content hash. These replay files go in the catalog along all that other data. And something I want to highlight is cool about how this works, especially this is contrastable to other pure functional build systems. Catalogs are based on this concept of snapshotting the data and having the hash of the data and then associating the replay instructions with that, which is cool because you can actually have cycles, right? We kind of just saw one right here. You can make a cyclic reference and that's fine because you're snapshotting it after the cycle is completed. This means we can explain even build graphs that have cycles, which is really cool if you wanted to say, fix the trusting trust attack of compilers. Replays also, of course, let us build stuff. If you're missing some content and you want to recursively build it from the replay instructions, you just add dash R to the run command. It's great. All of this stuff, when you're running it in practice, takes place in a workspace, which in Warp Forge is basically any directory that has a dot Warp Forge directory. This is where the scope of name lookups are. So that catalog file system, it goes in this directory. Workspaces can be nested for practicality reasons, so you can have one root workspace, which contains a bunch of catalogs that you imported from the internet. And these can be used as auto-complete suggestions or so on. And then our recommendation is that you should put another... No, that's not the slide head. Every Git repo, every version control root should itself also contain a workspace because we want that catalog snapshot and the module to be next to each other, the plot next to each other. So we have a vending tools that can take all of the references from your root workspace and copy them into a smaller one. So when you're using the tool in practice, you can ask it for the status of some module like our build instructions for Python and it will say, yeah, you have these references to these catalogs. I would love to explain these things more slowly, but I'm quite worried about my time. So put these in a Git repo, you're off to the races. It's a vending of lightweight metadata that's gonna let you reproduce stuff. Now, you're still writing lots and lots of JSON and if you want to do lots of things very efficiently, you probably want to write less JSON or you want to have some reuse layer. Now, at this point, I'm actually going to give up a little tiny bit and not necessarily solve it totally in warp forge because I want to avoid having my own personal opinions about programming languages, reduce the number of people who can share this tool. So bring your own JSON object generator, please. Or okay, yes, I'm going to include at least one batteries included answer. Starlark from Bazel is actually kind of cool. It's a Python dialect. We're going to introduce support for this into the core of warp forge. It's going to be non-special. It's just going to emit JSON objects. So if you want to build a competing system, you can just do that and I'm not going to fight you, but I want something batteries included. This is still a work in progress. So this syntax is wildly non-final, but the idea is having a language gives you some degree of being able to reuse concepts and components. Yeah, so in recap formulas, all you do basic stuff, plots and these other structures let you do graphs of it and then catalogs let you communicate it. And then at some layers above this, you can introduce whatever kind of templating you want. Maybe I'll offer some Starlark and that's the batteries included easy option, but anything you want to do is fine. As long as you're producing these structures like plots and replays, the whole system can still introspect, understand what you're doing, produce good audit logs, et cetera, et cetera. There's a couple of other cool things like we have quick commands for like, oh, you want an interactive shell? All right, fine, we can have this. You can have local host CI in this system basically for free. You use a git ingest. All of this works rootlessly just for the record. You never need sudo for any of this. Now as I continue to run low on time, briefly packaging. Yes, we're starting to package things in the system. Mostly I'm trying not to care about any details of that except for one thing. I want binary packages to be path agnostic, by which I mean if I put them in some install folder, I want it to work. This is a handy attribute because it makes it easy to compose things with this kind of JSON. And it's also a very high and mechanical sympathy to containers in general. I want zero step installs. I want no post install hooks so that when I put an application in a file system, it should be a read-only mount and it can go. That just makes sense to me from first principles but it's also fantastic for container usage because it means you can use a bind mount and just go to space. If you have things that need to be moved around on a file system in a container, an overlay FS will cause all sorts of invisible copy operations on the bottom. Things get really slow. If you can have application installs that you just mount and you're done, you are a happy person. If you've packaged binaries in Linux before, you might know that this is harder than it should be. ELF headers are kind of nuts. I'm not going to explain every part of how you fix this but long story short, there is a way that you fix this. You can play with ELF headers in Linux such that you can have libraries be relative to the path of the binary. If you follow this to its conclusion, you can have directories that you can mount in any position and they will work. This is a way to start building reusable components that do not need any support from the system around them whatsoever. This is a way to get out of the trap of distros building into their own universe. I've given a talk about this before as well back at All Systems Go in 2018. There's a much longer rant about the needs for this and the effects that it will have if you want more information. Or you can follow these links to like working implementations. There's one more thing that comes up if you're stitching together lots and lots of applications each in their own directory like that. At some point you need a path. The thing that your shell looks at to pick where a binary comes from. Use a Simlink farm. This is a solvable problem. That's it. I don't care about anything else in packaging. I just want things to work together by being isolated and it can be done. Okay. All of these systems that I have introduced are early. There is working software and if people want to come to those links and like get binaries, try them out later. Please do. However, it's early. We're trying to build APIs and like sort of functional spaces where people can get together and build stuff in the future. As demos, there are some packages for like GCC and Python floating around, but there is so much more stuff to do to make a practical environment with lots of packages. If anyone in this room or anyone watching would like to come along and help with that, please. We have lots of future work in the tooling, the Starlark thing, like I said, is coming up many other things that we would like. And I would like to package more stuff as well. We are currently resisting building a distro and I hope we continue to resist doing that. But if we follow some of these packaging recommendations and we can produce things that are reproducible, hermetic and easy to rebuild with Warp Forge and are usable everywhere, gosh, that would be cool. So anything that anyone sees in here and they want to help make it better, please come talk to me later. It will take many people. Hopefully, if we are successful in this project, it will produce a cultural shift. I want reproducible builds to happen. And I hope this tool and the way that it exposes hashes and makes things visible and debuggable will actually help us get there. Not necessarily by solving reproducible builds themselves, like patches to everything that has some random component is necessary to fix that thing. But this tool chain should help make that visible and help make it fixable. I'm hoping that this eventually makes computers easier to use and I'm hoping that these packaging conventions mean more people can share more stuff and honestly make the world a better place. I think I've made it through in a halfway decent time. Does anyone have any questions? Maybe, how is my time? Your time was great. So we have 12 more minutes for questions, which is more than we expected. This is great. You made up time great. So yeah, raise your hand if you have a question and I will pass you the microphone. Yes, you can. What kind of tools or languages are you using to build this? Oh, good question. I also think you know about the same language. Yeah. Is the mic, can you throw me the mic or something? Sorry. Okay, try again. So what kind of tools and languages are you using to build this? So all of the Warpforge tooling is built in Golang at the moment. I try not to treat that as too interesting of a choice. But lots of things in Golang are very practical. It's also very easy to get reproducible builds out of the Go compiler. So we have Warpforge building Warpforge with a consistent hash already, which is very pleasing. The container system, we're reusing RunC. It works. It's the same thing that's at the bottom of Docker stack in practice and almost in install. But RunC is just the containing parts and none of the image transport parts, which means thank you, whoever isolated that API. It's like very correct. Okay, awesome. Hopefully also lots of things are pluggable. So like where the hash said where, call in tar, call in plet. There's a whole plug-in system there for these transport and hashing systems. So if you wanted to plug in somebody mentioned IPFS earlier, that should be easy. There is like a switch case where you can start dropping more code in. Any more questions? Yes, please. Hi. Is it on? Yeah, it is. Thank you first of all for this quite interesting work. And the first thing I was wondering if I want to start putting my tool of choice which I wrote into the system. Do I really have to make packages for all of the dependencies including GCC and all of the compilers? Because I think it kind of comes down to like everything should be built in that system, isn't it? So you can use the system in a pretty degenerate way of just like do the whole monolithic snapshot thing like other container systems and then build. If you want like a quick escape valve to let you do something without thinking about it too much, you can do that. And of course we're doing that like constantly as we bootstrap things as well. I'm hoping that we will build a larger ecosystem of packages very quickly that will make it easier to do more stuff faster. Like we do have a GCC and a Python already for example. We even have a path agnostic Python by the way. Which is really cool. I don't think anyone else on the planet has that. But yeah, inevitably there's going to be a lot of stuff that needs to get built in order to have the nicest possible experience. I hope the fallback to monolithic snapshot helps people accomplish things and increments as well. Okay, more questions. Yes, yes. So at least if you are using something like tend to you probably ran into issues with like GitHub changing the table beneath your wall. But basically, and it obviously breaks, hashes. You got any solutions to that one? Because one thing which could work for example is not relying on the table GitHub for example provides but doing a shallow clone of the repository itself or something. Yep. Yeah, I remember the day that GitHub changed their hashing algorithm or their exact export algorithm. That was fun, good times. There's a lot of different ways you could solve that. I think just mirroring the actual binary data is like always your ground truth. This can never hurt you sort of choice. The transport plugin system is also part of my answer to that. So this tar system we're currently using actually does a semantic hash of the tar ball believe it or not. I wrote that code. It means that you're immune to some of the details. I don't know, you're giving me a thumbs up but I actually don't know if I regret this or not. It's more code to maintain and that's worse. But it does mean that we're immune to some things like the tar file order. The actual hash in that system is based on the syscalls that I give to the file system. So it's correct and nothing additional. Nothing about the compression algorithm either for example. But yeah, at some point you just have to have the data available, right? We're gonna use cryptographic hashes so if you can't keep that data around at some point you're gonna have fragility issues. Yeah, since we're on the roll, more questions. Oh, yes, yes. Also, you can repeat the questions that the stream didn't hear. How would you compare Wobforge to BitBag from the open embedded ecosystem? From compared to what? BitBag. Oh heck, I'm less than familiar with that one. You should look at it because it's basically similar in many, many ways to what you're building here regarding repeat usable builds using hashes to exchange artifacts to see which parts change and which parts can be reused. Cool. If more people are trying to approach the same goals that's fantastic. Mostly what I have seen from other systems and I'm not sure if this is true of BitBag is that people are working on having build trees and there's less emphasis on the communicable subcomponents and like being able to detach whereas something that's very unique about Wobforge is every time there's a catalog, you can detach. Those are like points where you can collaborate with other people asynchronously. But I'm less than familiar with BitBag so I'll have to research that later. Okay, we might have time for one more question if there is one. Yes, last question. Maybe it's not really a question but an idea. I think you have a lot of work to do building a very large catalog and maybe you already came across the idea but did you ever thought about there are a lot of distributions which already have a lot of recipes how to build software and trying to invest into writing a software which converts that build recipes into your recipes? I think it will be kind of hard because most of them are having like programming languages which are too incomplete, that's a little bit annoying but I think maybe you can make a first guess and then get a recipe which like 90% works and then you do some hot fixing or things like that and then you can build a large catalog fast. I've definitely not tried to automate things at that tier but I will say that there are some distros that I enjoy looking at the source code of a lot more than others. Looking at Arch is usually a fantastic source of inspiration. It's bash, oh no but it's usually very linear and you can tell it's going on very quickly so those have been generally inspirational but usually you still end up with some manual transformations and I have no idea what we're going to do once we get Starlark in the system by the way. All the stuff we've packaged so far we're writing JSON files and so our copy pasta level is still like a little bit high like there's certain incantations for how to do the R path twiddling and then the LD shim thing that are like manually copy pasted right now. So I think we're going to end up with some sort of functional templating system to do that and I don't know what it's gonna look like yet. So there'll be lots more to discover once we get there. Okay, maybe one short more question if there is one. Otherwise I think this is a good point to end the talk and give the next people more time to prepare or at least some time because we had the technical difficulties. Thank you Eric, it was a great talk. Thank you everyone for the excellent questions. And thanks everyone for bearing with us with the... Oh, I had one more slide that I should flash. Here's all the links to all of the things. You can get more documentation at projectname.io, warpforge.io. Source codes here it's under my personal username on GitHub. Here's where we're packaging stuff so far so there's a bunch of JSON files there and if you want to chat, please do. I'm a Matrix fanboy so that's where I'll be but it is also bridged to the same name on Libera if you're IRC people. So please come hang out. I'll be lurking there momentarily. Thank you again.