 Everybody, welcome to my talk about flamenco for the schedule. I was asked to shorten the title a little bit So it fits better on the paper But it was actually titled simple simpler better faster stronger distributed rendering with flamenco version 3 My name is Sieglen I am software developer at Blender headquarters and Since January we've been working on a new incarnation of flamenco And that's what this presentation is about Please keep questions until the end. I hope we have time for for super interesting questions, but I think The answers will make more sense once you know the bigger picture So to give a bit of an overview first, we look at what is a render farm for the people who don't know yet What changed since the previous incarnation of flamenco? We'll do a five-minute install and getting things running We look at then and that's why they're a bit separate a little bit more in-depth on two topics the storage system and the job time system and Then I will tell you that we want developers to get involved So first what is a render farm? typically thing if people think of this as like a complicated thing that's high-tech and difficult and then you think of something like this I Was actually looking for like farm pictures and stuff and I found that in in Rotterdam in the harbor There is a floating farm and if you want to know more it's floating farm.nl. I think it's a cool project And and so with flamenco you can do something like this, but also You can get something like this The flamenco is already useful if you only have one computer and you just want to queue up your render jobs And go to sleep and the next morning see what failed So Compared to flamenco to what changed? Well, maybe you were there some of you were some of you may have seen the video In Bledner conference 2019. I also gave a workshop for 60 minutes and we barely got things working and That that was clear something had to change in in the structure of things in the simplicity of things So for the new flamenco, we really sat down and started talking about what are the driving principles behind this new new incarnation and Number one simplicity Every decision that we made in the design of blender was mostly driven by simplicity and That started by taking out all kinds of components But it ended up also like not even requiring like a postgres database or something to store stuff in First I was arguing like against that. Oh, this is easy. Just postgres is simple. You just have to get install postgres done and Then I try to install it on Windows And I figured out like how many times you have to click next and Within the 500 next clicks there is this checkbox that you may or may not have to check I still don't know what it means and then somewhere in between you have to choose an admin password that you have to remember And that you will never see ever again and I Got confused. I got scared of the process. So now I read Let's give it simple. No database Also, it should work offline No more connecting to the cloud No more sending stuff We know don't exactly know what to exactly where Everything has to be local. It has to work when you're offline when the internet connection goes down your your farm needs to keep working Transparency is all about it should be simple to find out what the thing is doing in One view you should have the status of your farm and which jobs failed which job succeeded which are still in the queue and That leads us immediately to interactivity like that overview should be interactive it should show the current status and not The status that it happened to have last time you refresh the thing And then in the end there's robustness of course like it has to work it has to be stable But also when things crash which will happen because your scenes will be too heavy for that one machine It still has to keep running and then we started looking at what is our target audience? Who do we make this for? Well, of course for blender users But maybe not for the bigger studios as ton also said in this keynote like Hollywood isn't really waiting for us and There are so many more small studios people at home like One to ten artists working on something and this directly connected to the simplicity so Flamenco doesn't have any authentication because it assumes that like you can trust your colleagues and do not mess up and and Not having authentication not having to set up an authentication server and create accounts and manage all that may think so much simpler Also when it comes to the number of computers in the firm Yeah, roughly a hundred. I mean we have tested I think on 12 or something Hundred will be fine. I think That order of magnitude but not like 10,000 cloud machines all running at the same time For file sharing just the stuff you could use at home. So in our studio, which is mostly Linux based is NFS But Samba or safe stack the Windows file sharing should also work We kind of assume it's all on one network Probably it will work. We have a VPN, but that's up to you to to complicate matters that far if you want and Then finally Linux is really our platform of choice It's open source. It's what we use. So it's also what we test with But macOS and Windows should also be supported This was the Architectural overview of Flamenco to have your own computer and they have a blender cloud with multiple things And then you can have my hook multiple firms up to that thing and then you can have multiple Databases and workers and everything This is the new design Yeah user artist runs blender has an add-on that add-on talks to the manager That manager is a process that is running somewhere on your on your computers The manager talks with the workers instructs them what to do what not to do and all of them Can communicate with a shard storage so that is where your your vendor jobs will go whether you render output goes But maybe also where the the worker executable is located so that you can just shut them down Put a new executable in Put them up again, and then it all works And this is what it looks like beautiful weapon to face designed by Francesco and by Pablo and I think they did a brilliant job at it And everything that you see here is responding in real time to changes so you don't have to refresh anything The colors change the text change if you click on follow task log at the bottom It will pop up and it will just be scrolling and showing you what blender is logging And we made a system that can That tells the back end basically what you're looking at so that doesn't send you all the info of everything Keep things a bit scalable So it's a bit of an overview of where we are now and how we came to be here So let's see if we can get this thing running We'll follow these steps except for the gray ones because I already did that So I already downloaded flamenco from the website. We'll pick a storage which is see temp. I Already have blender on my machine, but we'll start from starting the manager And now I will duplicate Monitors, yes, so you guys see what I say So here is this readable should I increase font size? It's okay Good This is what you get when you download flamenco So you have flamenco manager and flamenco worker One thing I still have to work on is getting a nice icon in there, but that's for later I've read me a change log and The directory with tools that contains FFM bag again for the simplicity of not having to Download FFM bag for somewhere else and then have the right version that's compatible with the common-line options that we use and just Plunk it there. You start the manager and that shows you stuff and then opens a web browser Gives you the setup assistant explains the structure of the thing which we've just been through Well, let's go path to a shared storage See term Flamenco usually this would be on a network drive somewhere For now this is simpler Then choose which blender and because my laptop happens to run Windows We have some code that checks which blender is associated with dot blend files So the suggestion you get is what you would get when double-clicking the blend file Again for simply if we want to have this for other platforms as well But there it was a bit trickier so we didn't get around to it You can of course point it to whatever blender you want to use But for now, let's just use the de-text it one Yep, confirm and now we're there because there is no render job yet It presents like hey, you don't you're just starting so I get the add-on The add-on will have to know where the manager is running because it could be on this machine It could be on any other machine, so I need to pass it the address which is here. I just click on it now It's on my clipboard Get the add-on this doesn't go to the internet But the add-on is baked into the Flamenco manager executable Which means that whenever you update flamenco you get the add-on with it The only thing you have to remember is to upgrade it so I've already prepared a beautiful animated Suzanne and here we paste in the URL Tells us from ankle 3.1 has been found and it also tells us where the job storage is located And this is another one of those small small improvements that we made with the old flamenco You had to copy paste that information everywhere now the manager tells the add-on look this is where storage work Where files go so that's good I've configured it to render PNGs output doesn't matter because flamenco overrules that anyway And now here we have a flamenco panel Job name becon 22 Priority determines like when multiple things are queued which goes first you can edit that after submitting by the way and Then this button is very new and it's something that was so clearly wrong Well, it represents what was so clearly wrong with the structure of flamenco 2 If we wanted to change how The jobs were structured like I want to render this how does that translate to actual render tasks that need to be executed? That had to happen on multiple levels and multiple like in the add-on if we wanted to add something new and Then in the cloud and then also maybe somewhere else and because it had to go into the cloud as well That means if we change something it changed for all the users of flamenco Which was really annoying because we didn't dare making those changes because it could be really disruptive for for other studios But also studios themselves couldn't adjust to flamenco to their needs So instead this button fetches the job types that are available on our manager And we'll look later at how to extend and then change those and It gives you a number of options like which frames it defaults to the current scene The trunk size is every time that blender starts to render a task. How many frames does it render? the render output route See temp flamenco renders. That's all right then at path components is for Adding path components of the current blend file to the render output path so if your blend files are stored in Shot name slash or scene name slash shot name slash whatever dot blend and you set this to three then the output will go to slash shot name slash scene name slash whatever and This is what it would look like so the you can't edit this property But it shows here. It's like the storage slash beacon 22, which is the job name Then you get a timestamp, which is a magic string and then the the frame number submit to flamenco Job submitted and now you can see it's queued The hey, you don't have anything yet get the add on here is gone And we can see that it has no preview yet. This is the info You can also see that that magic string for the timestamp has changed it into an actual timestamp so you know where it goes and we have a bunch of Bunch of tasks now double-click the worker I Tell windows is fine. I made this software run it. It's okay Now it should Find yes, there we go. It automatically found the the manager even if it were on a different Computer it would still find it. You can see that stuff is already rendering There you go more frames are coming in you can see everything live updating and The final thing that's queued now is rendering a preview video Which is now also done and here in render speak on 22 the timestamp. There we have everything and the preview video and That is how simple it is now like last time 60-minute workshop barely got it working now click the click done I'm super happy and thrilled that we got that far. Oh By the way, if the auto detection doesn't work off the work in the manager on the manager machine Stop Spotify and then restart the manager because Spotify is blocking that port that we use So now let's dive a little bit deeper into those two topics and storage is a really big and really important one Basically what you want and you want it to be simple of course first principle should be simple You want it to be efficient? You want it to be isolated that every rendered job has its own isolated set of files Yeah, that's what where we are now like you have to pick two So these are our storage option you can do what I just did now where the I didn't show that actually But you'll trust me on it The blend file that we opened was already located inside that shared storage This is a common way in which studios work. Everybody just works on the same set of files on that and ask somewhere and They want to submit to flamenco It doesn't need to copy anything because everything is available on the shared storage already So it's super simple. It's super efficient because nothing get copied But you don't get that level of isolation that you may want So with this if while rendering somebody changes the texture half of your render will have to the new texture Might be good enough The second one is how blender studio actually used to work For quite a while and that is also very simple for every rendered job. Just copy all the files We have some analysis going on that sees which files are necessary for that particular job and Then just copy all of that into the job directory and then you have your isolation as simple it works and we use it for years But it may Andy and some other people very unhappy Because like for spring one of those big tall legged monsters was one and a half gigabytes And we had five or six of them and that got copied for every rendered job So that's why it's not efficient and Then the last one is what I will dive in a little bit more. That's a shaman storage system It's not so simple, which is why we have to dive into it with other sheets instead of just showing here But it is efficient in terms of storage because you and and it is isolated Who shaman do I know him? Of course named after him But also because that system identifies files why they're Sha checksum the only thing you have to upload to the farm is anything that the farm doesn't have yet so new files and A changed file has a different checksum and thus it's considered a new file because that's the only thing the system knows about it So you upload everything the client the the flamenco storage doesn't know yet and Then once the rendering has to start flamenco will create the directory structure for you and create symbolic links to all the files in the storage and that is super lightweight and Still gives blender access to the files in the location where it expects it and your render job can continue So this is an overview of what that would look like the dot at top is the root of the shared storage And then you have a file store with all these blobs Because the file name itself is insignificant And it's just a long numbers and the file name itself is the length of the filing bytes because why not and Then below you see the shaman demo render job And those are actually the files that are necessary for that particular job You can see that they're in a different color because they're not actual files But symbolic links that point back to all these jobs and In this way well we have all the property well almost all the properties except for that super simplicity that we wanted This works super well on Linux with NFS. It's no problem Samba has certain extensions that make it understand symbolic links as well But windows is still lagging behind a bit on understanding the usefulness of symbolic links so there We're running against some some some barriers The good news is that this system is technically only necessary between the manager and the workers So if your storage supports it and the manager and the worker are on on Linux, it's all good The artists they send in the files via a web call just HTTP To the manager so they don't need to know about this system at all So if your artists are on Windows and the rest is on Linux, it will still work for you That's one thing The next oh, yeah, oh, huh. I missed that one in my list of things. I wanted to talk about But this is this is connected to the storage the variable system and the variable system is there to Allow you either configuration But also multi-platform support So for example, there's two variables that are supported by default That's blender and you can say well blender for a Linux machine is there Blender for a Windows machine is there and for macOS. It's there The the plot the the operating system names are just strings So if you're running free BSD or blog OS or whatnot You can just type that in and it will it will work and You can also just not type in anything for the operating system and then it will be for everything So blender args is the arguments that will be always passed to blender by default this is the default so run in the background and Say yes to running Python You can also change that if Say during a project that you're rendering with ev for the first time in the background on flamenco And that new GPU subdivision for the viewport that doesn't really work. Well, you can just turn it off This is a variable system When we look at the job types We will see where that blender and blender args actually comes from because those are used in those job types They're nothing really special about this except that flamenco comes with default values for them for the default job types But if you as a TD you want to have your own variable system like your own variables and values and use them in your own job types Completely free to use the system Apart from this you can also specify an audience and the audience is workers or users So if you want to have some variable when a user looks at it for example in the web interface Have one value But the workers who might be running in the cloud with all different infrastructure need a different path then that kind of thing Is also somewhat supported a little bit better in flamenco 2 Still need to port it to flamenco 3, but the idea is there So these are one-way variables Whenever that variable blender or blender args is seen it replaces it From supporting mixed platform stuff we have two-way variables and They look like number of variables. They have a name. They have a platform and for that platform. They have a value Except that these go both ways To give it an example of that. Let's say you have an artist working on Mac OS Submitting a file that is on volume shared from ankle short file the blend and your manager running on whatever OS Will store this Because it recognizes that that prefix is the macOS value of that variable So it will store it as that that variable and then when a worker running on Linux gets it it will replace it with prefix for Linux for Windows and because the system is made for path translations It will also flip the slashes the other way and that seems to work quite well then we get to the job types and This is really at the core of of of flamenco of how to translate like which parameters to show to people when they try to submit a thing Etc. So it's defined by a file on disk it's the JavaScript file and We've already seen it so within that flamenco 3 panel. I chose the simple blender render, but there's more There's also the echo sleep test This doesn't do anything with blender it just Echoes a message to the log and then sleeps for a number of second a number of times But it's great to test stuff with and to explain things with because it's so simple So this is stored in scripts slash echo sleep test.js If you want to override it you can just next to a flamenco manager the accent You can create a scripts directory Plonk a file in with the same name and it will override the built-in Job type or you can give it a new name and then it you will print your own job type So this is your script file with a tiny hint of Python We'll see that soon, but it may get a little bit schizophrenic if you're not used to mixing languages in the same file but it defines things that are shown in the blended user interface and They those can contain Python expressions that are evaluated within blender itself So this is that file and all of these job types consist of two parts Inspired by the blender add-ons that always start with a be all info dictionary at the top This guy always starts with a job type dictionary at the top and then there's a compiled job function That gets the job and that needs to do stuff. Is is this font size? Okay, by the way Good. So here you can see the things that we saw on on the lenders interface You have a message that's of type string. It's required The sleep duration is seconds. It's integers sleep repeats is integer and you can also pass more options there Which we'll see later And this like this is the thing that I find so nice now that you can on the manager You can configure what kind of parameters a job has And then compile job all they all work in the same way roughly. So you get the job settings from the job You create your author some tests They the author dot task function has two parameters. The first one is that the task name Completely up to you. Whatever is sensible to you. The second one is the tax type This is a freeform string also again. What is useful to you? But there are a bunch of defaults like misc and file management. I think an ffmpeg and blender and With those you can configure which worker can do which kind of tasks So maybe you have a machine that can't really really run blender But it can do file management like moving files around and renaming things or maybe can run ffmpeg and with these kind of things You can configure that It's not a huge system, but you know, it kind of works So you author the task then you add a command to it Task consists of a bunch of commands one or more if one of them fails the entire task fails That's kind of the idea and they're running sequence And these commands are what is actually implemented in the worker So the worker knows how to do an echo command and which parameters to expect and same with sleep Then if you want to have one task depend on another you can just tell it in that way so first we want to have the message logged and then All the sleep tasks are depending on that, but the sleep tasks themselves can be running parallel on multiple machines I want you have that task authored. You can add it to the job And that's it. Then you have your render job or your Debug job Let's look at a more complex example the rendering This is the job type and you can see it has a lot more parameters than you saw and in the interface So the simplest one is frames. It's just a string is required because Flamenco needs to know which frames to render or which not And you can see it has an evil key. That is just a point in expression That will determine the value of that thing You can still override it. You can you can type anything you want in there, but if it's empty Then it will automatically fill it in if you press the re-evaluate button It will overrule what you typed in and replace it with the evaluation And here we have the render output root and the number of components to add from the path And you can see they're visible only at the submission level because once this is submitted And I would really care if you just want to know where the output is going to and not really how that was constructed This is a way more complex thing that Does all the thing you saw to combine those paths and this is a compiled job Oh, by the way, the rest is also it's now hidden, but it's all Grab repeats of the same thing So poison expressions that get evaluated get attached to those job settings that you can then later use in the script To give a popular the answer to a popular question if you want to change Where your blood your render output goes? Maybe you want to include the scene name Maybe you want to include the camera name. Maybe you want then then this is the thing to to adjust If you want to render it from a specific scene Within that shared storage you use the simple approach without the isolation You may want to include the scene name in there and then pass it to the blender So let's look same thing compiled job function and This is for the render task. So this loops over This loops over the chunks of frames The frame chunker that is kind of a more complex thing You don't want to write that every time. So that's built in and this is what you would alter if you want to change the the Blender arguments, this is the executable that should run these are the arguments that should it should have Then you can define some arguments before the blend file the blend file itself and the arguments after the blend file And this is also something that is new in the new generation You couldn't specify except the deck you couldn't set the define whether it was before or after the The blend file So that is a job type system What we haven't seen, but I will just briefly touch upon There is a worker sleep schedule. So if you have artist machines That you want to use as a flamenco workers Well, you can input the work schedule of the artists these days these times and then flamenco will sleep and then when they leave Flamenco will wake up and start doing stuff There is for failing workers. There is a whole system of block listing and auto-requeuing We have situations where one computer was too tiny and puny for that particular shot and it would take a task Start running it crash fail task. Oh next task Fail and it would run over all these tasks and and fail the entire job When that happens now It gets onto a block list. It's no longer allowed to run that task type on That particular job. So even when it fails running blend Oh and all the things it failed before will be recued for other workers to to pick up But if it fails blender rendering, it's fine to still run an ffm peg or move files around And this is also where those task types come in and finally we the the core of the new flamenco is an API based on open API Which is a formal description of these operations have these URLs this expected input these possible outputs And you can feed that to a code generator So the flamenco worker generator is just built on top of generated code for that API in go The add-on is made on generated Python code for that same API the web interface is Java script generated for the same API So if you want to like interface with flamenco in any way This is the easiest way to do it So as an overview we've seen what is a render farm? What has changed in five flamenco since the version two? We've done a five minute. I think it was even way less than five minutes like super fast install We looked at the storage system the job types the variables and I've told you that I will be telling you that We want people to get involved So I want you Thank you, and I've been talking for 35 minutes, which I think gives us 15 minutes for questions Why is the executable science 15 megabytes too big or too small? It's very big. It's because there is the add-on in there, which is huge. No, just kidding No, the whole web interface is in there the like bunch of other files are in there one of the reasons we chose go for Making these executables is that it's a statically linked executable. So there's a lot more stuff in there than for Dynamically linked executables, you will typically see But also all the assets are in there for the web interface all the alpha script all the images all the CSS the Add-on the Java script code itself for the job types, but also it contains a complete Java script engine So the app scripts running inside of the manager for all of that combined is this size Good Is there a capital priority range? Yes Correct, and the higher priority is the higher priority. Yes So is there any scripting knowledge required to get this running no no you can just Like in the in the quick install you saw it just it's just work if that works for you Good use it. Don't think about it. If you want to modify things. Yeah, then you may need to get into the depths of things so Good question. Is it possible to distribute a still image? So Cut up an image into multiple blocks and then have different tasks for different blocks Technically, yes So you will have to write your own job type for this So it's not one of the builds in job types. It's really made for for a video rendering But I would love it if someone wrote that I got I'm now more focused on animation and rigging than on flamenco I still want to support flamenco when you come into the flamenco chat channel I will definitely help you out and answer questions there But development time is limited which is Why that? But I think it should be relatively straightforward as long as you can figure out how to work which command line arguments to pass to blender to make that work to to it will basically set the render boundary to that particular image or that particular part and Then you just have a for loop over all the possible little blocks and generate the tasks And then do some magic with another blender insta instance that has some node groups or something to push everything together into one image and with a dependency system that is very much possible to just Run all of it and when everything is done run that final merging task Yes the question was how do you tag those different abilities So I'm when like it's in the dark You're not seeing me In the documentation it says which file is where On Windows. This is an update a local blender foundation flamenco, and there you have flamenco worker escalate flamenco worker No, I'm talking the wrong thing. This is useful. Not what you asked for a File sitting next to flamenco worker would could could have that So you can next to flamenco worker.exe you can put a flamenco worker.yaml with configuration file. It's in documentation like Which structure that should have? And there you can put it Is it possible? Yeah, is it possible to run like multiple workers on one machine? Yeah, absolutely I've heard people who Had scenes that blender wasn't parallelizing particularly well at least not to their taste So they just run like two or three workers so that it would run two or three Task at the same time. That's very possible The only thing you have to take care of is that well that file that I accidentally went to That is location of the credentials file and that credentials file determines the identity of that particular worker when communicating to the manager and Of course, it's a fixed location. So if you run that worker three times The manager will think it's the same worker. So you have to set I think you have to pass a committal on argument or set an environment variable. It's in the documentation That was also one of things for this system. I wanted to have better documentation and we do so that's great Yeah, what yes, so the question is would it be easy to just have something outside of blender sending in stuff? Yeah is for the shaman you need to know the shaman protocol and and Use that to send in but just sending it saying to a flamenco. Hey, there is the file go render it Yeah, that's super simple. I Think I can show a little example of that So this is basically a curl command that sends something to the farm. So you could use this or you could use the Just take a copy of the Generated pricing code from the add-on or run them with the the pricing code generator yourself And then use that API. I would do that rather than write this But this is just an example of how simple things are on the water. You had a question Yes, oh, yeah Totally how long does the work? Okay, so I will save in the microphone for the Recording your first question was whether it's possible to split up frames not by by blocks, but by sample count Yeah, totally cycles has options for chunking the frame at the Sample count so you can say I wanted over divided over 15 chunks now render chunk number one and on the next tasks Trunk number two and then again a final task to combine the things Second question was how like what is the life cycle of a worker? Typically infinite until you stop it So at the studio we integrate it with system D that will automatically restart it once it shut down So when we have an upgrade, I just Overwrite the executable with a new version I tell all the machines to to shut down through the web interface of flamenco and then they automatically start up again So even I'm searching through all my tabs to Flamenco so even when When asleep So when I say now Go to sleep the process is still running So that it can respond to wake-up calls But it won't do anything Also something that we I didn't explain but this list of actions That's pretty much the same as in flamenco to so you can tell it to shut down You can tell it to shut down like now or after you've completed your current tasks So that you don't lose any render time and the same with going to sleep so I can now say well shut down and There we go and now that window is closed and we only have the manager left Yeah, okay, so the the question is how do you ensure that the version of Blender you use to submit the file is the same as the one running the render job With flamenco That's up to you So that's not a feature that is supported You could set it up by having a job setting that Takes that version puts it into a job setting and then instead of just putting in that Blender variable you could use that version to change the variable name and Then within your list of variables you can say well Blender 3.0 is this Blender 3.1 Is that Blender 3.2 is that and then point it to the executable so you can make it work? But it's not like an intrinsic feature of flamenco. Yeah, this is what we do also like Yeah Yes, how easy it is to extend to user interface like the the web interface Custom without like recompiling your own flamenco. It's not possible except for the the job-time dependent bits so like the the job settings that you see here that You can define whether settings are visible here or not, but that's about it Yeah, the ones you've seen Yeah, that's it. Yeah If yeah, and again if you have interesting job types or Written out and you think oh, this is something that is generically usable like tiled images or splitting up by samples I'm really interested in like bundling this Yeah, good question. Where would errors end up if that JavaScript file contains errors? Depend a bit on where It could be when submitting a new job Because that actually runs that JavaScript and if that throws an error because you have a throw don't do that I'm incompatible with video rendering because I'm made for PNG files like we have here That would show up actually in the blender user interface So it shows of course in the the log output of flamenco manager but it also shows in the web interface or sorry in that the submission interface and It kind of nice because it also tells you which line on which file and that kind of info Oh, good question like will there be packages like in Linux packages in Windows installers that kind of thing Well last slide I Don't have the time to do it But I think it's it's a super cool to just offer it and easier to use install things Yeah, yeah, so the question is how to handle multi-camera support with within one land file That would again require you to make your own job type basically copy the JavaScript file from the default and adjust it just add one more parameter for Typing in the the camera name and then pass that to the Blender command line as an argument the render from this camera And same for multiple multi-scene renders from the same file that kind of thing good question Is there any interface for the workers locally on the worker to send it to sleep and such? nope But You there is a hidden feature that you can send a unique signal like a post it signal user one user two to send To sleep or to wake it up on Linux which could be the basis for a separate user interface, but There is not at the moment, but I Think it would be cool feature to have some graphical user interface. I've been thinking about making it actually But then you also have to think about like what on a headless system that may not have Like a graphical interface to show that you are I So it requires a bit of thinking, but I think it's it's a necessity at some point But they can always go to the web interface and just turn their worker off there, but it's a bit more cumbersome Yeah, yeah, like there is no authentication here, so they can just click on their own worker and Just say well send to sleep So let's do one more question, and then we're only five minutes over time Yeah, oh, I would love that like so adding simulation baking and those kind of job types I would love to see that like but That is up to the people who need it to actually write it out and and share that But I think it will be super nice to have some place where we could collect these job type scripts and And and share them with people Yeah, yeah, and like this is exactly why I wanted to have this job type system because so many different Questions actually get answered by yeah, you can do this with just a little bit of JavaScript and then we can share stuff and then Get it to work together. Thank you all so much. It's been great