 Thank you so much. Hi. I am Siebren. I'm a developer at the Blender Institute and Today we're going to do a workshop about your own render form with flamenco So this is about the flamenco. It is not about a bird But it's about the dance So I'm not going to dance for you guys No, sorry about that Yes, so that that's where it comes from from the all the dance of all the little computers that are doing stuff for you It's a little bit of an overview First I will talk a little bit about the structure of flamenco like how do the components fit together what it's all made of Where to get flamenco? Maybe you already know How to set up your render form and how to set up the render cloud part because it's roughly separated in two parts One part you run for yourself with your own hardware on your own infrastructure And then another part that can be run by the the Blender Institute on Blender cloud infrastructure or you can host that yourself if you really want I Won't go into hosting the server part yourself because that's a huge topic And we wouldn't be able to cover all of it in just one hour After that we'll go look at scheduling of the workers How failures are handled because I think that's quite an important thing when it comes to dealing with rendering Stuff always falls over and a little bit about logging because I noticed that people don't often quite know where to look for Which log file in which situation? So before we begin Two things that I always ask for in my workshops one interrupt me At any time if you have a remark if you want to say Do you want to ask something if it's unclear or if you have a different opinion than mine? Just raise your hand shout it out if I don't see your hand I Want this to be to be interactive and not just a one hour of me droning on about Flamenco and ask stupid questions That's number two So who of you has seen Andy's talked this morning? All right, let me flip it who hasn't seen Andy's talk this morning Okay, and he covered a bit about how we work at the blender animation studio the structure of the files and everything So I will I will also discuss a little bit of that then I started working on Flamenco fairly soon after started working at the Blender Institute, which was back in 2016 and The first open movie that was rendered with it was Agent 327 After that also the David weebs spring was also using Flamenco and now the work on on rain or the test renders and everything It's a new character for character Development workshop on Blender cloud So the goal of flamenco really is to support The rendering of animation Because that that's what we do at the animation studio. So that's also what it's aimed at So one of the things that it cannot do is tiled rendering of single images Who's interested in tiled rendering of single images one. Oh, I think the infrastructure is perfectly fine for it The only thing that is should be figured out then is how do you? Get Blender from the comment line so far that it just renders one tile How do you collect the tiles and turn it into the final output image? So if any of you want to help me out on this and write some patches for for the pipeline so that it's supported I'm all up for it. It's just that we just didn't need it ourselves that much yet Also, it's meant to be used in a studio setting And this can be a small studio of just one person at home or that could be a medium-sized studio as we have at the Blender But it's not necessarily meant to be the easiest thing to install it does Require a bit of knowledge about how things are meant to be set up It does require a bit of knowledge about how to move files around and what a directory means and so some people try to use Blender even And I think they can just install flamenco and render it on the cloud with clicker three Unfortunately, it's not that far. I wish it was that simple, but it's not So let's look a bit at the structure of of flamenco Yes, so I have a bunch of cubes here that represent computers The green arrows represent the the command and control messaging And then the blue arrows represent the files that are being transferred back and forth So what we see is the render farm Consists of the workers that are actually running blender and doing all the rendering they do they do the work They're being controlled by a manager So if you start your own render farm based on flamenco That's the stuff that you would install in your own your network And then the workers access some shared storage that will contain all the assets that blend files everything Then you have the server which is Part of blender cloud you can run in yourself as I said, but it's easier to just use the the cloud one And the artist could be yourself the the user of the system Has the wrong computer of course running uh blender So basically they open a blend file they go to the flamenco panel they click on render to flamenco All the files are written directly to the shared storage And then the workers can reach them and that is a situation that we've been using for a long time at the blender institute Our situation is really simple. We have uh slash render which is an nfs mounted directory that is on some server Which with a lot of hard disk a lot of space um And the artists can write to it directly because they have slash render Mounted on the same spot as all the workers do So without any smartness paths just work because it's all the same anyway Well, this is nice, but it's also very dumb um We work with um subversion for Managing the assets So every time an artist wants to work on something say they want to work spring Once you do some animation of a shot They make a checkout of the subversion repository. Oh, who does not know what subversion is? All right, basically it's a simple way of of um Communicating changes to each other So it was made for a source code for developers One developer every developer has their own copy of all the files. They can make a change to a file They sent that change to a central server and then other developers can get that change And that way you can get the latest version of everything and it's all relatively Um Relatively small because only the changes are saved and not a copy of the entire file all the time And it also it not only works for for a source code, which is text, but it also works for binary files like blend files So the artist says, okay, give me spring he gets all the files for spring on his local hard disk Makes a change and then wants to test that change In in flamenco, but that change may not be the final thing. He may not want to Send it to to his co-workers yet. Just wants to try something So then all the files that are necessary for that render job are copied To the shared storage And so with the very dumb system is very very dumb because every Render job has his own copy of the files um This was doable because we had a lot of space but um It also had a problem when we wanted to try and move stuff to the cloud So if you want to run this on AWS or asia or whatnot you have a problem because you're transferring gigabytes and gigabytes of files all the time um So what we now do is The manager has a deduplicating file server which means that Blender sends all the files to the manager, but only those files that the manager doesn't have in storage already So only the changers are sent The manager writes it to the shared storage and from then on the the dance is still the same with the workers accessing the shared storage and um these What is the same file? Well the file is the same if it has the same content And the same file length and everything then we call it the same file. So it could be Renamed it could be anywhere on on your file system It will just make a check some of it using a shaw some and it's just the three letters of the creators of the The algorithm Uh, I don't know the names But because this is shaw some is the base for the thing we call the thing is shaman So where to get flamenco you can go to two websites One is flamenco.io where you find flamenco itself the manager the worker will take a look at that later And you can go to cloud or slash services to download the better Mac cloud add-on Which will give you The access to flamenco the ability to send your files to the to the firm from within blender Any questions about this? so far Yes One in the back How we how easy it is to send accounts the custom render job you say or custom commands Uh, it's tricky So we'll we'll come to that later when we look at the actual ingredients of what makes a render job and we'll see a bit more about the ingredients of The choices that are made and maybe you completely don't agree with it Yes I'll I'll repeat the question for the recordings and people at home. So the question is What basically what is handled with subversion? What is actually on slash render? And is there anything else in between? Is slash render version managed? Um slash render for us. It's just a big file system So that's not version managed Um, it's the shared stuff slash render is the shared storage that we use um So version has all the that's the ground truth that has all the assets and everything The checkout is done in a different directory. That's locally on the computer of the artist So that the file access is the fastest because they have it on a fast SSD When you say okay render this file Some software starts analyzing that file Looks at all the dependencies that it has. So then it finds that share that linked in character asset blend file And it recurses into that So then if you have a probe in that file that comes from yet another blend file, it will also Copy that one. It will also copy all the textures that are used and everything that's all copied into one bundle and stored on slash render in our case Can the storage be bypassed if all the computers already have access to everything anyway? Well as of uh before yesterday No but Adi told me like It's not working for us. It's making all these stupid copies I don't want that they can reach it anyway. So as of yesterday. Yeah, you can All right, let's go look at what actually makes up a flamenco render job So it's basically jobs tasks and commands um A job is the highest level thing. That's the thing that you give to the server you say, okay Render this blend file from frame x to frame y Just do it It's a small json document that is just shoveled there. Uh, so that's a high level This is the thing that gives you high level control. There's the status. It's cute. It's active. It failed It's complete It has a priority So you can just say this is priority. Uh somewhere between zero and a hundred in our case High priority means a high number Other i always get confused when it's like priority one is high but priority hundred sounds higher So it's just high is high And then the job once it arrives on the server It gets converted into a list of tasks These tasks are like Render these frames again like the job but probably a smaller range So you can do one frame five frames ten frames per task And these are the things that are being sent to a worker So a worker doesn't know anything about jobs. It just receives the task it has to do and then it does it um, if the task also has a status so it can also be active and queued and uh It can be recued if you want so if you have a failed task because Some some file was missing you fix it quickly in the meantime and then It can be recued without having to recue the entire job You can just click on one button and recue this task and we'll just do that If there's too many failed tasks, then the entire job will fail It will cancel all the other tasks in the job So that the farm can render on something that is not failing. Hopefully the next job With so the job has a priority So all the tasks are first sorted by their job priority and then within that job. They also have their own priority um Again simple sorting order basically the first task is run first But there's also dependencies So if you have chunks of frames to be rendered and then they should be converted into a small video and then the next chunk of frames to be rendered also converted to a small video Those conversion to video has a really high priority But they do depend on their input frames So as soon as the input frames are there something can start working on the video but not before But the next frames don't depend on that video So rendering the next frames can also already happen in case there's nothing that can do the video rendering So It's a double system um So then In the same way that jobs have a list of tasks a task has a list of commands and Here we get a little bit to how easy is it to customize it and to to send another command These are not literal Command line commands that should be executed. These are more abstract things. They say um render frame a to b again With bender this landfile, uh, they say merge the sequence of png files that you find there with ffmpeg Um and merge that into a video or combine these two exr files with different samples of the same frame merge those and create a new exr file and um The here if something fails that just fails a task the command doesn't have a status It's just a part of that task and the if a command fails the task fails So here you the the final command that is to be executed Of course that needs to be constructed at some time, but that's done by the worker at at the last moment It constructs the the command line and executes it and for development this was really really easy because um I could keep the job definition as is I could keep all the tasks as they are if there was a mistake in the command line I could just fix it in the worker Just deploy it to the workers and It's it's done It doesn't need Recompiling nothing needs to change in in any of the systems. It would just rerun run the new command But it also makes it slightly trickier to get a new command in there because you need support on the server because it needs to construct all the tasks that contain all the commands So you need some support for it on the server and you need support for is on the on the worker any Unclarities this is clear. Yeah, you're referring to techniques. I Almost hear the the name of but can you say it again? Right. Yeah, I've Yes, I've tried it. No, it's no JS But in the end the worker is like a task runner. Yes, it asks the manager give me a task please It executes that task it reports on all the longing and everything um Also one detail that I skipped in in these images The the arrows are the data flow So of course all the systems collaborate with each other But the actual TCP connection Is only made from the workers to the manager And from the manager to the server And not the other way around So that means that um Your manager could be somewhere in a in a demilitarized zone Between your internal and your external firewall so that maybe an external client can connect to it um, and then your workers could be Unreachable for the manager itself safely on your local network And still they can communicate with each other and the same the server doesn't need to connect to the manager at all So that could be again completely inside your own firewall on your local network Right now I can't reach our own my own manager at blender because I just fireball in between So from a Also from a privacy point of view, I've heard a lot of people say well I want to use flamenco, but I don't want to use blender cloud as a server because I don't want to send more files there Well, I Wish we had the bandwidth and the cpu power to handle all your files, but we don't So no files are sent there. It's just uh, the commands themselves. So the file names Yeah, if you're really sensitive about sending file names somewhere, then you shouldn't use it tiny bits of logging and up there if you if you don't want to share that with Just the people in your project Then also don't use it. Otherwise, it's fine So looking at setting up the firm then. Oh, yeah Can you prioritize the jobs in the queue? Yes You can just click on the priority type in a new one and it will be sent to the manager. It's all fine Right. So the question is how Connected is are these commands to using a particular render engine? Yes or no We'll look at the the current existing ones And we have one that really uses the the sample chunking that cycles provides. Well, then you have to use cycles We also have just render these frames and then it doesn't matter which engine you use So when you're thinking of setting up the farm that this is always what I think about the render farm big machines lots of them But if you want you can also do it like this You see all the components here that you have render nodes You have network connections. You have to stick shared storage and it doesn't need to be more complex than this actually Flamenco manager flamenco worker you can download from flamenco.io It's all developed on developer.blender.org because it's a blender created project So there you will also find the sources for flamenco server But not in an easily downloadable and runnable package like you have here So the manager uses mongo db as a database I did bundle mongo db into the downloadable package So we should just start it for you and stop it for you if If you want it I would recommend just install it yourself and manage it yourself because it's a bit more stable It uses mongo db because blender cloud uses mongo db So when I added flamenco server to mongo db to the cloud it also used mongo db And it's basically a synchronization process. So it just made sense to use the same database here as well So the manager links to the server At this linking process entails Exchanging an id and a random password. You can do that through the web interface. I will show that in a minute And then you can also configure it through that web interface If you really want to tweak some advanced parameters, you have to go into a config file But we even have a config file editor in the web interface Then there's the workers the workers they can just detect flamenco manager on your network If it all works that some company policies are that are blocking multicast yadda yadda bladda bladda and then it doesn't work Sometimes you have spotify running on your machine And then flamenco manager will say no because spotify takes up that same space of auto detection as flamenco uses And they're configured via files and those files they are read it from a few locations Um As we have slash render for all the the render assets we also have slash shared for other shared stuff So we've shared software flamenco worker and there we have the configuration file for our in-house farm That's read first And then from the home directory of the user running the worker It also run gets some other Configuration it merges the two so you can put worker specific stuff into one file and still have most of your configuration Spread throughout your well used from the same location Can maybe open the door? Yes um So what we're talking about setting up the thing So, uh, yeah, if you open it all the way it will just magnetize Um, so when you're setting up flamenco, what are the important bits? Well one is where is the shared storage? Of course quite important because everything revolves around those files Where's blender? And ffmpeg and maybe some other tools that you're using And how do people send files there? And I want to look at the last thing first Which is the shaman? Uh, I've already mentioned this guy is now built into flamenco manager um, and That means that as a location instead of giving a directory You say okay, give me shaman colon slash slash and then the name of your your manager Uh, that implies htps because we're living in a modern world. Everything is encrypted anyway If you don't want to set that up that's also fine But then you have to say shaman plus htp And then we'll just use the ittp protocol So how does it work? Well Blender starts analyzing all your files finds those linked assets Runs over them computer checksum And then sends a list of the checksum the file size and the path to uh, flamenco So it sends to the whole list of files Then the shaman checks. Oh, I have those I have that I have that Sends you back the file set it still needs It also keeps track of who is already uploading that file because maybe Somebody regenerated some Cash somewhere that needs to be sent to the farm. Maybe your colleague is already sending it and then you Hopefully you don't have to so it sorts the stuff that is already being uploaded at the moment at the last And if you send something yes Yes Yes, uh, so in short tip we implement bit torn for uploading. I think that would be amazing um, but no But there is a thing that it will just Kick just close your connection as soon as the other guy up is finished So you both in the worst case you'll both be uploading the same file at the same time If it turns out that Your start doesn't end after his finish So if he's still uploading you're still uploading then But when the first one stops the other one is also cut short. Oh, that's interesting one Do a pull transfer, but then That chunks transferring Yeah, maybe make that work would have to Rethink the server client because then the you make the connection, but you are the server interesting Then the server makes the connections no matter yeah, exactly well It can work, but I have to think about it So then after everything has been uploaded Um blender requests a checkout So by that time it also registered the job to flamenco server It has the job ID it then says okay flamenco manager Please give me a checkout of these files with these checksums as these paths Um, and named it checkout with this ID and then The manager starts Sim linking it starts creating your entire directory structure. It starts sim linking all the files in place And without taking up any extra space you have your whole directory or your job storage structure complete So I made this for ascending over slow network connections But in the end my colleagues were very happy with it because the uploading even to the local storage was way faster No more duplicate files Did the big monsters in spring are 1.4 gigabytes each Because of limitations of blender we had to have like five of them instead of Only one and then duplicating it So every render job was really heavy and this really helped a lot but It only works on a platform that supports sim linking And on windows, I think it is supported sort of if you're a super user Which I wouldn't run flamenco Worker as a super user because it can delete files and I don't know if it's bug free I know whether it's bug free So this is what that file store would look like So you have Shared flamenco file store. That's what I told flamenco. Okay store all the files there And then it has a stored directory and uploading directory Nobody is uploading so the uploading directory is empty right now, but it would have the same structure otherwise And you have to check some We take two characters of the front and then so that it's a bit nested directory structure And then the blobs which are all the files and the file name is just a file length So in the case that two files have the same checksum, they're probably not the same length and this should Not result in any collisions And then this is what it would look like In that checkout so every file that you see here is actually a sim link To something something dot blob in the file storage You also see an outside project directory And I will take a look at that Yes, there we go I put one panorama photo in the background of the image of the scene. Yes How do you handle the user uploading and overriding files while the render is being done? Well, the thing is the render that is running at the moment Uses files that are there already So if the user changes a file It contents change If the contents change the checksum changes if the checksum changes it means It's in It will be in a different directory here Which means that Even though The file name here will be the same people will link to a different file in the file storage Because it has a different content. It has been changed. It's in a different location in the file store And thus it will use simply the the new file here So the question is what would happen to links files if you change the link Sane I don't quite hear the last bit of your question. Sorry Yes Yes question is If the file that is changed ends up in a different directory because of the checksum Doesn't that break your links? And fortunately not because of the sim linking because these are The files that blender will see And the sim linking on on linux is transparent for the program reading it So blender thinks it's reading this structure And this is the structure that you gave as an artist when sending the files there So we'll still see everything that it expects to see Yeah, I just wanted to ask how many versions of the same file are kept because it can grow very large very fast, I think Yes Everything is kept At the studio we have We keep all the rendered jobs And if we don't For a reason it can happen But then it's a it's a human going in there and deleting that the directory for that render job I mean the blobs. So yes, I'm getting to that That means that the sim links will be gone And then there is a background process inside flamenco manager that once every hour or so you can configure this It looks through the thing and Starts cleaning up those blobs. So if blobs haven't been used in any of the currently linked structures And they haven't they're older than x days. They will be removed Okay, thank you. May I ask another not so related question? If after that you give it to that one So are there any plans on supporting docker or Getting like a docker setup any plans on supporting docker. Well There is nothing really to support in that sense is rather simple, uh, I don't have plans on providing docker images We do have um because it's all open source for the server We do have our own docker setup because blender cloud also runs on docker So if you want to set up your own server you want to use docker for it Go to the blender cloud repository and see how we set up there That will be a good start of uh of that part anyway My question is you mentioned if i'm correct subversion and you said words like check out and this is pretty much the words from the version control system. The question is Do you use your own version control system? Are you kind of utilized svm or git or something? I mean the the blobs structure, etc uh, that is just custom made it's not using anything else um Because I it is so simple that uh I didn't really feel that we should build up on some other technology that has way more features that can get more complex So no, it's just as simple as i've just explained that's all there is to it so One thing that flamenco also supports is um Concept called variables Because maybe your Workstations for the artist run on windows But maybe your render form is on linux Or the opposite or you have a linux and a windows and a macOS machine sitting idle at home And you want to use them inside flamenco so We have these variables In order to deal with differences between platforms so macOS and windows linux those kind of things and environments so and then I see The users or the artists the people running blender as an environment And where is the render form as an environment? um for example at the blender studio the workers and The the artists are in the same environment. It's all mounted same place directories are all the same so And everybody is running linux anyway, so we don't really have a use for this But when you have maybe a render farm in the cloud somewhere Probably the directory structure there and the the network infrastructure will be completely different than what you have in your office So we have two flavors. We have one way and two way variables One way or a simple Notation like that curly braces blender and That can have a value for linux it can have a value for windows it can have a value for macOS and Those are handed out by the Deal with by the manager So the manager knows the operating system of the worker So when a manager hands a task to the worker it first is all the variable replacement Gives it to the worker worker just sees paths and is dumb and stupid just does what it's being told So this is what the configuration looks like in the manager The variable blender you can toggle it between one way and two way, but this is just a one-way thing and then the values for the different platforms This one is only used for the workers. So there's no need to set it up as Different for users and workers because users don't use it anyway um And shaman is another one. This is actually used by the workers Why I have to make a new fix for the manager the little explanation there at the top is wrong. So ignore that This is used by Workers to find their There are their files. Basically. This is where the shaman made all those sim links And where the the files end up and then there is a two-way variables. These are a bit confusing um because they are the same as The the other ones with with the twist They're also translated by the blender cloud add-on when you push your job to the server Um, and we by default we have two of them. We have the job storage. We have render But this is really freeform up for you to to determine what you need what you don't need So why is it called two-way? Well, it looks at the value when submitting And if it sees that some string matches that value it replaces by the variable itself with the curly braces notation And then it works in the same way as in one way when the manager gives it to the worker. It is replaced again by the value So it's a bit weird you take a the value and then you end up with the value But these are bound to the operating system So you can just do things like this You can have shared flamenco jobs on linux And s flamenco jobs on windows and volume shared flamenco jobs on macOS and when I submit something Uh That I stored there It's replaced by that And if a worker then runs on linux, uh, sorry on windows It will read that And this is the way we deal with differences in in in platforms So far it's the only thing we really need it because the specific commands the worker can handle anyway It knows oh I have to render from this to that it knows which parameters to give It knows the operating system that is doing it. Usually there's no difference anyway But if there is a difference then it can handle that as well Aren't these just a workaround for somebody not using a one-way variable in the first place? Not necessarily because um I'm rotated that way then yes, thanks. Um Unfortunately blender doesn't have variables So what I use in in the blender cloud add-on is actually the file name of the current blend file And that's all there is to it. So Yeah, you can tell you can tell it that It should go to I maybe it could be a little bit smarter, but it's It's tricky I think I think that It would be a good idea to have this variable system Be a bit more extensive and be a bit better understood by the blender cloud add-on itself So because now it requires quite a bit of setup manual setup and We're thinking about pushing more configuration from The manager to the blender cloud add-on. So the blender cloud add-on just gets the information instead of you having to type everything So and then that case this would also be solved. Yes so Those workers I think we discussed all of this pretty much The only thing is that uh The workers can also declare their capabilities Maybe you have a simple machine that's just not doing anything But it's way too small to handle any of the Files that you have for rendering, but it can move files around. Maybe something is fine for rendering for running ffmpeg So you can have little Words in that configuration file Let's say blender render video encoding file management exr merging to handle Who runs what? so I Guilt, I have to speed up a little bit because we only have 15 minutes left So looking at flamenco server The thing that you need to do before you can run the whole thing is you have to set up a project And a project is something like spring h and 3 2 7 daily dweebs. Just that is the what we call a project I've seen people call every individual ads asset a project as well because it's a lot of work But just the film project that you're working on You click on that little pencil there On blender cloud. Oh, sorry On blender cloud itself. You can go to my my projects Create a new one has anyone Done this before Few people all right cloud.blender.org slash p for projects you can create your own project there Give it a name nice image You click on the edit button there You go to the flamenco thing set up project for flamenco And then you can choose which manager you want to to add and For me as an admin that's like 741 managers um This is the flamenco manager setup web interface It starts here automatically by the way if you start flamenco manager for the first time You get a lot of text And at somewhere it says point your browser at any of these URLs if you have multiple networks or multiple ip addresses It will just list them all and you just Go to one that looks good you get here you can link to cloud.blender.org or Your own server you just type the URL of your own Then you can say okay. Well, which one are we going to do well conference demo or maybe you want to make a new one you can just type a new one You get back to the server. I just chose to restart it just to be sure And I was linked And this means that if you now create your project you go to all those settings that manager will show up You can click on link and then that manager is linked to that project And this way if you have a few different projects that are in the few different Render farms you can manage better. Which one is available to which project For example during spring we mostly used our internal farm, but we also had the IT for innovations Super computer farm at our disposal. So we could also send stuff there and we link both through the same project So many windows So what's left final ingredient is the blender cloud add-on It's built up on blender id for the identification. So you need to create a blender id account Enable the blender id authentication add-on which is bundled with blender You log in tells you you're logged in And then if you go to the blender cloud add-on you see you're logged in and You have some things that don't have to do anything with Flamenco, but are pretty cool. Anyway, you can sync your settings back and forth You have a texture browser, but we're going to go down here to the project settings so Up there you can select the project You can configure your local project path and this is something that is different for different situations In our case, this would be where you make that subversion checkout. So it's the root of your project directory And at this case, this will be on the shared storage already so Job file path is Where are the jobs sent to? That could be the same directory It could also not be a directory but a shaman URL That was what you would type there The job output path is where the renders would go to So it doesn't matter what is configured inside the blend file. This is what will be passed on the On the common line of blender to send the the rendered files And it gets a bit of a suffix. So it does include the the job the the name of the user that kind of stuff so finally we have We have an exclude filter Let's say in our case It's all on the same network The Cosmos land room at tornado was One point something terabyte You don't want to include that. You don't want to have that as a version You don't want to copy that onto every artist machine and then copy it back into the farm and have a different copy for every render job You just want to refer to it once So then you can say okay exclude the vdb caches exclude the Olympic caches exclude whatever Another approach would be to have this checkbox checked Relative paths only If anything is linked in with an absolute path, it just assumes that it's working. It assumes You know what you're doing Yes, I know It assumes, you know, it's working and it won't copy those files So if you have the alembic files on the shared storage somewhere you refer to it by absolute path. It won't be copied The project path is something that is quite important because it relates to the remapping that we do What you saw in the file browser that I had open just earlier was that underscore outside project And then you saw home sebran pictures yada yada yada yada the picture And that was because that file was outside of the project directory But we want inside that job definition All the assets to be findable I did not have relative paths only checked so it was an absolute path But still included in the job anyway, it was sent to the farm But because we want to have all the paths local to that one job directory. So it's a self-contained unit The blend file is actually rewritten so The blend file that references that asset is rewritten to Point to the new file And in order to prevent Many rewrites you just want to set the local project path properly So that everything is in there anyway and then finally the Strip components you can see below here. This is where it would save the Save the the job Sorry it would render to And structure to flamenco slash flamenco overview I had See home see we're in on cloud blender be calm 2019 in structure of flamenco. So flamenco overview don't blend open at this time so it uses those The path relative to the local project path also in the render output so that the render output matches your Project structure and structures are a bit familiar If you want to get rid of the first bit like structure of flamenco if that's always there You can increase trip components so that it will just cut off stuff until it just ends up with the file name Then finally We have the actual render Flamenco render panel Where again, you can select the manager. You can select the job type. How many frames per task So that will be a five frames per task if you have something that's really long to Load but quick to render you want to increase that If it's faster load, but slow to render you want to decrease it. It's a bit of a balance progressive rendering That is doing the cycle specific stuff So not only does it chunk the frames it also chunks the samples At first first you render everything with a very low number of samples like one or two samples Then it combines that into a video and you have a very low resolution Video well low sample video and then it starts incrementing In an exponential way the number of samples because if you see it's all right Maybe you want to see it with 10 samples, but after that you don't want to do another 10 You will maybe want to do another hundred or another thousand Yes Nice I've got a question for the frames per task option Basically this depends on the person making the project on doing test renderings and finding out whether that is something that is Long slow to load and quick to render But wouldn't that be actually something that the computer is a lot better at judging it? Yeah, so like like maybe doing one or two test frames and then finding out How many frames per task that should be there are a system that can do that But not flamenco unfortunately, okay And the final thing I want to show is the video chunks Which is we use for the edit So then it just takes a bunch of frames and Puts it into small video files then concatenates all the video files together quick way through scheduling You can Configure a schedule on the manager. You can say okay monday through friday from nine to nine The computer is being used by an artist and then at night it can run the jobs Little bit another thing that we had a problem with we had a worker who didn't mount the render file system properly So it starts up gets a job it gets a task Doesn't find the blend file fails a task gets another task It doesn't find the blend file fails the task raised through everything everything cancelled So now we have a pre task check You can configure it yourself So you can say okay write make sure that that directory is writeable make sure that that directory is readable before you even ask a task just a And then when a task fails If a work has too many failures On a certain job of a certain type so blender rendering on on this particular shot It gets blacklisted. It can still do um Exxon merging moving files around but it won't be allowed to render that shot anymore If there are other workers available to retry the task It It get that failure is soft because somebody else can take over if you have one machine with a little bit less memory A bigger brother can still try it out if there is no bigger brother is a complete failure and then It fails also if three three or so workers have tried it and failed all then it just failed One thing to know about this is that offline workers also count So if you say three workers that are still in the system But will never run again You may want to get them out Because they still count because if some worker is just now rebooting will be up in another five seconds It's a shame if you're a task got cancelled And then there's a server part That if It gets too many failed tasks for one job It will just cancel the whole job all the other tasks and it's done So what do you do then you can recue it everything if All tasks were completed. It will just recue all the tasks of that job If some failed some were cancelled some were okay All the completed ones will say at completed status and it will just recue the The the non-completed tasks I think we don't have time to go through the logs But if you have any questions, then this is the blender channel where At least I hang out and but also other flamenco users are there to to help you out And that was a race through flamenco