 Yeah Hey everybody Thank you very much for showing up it has been a long but very awesome conference day And now you can sit back relax and enjoy this case study of our virtual fashion pipeline. Let's see if this works Yeah, so virtual fashion is a very broad term But first and foremost when I talk about this I mean bringing garments and clothing but also access was like shoes and helmets From the real into the virtual digital world and create 3d representations of them But besides creating pretty pictures and viewing them from our angles There's a lot more you can do with it. You can do virtual try on you can do mix and match of outfits You can do a size recommendation and you can even do a virtual fashion imaging And I will touch briefly on most of those in this talk, but first let me introduce myself My name is Philip gosh. I'm a technically artist currently working at Reactive reality where I'm most of the time coding our own custom real-time render engine But I'm also the local blender expert So we're using blender in a lot of places in a lot of different departments And whenever there is some questions issues or we need a visibility Study can we use blender for that people approach me and I really appreciate that Besides that I'm doing a bit of freelance at my own home page So far studio and I'm part of the sketch that masters team Which I'm still really proud of and feel free to connect with me on your favorite social media of choice So this is what I will be talking about will I which I will be talking about in this talk So let's jump right into it the overview of virtual fashion and the use cases for it so Generally speaking the fashion artist currently striving if you go to art station and search for clothing garment or virtual fashion You get a lot of really awesome artworks that artists created uploaded rendered and animated So I definitely urge you to do that. It's just very beautiful to see But there is another point which is using virtual fashion to actually help fashion companies and fashion brands In the different stages of their production where it is really helpful to use virtual fashion So you can do for example do virtual prototyping where you can Test the look of your garments before ordering fabrics and manufacturing any of them You can do virtual photography, which is an area where you can combine model photos or models 3d models with different garments and use that for product shootings or marketing without Doing any real photo shoots and of course in the end for retail You can do a virtual try-on in webshops you can do a omnichannel presence where you have a brick-and-mortar fashion store and You put some magic mirrors or iPads or whatever there and people can do a virtual try-on before going on and actually ordering at the store the Garments fashion clothing that is added on So that brings me to the virtual fashion pipeline we have established at our company reactive reality Let me introduce the company real quick first. We're a fashion tech startup Based in grads, Austria where I'm also from we have offices in grads in San Francisco, Paris and Tokyo founded in 2014 and around 60 employees and We're working with a bleeding fish bleeding edge fashion technology So we're using a lot of artificial intelligence and machine learning to enable the creation of Virtual fashion at scale. This means that a fashion brand can approach us and say we have a collection of 10,000 individual items and we need optimized 3d models of them as soon as possible and then we Go and say yes, we can do that So here's a short video about what we are doing so first and most obvious use case of course is having 3d models and representations of the garments and clothing in a web store then here you can see the virtual try-on Where users can try on different combinations on of outfits when shopping online We try and we do actually Digitize a lot of different garments This is an important part where users can create in 3d avatar of themselves But just simply taking a video or selfie with the smartphone and then use this 3d avatar of themselves to do Closing try-on on it and as you can see we use blender in our official marketing videos. That's how much we love it And a lot of our input is based on photogrammetry scans I will talk a bit more about that later in the talk and you just own our photogrammetry studio and Yeah, that's basically what we are doing So our technology runs under the pic-to-fit name and there are mostly three big Groups of products. We have the capture kit that I've just mentioned which is used for Photogrammetry captures of garments in an automated fashion Then we have a content service which we use for managing all the assets or the clients or the iterations or the different parts and processes of Creating the garment and we provide a software development kit to our clients Which they can use to integrate all the functionality we provide like size recommendation virtual try-on and so on and so forth to integrate into their mobile apps or web apps and This is how our technology stack looks internally. So we have different sources where we are getting our our base data on and I will talk about this a bit more in detail later and Then everything is fed into our content management system Which is basically a big database of all our assets or our garments or our avatars All our clients or the collections of the clients or the different iterations of the assets we are doing and so on and so forth and we have an Convenient web interface to to work with all of that and it's integrated with the different other technologies We are using then we have a content delivery network, which is accessible for the clients and they are they can download the Textures 3d models or the rendered image or whatever they are Expecting from us as an output and of course there's a SDK that can be Integrated from the clients into their iOS Android or web Presence or using Shopify or whatever technology the client actually prefers So let's get to the meat of it How we are using blender so Yeah, this is The basic sources for fashion data we can get we can have photogrammetry data cat data It can be modeled and textured manually from concept images, of course or from photos and it can be 3d data Generated by an AI So one of the most common ones we are dealing with is photogrammetry So we develop a capture kit and the setup you can see here. It's three cameras. It's four actually nowadays and light sources and there is a turn table which is connected to to the other part and Which is automatically turning the mannequin with the garment taking photos and the photos are automatically up Uploaded into our content management system where there are some pre-processing and photogrammetry is done as Software for photogrammetry we are using Cepheus 3d But we do a lot of custom post and pre-processing to order or the images and the 3d data we get out of that Then the other big way How fashion data usually in the fashion business is created is by using specialized Tailoring software like close 3d marvelous designer brownswear and some others. This is actually software aimed at Dressmakers tailors which are actually Creating the fabric patterns and then they have a 3d editor where physically simulated 3d suing is performed and if you want to try that way of creating fashion fashion, yeah, I can Recommend these two great add-ons. I mean no way affiliated with them But I thought I should mention it for those who want to try creating virtual garments themselves. They're really great so we also have a team of skills 3d artists who are very versed in creating virtual fashion and They mostly do a final check of orders 3d meshes and textures we get out of our automated tasks They're starting and initializing processing on new iterations But there are also just skills 3d modulus and textures who are fixing Sometimes common and sometimes completely one-off errors in the data we get in process and There are errors while the automatic ways work really well nowadays it still happens that something goes wrong and Our 3d artists have to deal with it so depending on What input method and what processing we use we have other set of issues we look out for especially and I will show you some of those issues a bit of warning I call this the gallery of horrors for 3d artists because now you're seeing mostly mesh and texture issues and errors So this is one most people who have used photogrammetry probably know Lighting information is in the texture. Everything looks nice and once you go into a shaded few or you do a relighting which is the actual Problem then you see that the mesh is completely off and needs to be fixed Then we have weird topology Cases like this where two holes are connected and everything is just supposed to be a flat surface Are especially nasty because they still could technically be watertight and manifold meshes So it's hard to look out for them Then there is of course the floating geometry which doesn't belong anywhere We usually Don't do that anymore, but we had to do a lot of detail removals I hope you can see that where the Tabotans here and this part have been removed from the mesh because everything Every bit of information that we need was in the texture already Then as usual normal issues, that's not a fancy new Fashion style that should be a plain gray t-shirt which actually has normal issues and Then this is a UV layout you could get out of photogrammetry software. Yeah, it happens And I guess everybody can see there's a lot of potential to use the Rest of the UV space available Then you have Mesh connections where you don't want them. They might be fine on static models, but once you are fitting the garment onto a post avatar then The problems quickly become visible and of course there's missing texture parts This happens especially often if there are folds in the garment and photogrammetry software doesn't Have enough information from the input images to create our parts of the texture Then another thing when you're doing photogrammetry is you have take care of insides If you're scanning items with two parts like shoes where you have the upside and the downside Which need to be scanned separately then of course they need to be merged and Yeah, a lot of this stuff we noticed was Recurring so the same issues popped up and time and time again So we thought of ways to automatically check and fix for them, which brings me to my next point The tasks we automated quickly checking the time. Oh nice So most commonly to fix the issues what could be done was we fixed the topology which is Best done by simply remashing the mesh so it can be done automatically if you remashing you have a new mesh Anyway, which needs a new new UV layout So might as well make this at make that as clean as possible and after that's done We can rebake the textures and we noticed if we do those three steps automatically we can fix most of the issues I've just shown you before So we came up with a way to do that automatically and of course we created a custom blender add-on So repetitive and tedious tasks can be taken off the shoulders of our 3d artists We use a very powerful internal software where we do a lot of Processing a lot of preparing for fig fitting working with parametric body model representations and stuff like that and We realized it's really beneficial for our artists to have a seamless integration and Transfer between blender and our custom software so whenever you are editing something in our custom software You have an option to just transfer the asset seamlessly to To blender do some mesh texture or whatever editing there and then our add-on provides a way to seamlessly Transfer back to our software and load the appropriate data We're using some custom file formats for that because we have lots of additional data information For the garments and clothing we are using but we also provide gltf and obj import and export and Yeah, we created an analysis and a cleanup Tool more or less which our artists can use we realized that most of the mesh is issues can be found by the the add-on that already exists and is Designed for 3d printing meshes So thanks to the power of open source we could reuse a lot of those checks Then we have a big part where we provide very custom functionality that actually comes from our C++ SDK where we can do garment trial on physics Siemens subgroup handling when we are dealing with cat data and This is made possible by our software development kit Being provided with python bindings which means our whole C++ software development kit can be Compiled as a python module and just loaded inside blender by using the command import fix the fit core pi and all of our Functionality is available inside blender and of course this is extremely powerful and our add-on makes use of that So then we more or less have shortcuts for other add-ons We're using the great UV pack master add-on for For layout of UVs and the quad remasher for remashing and we're also running some Blender instances on server to do automated tasks Which is really easy to do and I can recommend and as the last point of my talk I will talk about leash machine learning data generation as I've said we're using a lot of machine learning to do very specific tasks and To aid the artist so then I don't have to do tedious tasks again and again and again and That also enables the scalability we can provide for processing thousands of assets in a short amount of time so now there is an issue with machine learning because We want a machine learning model to do some task on a large amount of data So a machine learning model needs to be trained and to train the machine learning model You need a large amount of data. That's already labeled So that's a typical dilemma the with machine learning. So what can we do in that situation? of course, we can use blender and Generate synthetic training data for us to train the machine learning model to then use on our real data So I will show one example real quick We have image classification for photogrammetry input which we wanted to automate so Taking one input image from the cameras of our capture kit We want to know which pixels belong to the garment which pixels here shown in red belong to the mannequin and which ones belong to the Turnchabers so we can do automatic mass creation and some more so what we did was we created a virtual version of our photo studio and Rendered image that as closely as possible resemble the input images we get from the cameras but when we additionally Enabled the object index pass this already delivers us with all of the masks and Basically the labeling we need to train the machine learning algorithm with that So here you can see the digital version of our studio with the lights resembling the real light positions and The camera is moved with a script and this is a low-resolution photo of how our actual studio looks like so you can see it's very similar and This is a rendered output. It resembles the Photos really closely which was good enough for the machine learning Algorithm and additionally we got ready masks corresponding to every render for the the different Pixel areas we were looking to classify So We then added a script to do that automatically and we Varied some parameters like the camera position the settings like field of view Where the lights are placed and how strong are they and different environment textures to simulate different photo studio situations and As a result when we rendered hundreds of thousands of training data images using this approach We got a huge training data set which works so well that our machine learning algorithm doing that classification can Could be trained basically So we are working on some more stuff in that direction currently and actually employing it so for example, this is 3d model creation based on a single input selfie nothing more and That brings me to the end of my talk. Thank you very much for your attention I will be around to venue if you want to chat and have fun with the following lightning talks. Thank you very much