 We're about to do our second lecture of the day in this hall. It's called Transhuman Expression, and it deals with interdisciplinary research in painting and robotics. It's at first guided by a professor who is not here currently, Oliver Deusschen. And here next to me I have the PhD candidate, Marvin Guelzov, and the artist, the one who is working with these robots, Lejad Kreiler. They will discuss both the technical challenges and the innovation aspects that go along in the development of an e-David robot. I understood the robot is there since quite some years yet, but it's always in development. And what I see then somewhere in the future is probably that we can imagine the old-fashioned serious painters of Leiger, Rubens or Ramband, Van Dijk, who had ateliers who filled up with assistants who were very skilled in certain specific aspects. But imagine then someone like Andy Warhol and his factory or so, that would be working in a robotized world. And I think these people are going to show us or demonstrate us the challenges that are in there, because next to that kind of understanding of working with robots as assistants, we have to look as well at the social and artistic practice that this way of working actually offers us. They do not go on strike, I presume. They take sometimes a break and you have to take care about them. Are they female? That's my first question. Yeah, I think David is really a masculine name given by Oliver Doyson. Yeah, it's still masculine. Okay, guys, are you ready for this? Then I would say light the fuse and put it in play. Let's go. As the robot is recording every breaststroke it has done, we can always take those breaststrokes and repeat them. We can repeat them and shuffle them and use them in different ways. A variation of repetition of something that is so minor and barely seen that make it, I think, to be something that the viewer subconscious will recognize it, but our consciousness part will not directly see it. I think that could be a very, very interesting effect to play with. I can try to play around with this completely abstract painting by the same time as image-oriented painting, where I have exactly the same amount of breaststroke, exactly the same kind of breaststrokes, but they each time going to be put in a different way. Just the orientation will be a bit different. The location with which one was under and which one is above will change. This is an idea that I was really looking into is the concept of entropy in painting, because the idea of just how cows is increased in an organized body, so each time that I'm actually making a variation I wanted to increase the level of cows, but in order to function as a system, the painting as a system, it has to still hold some kind of logic behind it, and this logic is very hard to control, I think, from the human mind, but you can actually use a computer to help you to create it. We built this painting machine, E-David, almost 70 years ago. I'm working in computer graphics for more than 20 years now, and the part of computer graphics is what we call non-phot realistic rendering. To produce images, computer images that look like done by an artist. At a certain point I had the idea not just to produce computer images, but to draw, to build a machine that mimics what an artist does. And that was the beginning of E-David. Basically computers right now, they used to just follow simple rules and have completely predictive behavior. And right now we're kind of transitioning to more intelligent systems, as we can see in self-driving cars or algorithmic decision-making and finance, for example. So we're giving more liberty to computers to decide for themselves, and we might even be able to advance this more by just going into art, where as a human you are, I believe, supposed to be completely free creatively, and trying to transfer that to computing systems. And that's kind of the interesting part, because I think it goes completely against what a computer used to be, namely being completely deterministic systems, which just do the same thing over and over again. However, I have no interest to make a painting that's going to be run directly only by the robot. I actually find it very interesting to do something that, okay, it done something now, and I am reacting back to it. I will go and I will do maybe a glaze on it. I will just go and give another very, very gestural brush strokes or splash of paint on it, something that the robot will not do, and see what it will create to the visual sphere. The lithography and monotype, the artistic tools that when they start to be used, they create it slowly, they're on medium, they're on static. It not only works as a printer, more like a self supervising or self controlling machine that controls the painting process itself by analyzing the photos that it takes after some time. So this is a big difference to other painting machines that exist, because most of these painting machines work more like printers, because they pre-compute everything and just print it out. Yeah, our initial idea was that we can reproduce this kind of feedback loop. We were able to reproduce this using the machine, but now your questions arose what the creative aspects are involved when humans paint. The small one, it's not as powerful and not as frightening as the big one, because the big robot, it's huge, it's got a lot of force and you can't really work alongside it, because if it does the wrong movement, you can really get hurt. You can just guide it to do something for you in your painting like here, fill out this area, so you can do something else in parallel. The question what is creativity is really interesting, because we can't really properly define it, and until we can define it, we probably won't be able to put it into a computer. So basically the combination of all the real world experience we have with some kind of inner world we have of our imagination, we have to bring both of these into a computer, and then there's some kind of process that combines it, and yeah, it somehow exists. As a human, you know it exists. As I'm struggling, you can't really put it into words or even a program. And it's a little pre-painting, and there is something that I learned about it. It was about the embodiment of the act, it's about how you move your body, and the use of ink on rice paper, it's something that you can barely control. You always create the same kind of a pattern of how a flower should look like, but you cannot control exactly how the ink will spread on the rice paper, and you cannot erase or change it. The thing that you need to focus on is how you move your body, it's not how you perceive what you see. And I think one of the most interesting points for us was this discussion about control and loss of control in the act of making a painting. As for the computer scientists, the most important part is to control and to predict everything that the robot is doing. Most people that come here really think that we want to reproduce other artists, but that's not really what you want to do. We want to explore how an artist in general work, other certain strategies that you can have for certain artists, and not really to copy artists. Actually programming a robot to paint, which not for you, I think it's a very important distinguish for me at least, using the robot as a tool, as a painterly tool, it's something that helped me to understand where my conscious is part of it, where is my individuality is part of it, and where it's about the execution of different kind of material. Currently, in most of the cases, the machine works in fully automatic mode, and we are fascinated by that. You start with something on the next morning after like 10-12 hours of work, you get an interesting result in some cases. But my final goal would not be to let the machine paint by itself. Instead, I would really like to work together with the machine. So my dream would be, like in an artistic process, I want to teach the machine in a very efficient way, what I want to have online on my canvas. So I had to learn that the best thing if an artist comes is to let the artist find its way with the machine. And as there was not many robotic paintings out there today, it's very hard for me to say, okay, this is the tool, it's what you do with it. I'm still trying to understand that. So each part of it is to really to learn to let go of control, let the machine, first of all, to take me by the hand and tell me, okay, this is what I can do, let's go, cool. So I will do this and I will react to that. And we slowly start to achieve, I would like to believe, a new language. Thank you very much for coming. We had this video so you could kind of see the robot in action. It's a bit difficult to actually get the machine here to such a place to demonstrate it live. A quick introduction. My name is Marvin Guelzo. I am now a PhD candidate at the University of Konstanz and my job mostly is to actually build the robot, program it, okay, not build it, we buy industrial robots as you have seen and then build the painting set up and kind of provide a framework where either the machine can paint autonomously or we can have an artist with us who works with the machine. I inherited in quotes the project in 2018 that was Linda Meyer who you saw in the video. He did most of the work up until now and I'm picking it up now more or less. And... Hi, thank you again, everybody, for coming. And my name is Daniel, I'm a visual artist. And over the last years I've been focusing on how we can actually work and practice image making in relation to the past digital age we're living today. Namely, we overrun by so much imagery and what is the difference and why painting is inherently different from actually seeing an image digitally-wise at pixels and so on. And it's worse that collaborating with the University of Konstanz since 2016. And during this exactly time the work in painting makes the subject matter of the entire work to be the process. So this is right now kind of the focal point of my work. Now quickly to go into the technical part of the machine works more or less. Firstly, the name. It's a background for electronic drawing apparatus for vivid image display. So obviously we wanted to call it E-David and then found something matching. We started it 10 years ago and it was really just that robot. In the basement it had some wooden canvas in front of it where we tried to draw in some way. We started in felt tip. We moved over to using ink brushes like Japanese calligraphy brushes and finally kind of graduated to regular painting with acrylic paint on different kind of canvases and different kind of brushes. We're doing it at the University of Konstanz as we said in the department of computer graphics because it's a bit difficult to find a home for such a project. If we did only robotics painting would be a bit, you know, it's a nice application but not the point that we're getting at and computer graphics is a bit of a sweet spot. We want to achieve with this project many years. So one is autonomous painting. Ideally we'd like to have the machine in the room, close the door, give it some instructions. Then it produces a painting by itself. We can pick it up and it's done. That would be one aspect but also to provide some kind of a tool for the artists. So either as an assistant, as mentioned by the Herald, so we could have an artist who just makes a sketch and the robot fills in all the tedious detail work so the artist can do something else in the meantime. Or as Liat is doing, figure out completely different use we did not think about for this machine because as computer scientists we think, as mentioned in the video in terms of controlling this machine, how to do fine detail work and not necessarily explore all the artistic possibilities. This project is a bit of a combination between engineering, robotics, computer science, art and cleaning because as you also see in the background down there having these robotics and paint which dries all over the place can get a bit annoying and cause some problems. We also have a mobile demonstrator for exhibitions. Not the small yellow one you saw. We got rid of that one. We now have about this tall robot by ABB which we sometimes bring to exhibitions. It's still a lot of work but maybe in the future we can show it off somewhere nearby. How the entire machine works is like this. We've got our canvas where all of the painting takes place. The robot knows it as its work piece and then by moving along certain points in the image and having a brush in its gripper it does produce some brush strokes. On the left in the yellow and green highlighted area you can see our paint repository which is just a pot where the brush is held into in some interval, picks up some paint, discards the excess and sometimes you also clean it. That's just a water jet where the brush gets... where all the paint gets removed and we are ready to move over to another paint. The entire process is supervised by the camera you see in the top left in blue. That's kind of what sets us apart from other painting robots that might exist. We have a feedback mechanism which can supervise and adapt to some errors that might occur. So here in the top left you can see what the robot in autonomous mode would start out with. It's just a normal JPEG. You would hand it to the robot and it would compute some kind of paint recipe and say I want seven colors mixed in these ratios from some basic paints. I put it in these slots and then it can start painting. Initially it just places some stroke to try to get certain areas to match the input image. So the blue sky will deposit a lot of blue strokes. The yellow sunrise area it will take its yellow paint and put it over there. The initial results of course are not very good because we're trying to get ahead quickly. We're using a big brush. But after we deposition some initial paint we create a difference image which you can see in the top right. In red the areas the robot considers it more or less complete. So it can't do much more to improve this area in terms of getting closer to the input image. But the green areas there it can expect some more progress. So for example the sun, the yellow area is not bright enough so it's marked in green while the beach in the front it's dark-ish enough for this level of detail. So it just skips that area for now until it maybe moves on to a smaller brush. Based on that we compute new strokes which are a bit difficult to see but this is our live debug output which we can see while it's painting which are then stored and executed in the painting process. And after repeating this many many times like over one day to sometimes even three days we get a result like that on the left which is kind of close to the input image and we can definitely say we have reproduced that image in some way. Our early attempts using only the ink brush look like this and we already we see some mixing like long strokes for the tree and short strokes for the grass below it. This of course was done by masking the robot doesn't really know what it's painting it just follows our orders in this case. And after, as I said, after the inks we moved on to acrylic and these are kind of our flagship examples the best results we've got so far building in the harbor of Constance and a car in the desert and you can see the details which the robot has managed to achieve using its feedback mechanism. You can also see some problems we still have like the antenna is a really small detail and our current mechanism it's not perfect. There are many issues in which it bumps into and I also noticed it's kind of an art to figure out what kind of input is good for the robot because it doesn't have a chance to produce a good result because we do have a bunch of stuff we would, we don't show we kind of got rid of. Speaking of stuff we got rid of. Okay, so there was a kind of a side effect happening while the robot is operating in automatic mode and beside the main painting it's actually creating an extra paper and on this paper the robot has been programmed just to sweep extra paint and it was being dubbed first and the brush is going to take in paint it's taking way too much so it's going into like the small paper to make those side kind of a small brush strokes and I found these actually paper to be much more interesting than the painting itself that was being produced by the system as someone who's coming from a different kind of a background of painterly background I always looked at those painting thinking why would you like to do this there was this interest of why we do what we do and I am coming from like I actually got educated in painting department here in Leipzig it's a very classical academy for painting in traditional European manner and when I graduated I remember like standing there and looking around thinking hmm it's all good but where I am coming into this entire discourse I'm coming originally from the Middle East I have a different kind of a background cultural background that make you think why do we adapt different kind of certain of aesthetics and how we walk around it and one of the major question that I came to understand in my exploration is that what for me will make in a painting right now the most interesting is looking into different structures that shows logic in them rather to be at the core stage more interesting than into going into narrative painting namely things with let's say semantic evaluation which is I think for the computer part as well very interesting how can you make the machine aware of breaking the painting into narrative elements this is a sky, this is a house, this is a bird how they can actually be taken apart together so looking at those works I start actually taking those pages and I interact with them, I edit some more unconscious kind of a brush strokes to go together or like more gestural one to go on this like very structured and coming from the algorithm of the system at the same time the machine the paint is dripping there is some kind of irregularity so it's always about this small relation between things that have 100% organization and things that are a bit less so if I would kind of try to think of a metaphor of it it's like taking the painting around and to track it as a rubber that you kind of pool as much as you can till the point that if I would just stop pulling it right now it have the strongest like gravitation to each other back or it can actually snap and rip and this is the point that I'm kind of investigating with the help of working with the computer graphic department so this is just kind of implementation that I start to understand that to actually before you even go and start talking about painting wanting to talk about the first thing that what painting is about and this is about brushstrokes it's about lines and this is the first thing of recording of movement and we have today not today for many years already the use of computational aesthetic and power in art especially in music however there is something about the sound that it's been done and it's gone and painting allows you to actually record every movement to record every action and to go above it and to manipulate on it and I found these elements to be specifically interesting and that was the base of the creation of some of my earliest work I've done in 2016 in collaboration with Thomas Liedemeyer at the time at the University of Constance where I took different kind of parameters that goes very basic mathematical structures and to see how one can build different kind of all over composition all of these works actually be done on the Gestalt psychology writings and I was just trying to take those ideas and translate them into basic mathematical equations and execute them into painting the other level that I started to investigate is the relation of a single brush stroke that you could have seen in the ones before in relation to actually how many brush stroke do we need to actually create a form and how that is being put in a composational area so it's a very technical questions but at the same time those structures are the ones that create and build our perceptions one element when the individual strokes become a part of paint so when the one become a group and how do like we understand when we get to the point that we understand how we make the machine to understand those situations and when to use different strategies when making painting back to be integrated in the visual feedback more autonomous painting methods another important part that we worked on is how we can make the strokes to look more organic so if you look in the right low you can see that the strokes are very straight they look more mechanically done so how can we introduce things that just a small deviation of the strokes or the brush not to go like 100% like 90% but a bit like this and a bit like rotation and saying we want to make it a part of paint here but don't make it directly some things to go a bit out and begin so all those small details is something that we are very important in creating something that we as people can relate to and feel this is actually a work of art and not something mechanical and distant from us by increasing the fields of organic representation so this is like one of the work play around that I've done with Thomas in the beginning of my residency showing the work on the right is done 100% by the visual feedback while the second was being done according to breaking down the entire concept of the painting into masks and using only three colors so both of them are being done with the machine just using different kind of strategies extracted from this picture of this lovely guy that he does not even know that he's being shown here right now I have no idea who he is actually lost the time and that developed to the work that we have kind of funny idea let's make of course what is like when you acknowledge a to be as an entity then this is a question one can take as well into the art world discourse, the commercial art world discourse but it was the classic self-portrait of the painter and this is kind of the preparation phases of how we done it and that was going again into the strategy of actually breaking down the painting into masks and giving different strategies of how we do those paintings so we actually based this entire work on a photograph that then I worked out a bit with Gimp and using the custom made the program that Thomas built for me I created different kind of masking and strategies of brush strokes to create the painting so this is this David self-portrait that we find to be very hilarious even signed his name and going back like after that all of those workers still had to this point of what makes a painting more interesting why we're still interested in it, why in this time that we overrun and overload with visual imagery painting is still important people still going to learn painting we still do that and I'm always like tricking between the elements of making and the elements of the perception of how we see it and that was a kind of this work was built on the idea of like how would I say like the action of making a structure of just a stroke it's very do the looking structure that was done by the computer and something that is still complex enough for human being not to be able to repeat only a machine can do it but the deviation of the changes and the variation of the work is actually being consumed or created through the use of paint so how often the robot takes paint more or less and then just make a feel oh it feels like it was being painted quicker but it was actually a bit drier so how one can make those small manipulation and tricks for the creation of painting only the domain of a brush stroke speaking of brush strokes what we have currently as you can see we're kind of painting in always similar style we have got a brush we apply it at the same pressure level to a canvas then we get some paint transfer and that's how we create our images because it's very difficult for a robot to actually handle a brush we don't have a tool center point which everything revolves around in these machines usually you have a welding gun you know exactly to the millimeter here the effect of my tool will be localized but brushes they deform not transform that shouldn't happen they get dirty it's really difficult to simulate a brush because of these many hairs they vary their properties when they get wet and so on and the provided software we have for handling tools and robots only works with these hard tools basically not soft tools like brushes and that's a problem for our painting and also what we can provide to the artists as a control element in here in the painting process because for example it's difficult to define a certain edge in a painting the painting process will draw over at many times and since the brush deviates a bit every time and the corner will just blur out hence we can't do many details and we don't really know the side effects the movement generates so if you take your brush move it up and then perform a sharp corner it will slide around on the canvas a bit and produce a certain effect we can't control for that yet so our solution for a stopgap solution in the first iteration is to actually measure what our tools can do in terms of width and lagging behind so if you look at a brush like this we think we have a nice defined tool center point but as soon as you apply it obviously it blurs out like that it deforms it might stay like that and as a funny side note if you overdo it with the robot and have some sign errors you deform it permanently so our solution to this problem is doing this we have some automatic procedure I'm not going to go into too much detail but basically the robot can paint strokes at no pressure levels which we limit to avoid what I showed you previously and then we can use our feedback mechanism to measure how broad a certain stroke gets at which pressure level and using this we can accurately reproduce strokes for example in this case we can account for width we can also if a human paints the stroke you see on your left with the red centerline we can record it using our feedback mechanism and paint some first attempt like we do normally with the yellow centerline then we try to figure out what went wrong when painting this where our brush might be lagging behind where we're not applying too much enough pressure we compute some correction and can perform the stroke again this happens over several iterations and in the end we can see a quite nice end result but here you see there's already a movement between our initial attempt and what we can actually produce and by storing these strokes we can kind of build up a database of knowledge over time which then artists can use to tell the robot put stroke number 25 in this exact location thus enabling much finer grain control and kind of as a party trick by just reproducing single strokes and writing in a funny way we can also make the robot just repeat your work and single strokes so as you can see like our work has changed the material before the work was done lots with acrylic painting means that you can work in layering and in order to really control and to master the way of doing the brush strokes we decided to move back into working more with ink and paper where we can actually have the ability to see every stroke into its details and to understand how we can control it better how can we pursue better what we want to achieve this is just an example to see the entire kind of spread of a stroke using different kind of points to say ok we have just the time in the second one just the time how long you're just standing so just to see how the material is reacting so we are talking here about not only the computer itself and the stroke but actually the material itself the paint is reacting if the brush is a bit too wet it's actually expand more into the paper a bit less or a bit more like the gradients of the color is changing according to how often it's actually deep into paint so in this situation it was deep every 80 strokes in paint but every 10 strokes in water so the water as well slowly diluted and there was kind of situation of gradients that happened through this kind of a system that goes to us in Indiana's viewers and together for this work Marvin here had built this software for me that generated particles and those particles have actually been created due to Newton's law of gravity I just decided to use this as a general law of how we can spread structure how things are operating in space if I'm looking at the painting as a space of creation like in the world and we can scale it into different situation and different sizes take all the elements from it and I use actually this work to create an entire installation build from many elements to execute at different parts so again this entire work was being done using ink and the rice paper here just to show you again the differences of how just using a different brush to make the outcome to be definitely just totally different although it's the same sections the same information that was being given into the robot for painting the fact that the robot is not aware of the material he's working with a lot of situation happens where the material is getting really deformed what probably as a train artist I would think it's maybe wrong or maybe I should not do it or I will try to avoid it and the robot is not aware of those situations but it's actually a way to handle with materials to say oh it's actually interesting situation to have those holes happening in the papers like what happened where it's too much going over the line so those work were being created I think 20 something rice paper work that I painted taken from this particle generator and being hanged in a room in an installative way it's given us the point again to observe painting not as one big thing that you have to look simultaneously but actually forcing the viewer to walk by it and to stop and to look about every little element and to understand and discover the process of making it so making the process much more visible this work again was being later developed into how one can exhibit the work so to go outside of the painting from just things on the wall and to take it out into the space so therefore there was a decision of framing those works in double sides using hanging frames and you can actually look at and see them from the side it was being painted assignment was being just absorbed and creating a spatial area where one can work and experience the process of the creation and not only the finish object as a painting so that's kind of what we're at today but the project is of course going to continue and be developed further over time one major area from my side I want to work on is painting techniques so right now we've got one style we can do well but of course it would be better if we could more emulate what a human actually does a human does not stand in front of his canvas and play stroke after stroke in random places we would like to make the robot scribble on the surface, mix some strokes deliberately maybe perform dabs and vary that style within one painting maybe as is appropriate of course getting a machine to actually do that requires a lot of input of certain movements and it's a bit tricky also I mentioned the database of movements we'd like to build so we know this movement will give us this exact result so we have a much more predictable process and something you can pick and choose from and maybe even tell the robot only focus on this type of stroke instead of this so you can in the end guide its style in a much more concrete way than right now where you can only very stroke length and stuff like that also our system design you saw a bit of the development we started out using a plastic bucket with some grid in it to clean the brush we're moving on prototyping as we go on and trying to develop better hardware so the robot can actually have clean tools, save tool changes and so on also openness is a problem at the moment and it's a bit difficult to get other codes into our project or to just get strokes from somebody else he would like to have executed on the machine so we're reworking the software architecture to make it possible for other collaborations because it turns out we've got competition in some way in a good way in Turin I think there's also a painting robot being developed at the university we're cooperating with Shenzhen University who are kind of reproducing our robots and then we have two of these systems to work with in different places also because I have to mention machine learning we need to figure out if it makes sense to actually use machine learning in our approach if there is some easy way to do it is it possible to maybe simulate some creative aspects you know these Google Deep Dream thingies would we take that technology get out some strokes and have it output on our machine and the interesting side note who actually gets copyright if we allow the machine to do more and more by itself so at some point we had a law professor in our lab and he asked how much are we still doing in this process because once the machine starts doing enough and reaches what in German law is called Schöpfungshöhe I think it will get difficult to attribute what is actually owns the painting which is being created and some future issues for Liat okay so the work that we are being done we are still working on I stopped for me right now in the moment of how can I use this tool to extend myself it means how can I go beyond the limitation of myself as a human being what I can do first of all from my own perception because when I am standing in front of a canvas a painting surface I am looking from one place my brush strokes ability is only limited to my sides of my arms so I would like to integrate more into those kind of elements to see how can we use this tool to create maybe a larger scale works at the same time idea of how one can translate data taken from different elements into the painting so the painting will become an abstract representation of maybe social structure if we take it from the movement data that I am right now collaborating with Casa Paganini in Geneva taking data from behavioural movement information and using individual human beings actually to be translated into brush strokes further it is the idea of how the machine could become more and more organic in its behaviour so it is not only the way it looks but the way it works and operates so right now it works very deterministic ways like if you give it a JPEG it works from it according to the parameter it is being given or it is giving through a deterministic way of an algorithm being given and just filling it up I would be very interesting to get to the point where the machine will integrate what is considered to be a mistake in the painting process something that we didn't expect it to happen we didn't want it to happen but it is there now and it has changed the entire world situation so how can we integrate and react to each other and the conversation that is happening between me and the machine is actually there for opening a new round for creativity because there is something almost objective that comes from this kind of an input that makes me question myself of my decision making process and my habits of visual information that is kind of what we have to say so far if you want to figure out more about the project we have got a website from the university if you want to see more of Liat's artwork she has got and what we presented today more or less the stroke thingies you can see in our last publication at MDPI self-improving robotic brush stroke replication a professional would have added the link here but we added that last minute I'm sorry but you can find it easily I have to use the term you can easily Google it sorry yeah that's our talk so far if you've got any questions I guess knows the time super great I'm actually wondering how you make a robot really shift at a certain moment you have something like hey come on let me get into this because isn't it there because that kind of communication would be interesting with an assistant you just kick it and he or she goes away but he David it could kill you if you actually that experience we have in our hacker space as well we have a diamond cutting thing and there the phone drops sometimes whatever we have lots of questions I assume really start number two please thanks for the talk if I understood correctly it's not a printer because you get more because you get visual feedback yes okay so I'm wondering what how does it actually can you scale beyond just visual feedback I mean there are a lot of different ways but is it feasible and that what would be the consequences because it could be another social feedback by for example us or the public of the audience judging the result I mean there are so many ways or how is it feasible practically speaking what are the consequences so the question is can we introduce some other feedback other than just visual information higher level more conceptual yes you could so in theory we don't have it at the moment obviously but if we had a way for somebody to kind of give the robot an overlay and say this corner isn't good it's kind of deviates from my optimum by a certain value then we would also need some function to determine how it should improve that and then we could also include other kinds of feedback of any kind basically the trick is always to find a function which describes how big the error is for the feedback and what can be done to fix it in cases of just black or white or color color spaces it's kind of easy if it's too gray you paint more white color over it but in here it gets difficult but it might be doable or do you have any reason well I think there is a lot of challenges to use of the visual feedback because there is the question always of perception when the visual feedback that what is actually getting right now it's from the photograph where everything is being scaled equally and there is certain kind of parameters it's checking if what Marvin just said is darkness, the level of areas and so on and it still does not have the tuning of understanding for example semantic situation to say oh here I painted the tree as part of the sky and it's supposed to be two different things this is I think the interesting part for us as human being to interact and slowly teach the machine and see what it's doing and correct it and hopefully like slowly the machine will remember through it's doing to get to a more defined or abilities of feedbacks that are probably much more developed from what we have right now I would love to see a live demonstration here number one, please shoot thank you for this very inspirational talk I actually got several questions but I will limit myself to two if it's one so first question is could you please elaborate a little bit more on this aspect of cleaning since you said it's a combination of art, robotics and cleaning and then question number two would be about what was for each of you like the most surprising in this project and cooperation so the first question was what's all this about cleaning I mentioned I mentioned it so much because it's my major headache in this automation process right now because over time when the robot paints a lot with that brush some stuff dries, some stuff it sticks in the brush and it's really difficult to get out then you contaminate your other paints and everything tends towards some grey brownish stickiness I think it's the difference of how computer science used to work in certain kind of a lab and suddenly they have an atelier happening there so when I was there for me it was like a fairly clean studio here so I think it's a lot of matter of habits but of course when you work with machinery and you work in a scientific environment you need to measure everything so keeping a clean environment is important to actually follow about the results and that I think one of the reasons why it's such a big headache for Marvin in this sense wouldn't it be better to come over to the atelier then with your robot in an atmospheric environment two different cities that's too far away from each other we tried that one and also another problem is if you see the table with the colours on the left there's also a magazine where the robot can swap out brushes and once too much paint had accumulated in there the tool did not properly enter the slot it was misaligned the robot then almost threw over the table when trying to switch tools it's also just a mechanical problem after some time the floor doesn't look like an atelier now I think yeah when you're there you see a lot more and the next question was should we do the second one what was the most surprising part of it I'll be quick when I entered the project I was like do we have a robot can I work with it, do you want to know what we do no and then I figured out this art part exists I had no connection with it at all and that was kind of surprising for me to stumble into this world which also is included in the project we can imagine number two please yes oh yeah the surprise I think from my side it's very similar to maybe my Marvel part because when I came I thought okay it's going to be coming from a very different world how can we discuss very similar issues and it was a pleasant surprise to discover that we have similar questions and we look into the world into very similar ways but the tools that we are doing and the perspective that we are doing is very different so the attempt of trying to communicate those two was for me the most interesting part and just looking into structure to understand how can I use computer language structural language to actually build painting and how similar and close to each other those two are organizing chaos or chaos organizing organizations something like that entropy please sir can the robot make use of some tactile feedback for example potential sensor in the brush we don't have it and it's very difficult because the forces that should be occurring are very small and might also change because of stuff accumulating and also as soon as we introduce the wire into a brush for example to measure something that will change the outcome so no but we are thinking about maybe having some life supervision of the brush with the camera but then you get the problem of a green brush on a green background and trying to separate that it's difficult so we're still painting more or less blind in that regard if you have a solution come up and hand it over number two, number one please as an artist experimentation is very important to try experimentation is very important to try out new things to do things that you maybe didn't plan to do and so on what happens if eDavid gets a brush that it doesn't know if there's a tool in the toolbox that it picks up and then just applies it to the canvas I actually done that with it several times and it just makes it keep painting especially when you work with a visual feedback because things happen that it didn't expected to happen so the machine think it's wrong I will just do it again and it could keep probably paint like this for eternity because it doesn't have this experimentation something that you can tell to a human because then I experiment something new and I understand something from the process and I'm learning from it the machine is still not in the situation that it's learning it has a goal, it has its tools and it's know how can I use those tools to achieve my goal it doesn't understand swearing or something like that no but sometimes I do swear at him or I can imagine it depends we have a couple more minutes it's very interesting what you say obviously it's dangerous because the machine does not have much sensory feedback but if we do it in some controlled way where it knows limits so it must be constrained in that way we could definitely implement methods where it just paints randomly records it and then somehow uses it in some further experiment number 2 shoot, shoot thank you very much I have two more art historical questions first the notion of the painter the notion of the artist as painter is a notion that is marked with a lot of markers of gender for example that paints but also of virtuosity of authorship and things like this have you think about dealing with robots with other media that isn't marked this way or maybe with painting but differently or how can one change this notion that kind of gets repeated and also I was wondering moment yes I was wondering about people like John Tingerly like people that were dealing with machines already I don't know in the 50s or something but they understand themselves as sculptures too like they were producing a machine that was mechanical and there was painting like a painting machine and of course what you're doing is differently since it's a robot but like if you relate to that tradition thank you so for the first question I am a painter and it's important for me to state it although according to my academia where I studied they consider it not to be a painting anymore because it's not exactly done by the authorship of a human being although I'm working with paint and brushes I was being sent directly into a video audio department when I wanted to produce to show this project there's like there is no audio video in the work why are you saying me there so when it comes to using different media for example now the project I was doing in Italy was working with using digital representation and different levels of visualization that I found interesting but to be honest for me the interaction between materials is the most important part and using the machinery right now is an extension of how one can use technology for its own creativity so I try to see it in the relation of art history as an opportunity to establish a new aesthetic and not to go into the discussion of authorship although it needs to be there and it has to be there because of course when the camera was introduced photography introduced like in the beginning considered not to be even art and it was very low value because the machine done it like the person today the discussion is not there anymore so I think there is this freedom that I have right now in this domain because I don't have the ultimate start to sit and look upon and say I have to imitate those like old European men in the way they've been painted because it's new so I can do whatever I want and no one can tell me it's right or wrong and this is a great privilege for the second question about the way of presentation I will still go back to say I am a painter and for that I'm interesting in the image itself and I think there is the fetishism of the machine especially today when you see the robots and people don't even care what it's doing to see a robot moving it's like whoa it's so cool and it's nice and I think you can do a lot of things in levels of interaction works and but I am really not into this I'm not a performative artist I'm not interested in making like kinetic art I'm more into really just how can you represent movement in the still in the still image and how that could in a way fight back against the overload of visual imager we have today because we maybe look at the painting for three seconds we get it we go on like as a visual artist today I'm being required to know who are my contemporary fellows I cannot be at the same time in Hong Kong and in New York and in Paris so I check them out in my shitty phone and I see this image and I say oh I get what it's done there and I'm continuing so it's a different way of experiencing information and I want to use this machine as a way as a tool to communicate back what the information coming from the machine from the computer that we are all using right now and reconnect and re-communicate with real materials again almost in the factory of Warhol but number one please we have just a couple of minutes left so thank you also for my side a very interesting talk and I'm very curious about the machine learning side of things so of course there you would need to define something like a cost function or a loss function which describes how well the system is progressing and I'm wondering both artistically and scientifically how would you go about what are you thinking about defining such cost or loss or quality functions that's exactly what I'm working on right now and what's giving me nightmares because it is very difficult you try to reduce the problem and then you already impose extreme limits on the machine so what would maybe be a good first step because at the moment we do no learning at all in the current setup would be reinforcement learning and just have the machine approximate some kind of template and maybe afterwards we could say certain class of stroke is aesthetic in some way but it gets very vague and difficult so we'll figure that out in the future I hope also I guess artistically it's a very interesting question right have you talked about how you define those questions this is always what is wrong and what is right in the arts so it's it's an issue if we find something I think it would be troubling if we could say yeah this old painting is 85% aesthetic yeah but I think there is today an amount of studies that I found in our science that I found a bit scary to be honest that's trying to quantify those elements and one cannot look at it as an absolute because it depends on the place and the time and the history that you're coming from and this is a huge challenge that we are facing today as we all live in this global society and everything is flattening out and everything is being done and being taught in art academies across the globe are kind of similar so I found it to be very dangerous and this is why for me this working with the machine is not about giving it the autonomy but using it as a tool to maybe even like learn its individual user so maybe we can create several kind of users and each user learn the machine will learn the habits of its user and therefore we continue into a relative absolute values I would say I really have to apologize because we have to close this session I'm looking forward to these new chapters of artistic behavior and humanity I suggest to continue these speeches afterwards somewhere here in a car in a bar or you will be will be available here I would love to thank Lea Marvin E. David of course and Professor Doshan that's the name isn't it please thank you