 We're about to do our second lecture of the day in this hall. It's called Transhuman Expression. And it deals with interdisciplinary research in painting and robotics. It's at first guided by a professor who is not here currently, Oliver Doshan. And here next to me I have the PhD candidate, Marvin Güldshoff. And the artist, the one who is working with these robots, Lejad Kreiler. They will discuss both the technical challenges that go along and the innovation aspects that go along in the development of an e-David robot. I understood the robot is there since quite some years yet, but it's always in development. And what I see then somewhere in the future is probably that we can imagine the old-fashioned serious painters of like Rubens or Rambant, Van Dijk, who had ateliers who filled that with assistants who were very skilled in certain specific aspects. But imagine then someone like Andy Warhol and his factory or so that would be working in a robotized world. And I think these people are going to show us or demonstrate us the challenges that are in there. Because next to that kind of understanding of working with robots as assistants, we have to look as well at the social and artistic practice that this way of working actually offers us. They do not go on strike, I presume. They take sometimes the break. And at this point we come to our translation of the lecture about transhuman expression or transhuman expression by Lied Graver and Marvin Goetze. That's still masculine. Okay, guys, are you ready for this? Then I would say light the fuse and put it in play. Let's go. As the robot is recording every brushwork it has done, you can always turn the robot into rubbers. Can we repeat these rubbers and use them in different variations? So minor and barely seen that make it, I think, to be something that the viewer subconscious will recognize it, but our consciousness knows how to understand it. I think that could be a very, very interesting effect to play with. I can try to play around with this completely abstract painting or with a picture painting. And in both I have played many brushstrokes, but in a different way. The location is a bit different. The position of the brushstrokes, so which top, which bottom is, is different. And we play here with the concept of entropy in the painting. The idea of just the chaos is increased in an organized body. So each time that I'm actually making a variation, every time I make a variation, then I want to increase the level of chaos. But the painting as a system has to have some kind of logic. And this logic is very difficult to control, but a computer can help you to create it. We built this painting machine E-David. We built this painting machine E-David almost seven years ago. I worked in computer graphics for more than 20 years and especially in non-photorealistic rendering, where we create images that look like they were created by an artist. At some point I had the idea not only to create computer images, but to draw. So to build a machine that shows the night what an artist does. And that was the beginning of E-David. Basically computers right now, they used to just follow simple rules and they used to just follow simple rules and they used to just follow simple rules and they used to just follow simple rules and they used to just work with the machines to implement those disorderly and self-driving cars.现在, we're giving computers then to compute and then to even be able to advance by just going into art where as a human we can ought further in the way in the arts and it's pretty creatively we are trying to transfer that because that completely contradicts what computers once were, so completely deterministic systems that repeat the same work again and again. However, I have no interest to make a thing that's going to be run directly only by the world. I'm not interested in drawing pictures directly, but I often want to react to what the machine does for example. For example, give a klex color to the picture and see what it does in the visual sphere. The lithography and final art direction create their own medium, their own aesthetic. The robot is not just a printer, but rather a machine that observes itself and creates its own painting process by analyzing pictures. This is a big difference to other painting machines that work like printers, which are just a predefined image. First of all, we thought that we could recreate this feedback loop. Then there are new questions about what creative processes are at work when people are painting. The small robot is not as big and not as scary as the big one. The big one is really huge and you can't really work next to it, because if you make a wrong move, you can really hurt yourself. You can just tell him to do something else, for example, so that you can do something else in parallel. The simple question, what is creativity, is very interesting, because we can't really define creativity and as long as we can't do that, we can't bring it to a computer. The combination of all the real-world experiences that we have with the inner world that we have in our imagination, both of them have to be combined in a computer and then put together in a process. As a human, we know that creativity exists, but as you can see, I can't really define it and I can't even translate it into a program. A year of time I was studying some Japanese Mandarin. What I learned was the body of the act and how to move the body and how to use tints on rice paper. You can't really control that. You can't control how the tints spread on the rice paper, you can't delete them or change them. I think one of the most interesting points for us was the discussion about control and the loss of control in the Mandarin. Informatics want to be able to control every small part of the program. Most people who come here think that we want to reproduce other artists, but that's not true. We want to explore how artists work in general, what strategies they follow and not really copy artists. There are different lines that you can see. But the next thing we have to do is look at these focus points. In the end, this is one-dimensional that you can have a line. Actually programming a robot to paint with you and not to paint for you using the robot as a tool. That helped me find out how my individuality is part of the whole and different materials. Currently in most of the cases the machine works in fully automatic mode. At the moment, most machines work fully automatically and that fascinates us. You start with something and the next morning after 10 or 12 hours of work, you get an interesting result. But my goal is not to paint the machine myself. I would like to work together with the machine. My dream would be to teach the machine a very efficient process that I would like to see on my canvas. I had to learn that the best is to let the artist find his own way with the machine. Because there are not many robot paintings today, it is difficult to say what the tools are for. That means we have to learn to give the control to the machine and to let the machine take us by hand and to say, I can do that, let's start. And so you react to it and learn a new language. Hello, thank you for coming. We had the video so that you can see the robot in action. Because it is difficult to bring the machine here to demonstrate it live. A short introduction. My name is Marvin Guelzo. I am a doctor candidate at the University of Constance. We buy and make the set-up. And we do the framework so that the artist can work with us autonomously. I inherited the project. Thomas Lindemeyer did most of the work and I will continue now. Hi, thank you again. I am Gaia and I am a visual artist. In recent years, I have concentrated on how to combine visual art with the media nowadays. The question is why painting is so different from the pictures we see digitally. That is why I work with the University of Constance in 2016. During this time, the work is painted so that the point of my work goes into the technical part. The machine works. The name is an acronym for electronic drawing apparatus for image display. We wanted to call it E. David and looked for what fits. That is the robot. In the basement, we had a few wood lines with which we tried to work. We used ink brushes like in Japanese calligraphy and then developed it to acrylic painting. We used different brushes and different colors. We do this at the University of Constance in the Faculty of Computer Graphics. It is difficult to find a project at home because only in computer science you can find a nice project. The goal of the project is to paint autonomously. We have to give it an introduction and then make it ourselves. We also have a tool for artists. As an assistant, we have an artist who makes a sketch and the robot can make the details completely different. We didn't even think about that because as informatics we go more to the scientific side of how it works and not to research all the artistic possibilities. The project is a combination of engineering, robotics, informatics, art and cleaning. Because it can go wrong and it can get a little annoying. We also go to exhibitions, not with the yellow one you saw in the video. We have another exhibition with which we sometimes go to exhibitions. It is quite a lot of work, but maybe we can show it in the video. The complete machine is built like this. The robot knows his area and he has a brush in his gray farm. On the left, in the green and yellow, in the yellow area, you can see the paint, where the robot is painting the brush. There is also a water jet where you can clean the brush. Everything is monitored. The camera that you see on the left in blue distinguishes us from other projects. We have a feedback mechanism that can monitor some mistakes on the left where the robot starts with a JPEG. I would like to mix some colors in these areas and then I can start painting. At the beginning he only makes a few brush strokes and tries to generate some matches. At the beginning it was a bit difficult, but after we added some extra colors, you can see it on the right. The robot thinks it is so complete and can't improve it but in the red areas, in the green areas, the robot thinks it can improve it. In the sun, you can do more, but in the foreground where the beach is, you find that it fits and then you skip it to a smaller brush. This is our live debugging output where you can see what happens and you can save it and repeat it over and over again. After a few days you get the finished picture and that is close to the original. That was our first try with ink. It has long brush strokes for the tree and short brush strokes for the grass and here the robot only follows the instructions. After the ink we continued to add acrylic colors and that was the best result we got. Once a building in the harbor of Constance and then a car in the desert. You can see the antennas which are very small. The process is not finished yet and there are many problems. It is hard to define which input is good for the robot so he can produce a good result. We have a lot to show when the robot is operating. There was a side effect with the robot in automatic mode. In addition to the robot being programmed just to sweep extra paint the robot takes way too much. So it is going into a small paper to make those side strokes. I found this paper a lot more interesting than the drawing that was produced by the system. I have a different background and asked myself why I want to do that. There are technical challenges but also reasons why we do what we do. This is my training from Leipzig. This is a very classic academy of the European school. I thought it was all good but where do I get into this discourse? I come from the Middle East and I have a different background that what for me will make interesting looking in the structures is the structures that show the logic of the painting. I think there is going into narrative painting things with, let's say, semantic evaluation. This can happen with semantic evaluation. This is an interesting question for the computer to recognize the different areas of the picture. I take these paintings and interact with them and add brush strokes and at the same time the paint drops and there are these small inconsistencies and the contrast between a very organized part of the painting and a less structured part of the painting. You kind of pool as much as you can and sometimes you pull until you think that the gravitational force is either very strong or it would even tear. That's why it's very nice to work with the computer graphics department to work together. And nowadays we have been using this computer aesthetic in the music but painting lets us draw every action and play it again. This painting was very interesting because part of my early work where I take different parameters to create these simple geometric structures to see how to put them together and I just tried to take these ideas and to transfer them into basic mathematical equations. I also researched how to use a single brush stroke and how many brush strokes you need to create a shape and how this can be added to the composition. These are very technical questions, but at the same time you have one element. When the individual strokes become a part of paint and when we get to the point where we bring the machine to the painting and how it can follow different strategies then it's important. And another part is how we can make the brush strokes look more organic. Here you can see that the brush strokes are very straight and how we can add small deviations that the brush strokes are a bit curved or small details can be added. This is very important to me so that people can identify with these pictures. This is one of the work play around that I've done with Thomas in the beginning of my year. This is one of the plays that I did with Thomas at the beginning of my time and we took the picture in masks and used only three colors. And I don't know who it is but we used different strategies to create these pictures. This has developed into the question one can take in the art world this is the classic self-portrait of the painter and here you see the preparation phase and that went on to the phase where we destroyed the picture in masks and thought about different strategies. I have a bit with GIMP and developed a program and then we developed different strategies to create the picture. This is the self-portrait of E. David. He even signed it. And when all this work was done we still wonder why this picture is interesting. And I'm always involved in the perception and the creation of the actual structure of just a stroke. It's where we do the looking structure that was done by the computer but also a person can't really do it. And there are different variations for example how often the robot takes up new colors and how to make these small manipulations and tricks to create a picture and that's a question of the brush strokes and where we are talking about brush strokes we are currently working with very similar styles because it's very difficult for a robot to use a brush for a robot and sometimes with other tools you know exactly where the effect is but brushes get dirty and they form. It's very difficult to simulate a brush stroke because they have so many hair and the software we use is a problem for our painting and also for what we can do with the artists because it's very difficult to define a certain edge because the painting process is often overshadowed and that's why we can't have too many details and we don't know exactly which side effects we want to have. We can't control that yet. Our solution is to measure what our tools can do with the width and the distortion. When you look at it it looks like we have a defined center and you can actually, if you're not careful, shape the brush permanently and our solution is this. We have an automatic procedure, I won't explain it in detail but the robot can create a brush stroke with a certain pressure and we can measure how wide the brush stroke is at certain pressure levels and we can reproduce brush strokes in this case. We can also control the width and when a person sees the brush stroke on the left side, we can record it and then we try to find out what went wrong where we didn't use enough pressure and calculate a correction and try the brush stroke again. We do this with several iterations and in the end the result can be very nice but here you can see a big difference where we can save brush strokes and we can build up a database and later we can say, hey, take brush stroke 25 in this position and with that we have a pretty big control. We can also bring the robot like a medieval monk to reproduce things. So in our work we have the mariette to really to really control the width of the brush stroke we tried to work more with ink and paper where you really see everything, every stroke in detail and then you can understand how to better control it and what you actually want. Here you can see an example of how you can use different brush strokes with dots. So in the second one you just look at how long you're standing how the material reacts. We're not only talking about the computer and the stroke but also about the material. And the color changes a little bit and it changes depending on how often you dip into the color so here all 80 brush strokes are dipped into color but all 10 strokes are dipped into water. And then the color got a little bit and the radians got a little bigger and here you can see some drastic changes in the work. Marvin had built a software for me that generated particles. The particles are generated with Newton's laws of gravity and the question is how objects react in space with different elements. And I used that to create a whole installation with different elements. So again I worked on different parts and I think in the right way. So it's the same section, but the same information was used for the image. And the robot doesn't know which material is used. That's why it's sometimes very deformed an experienced artist knows what he's supposed to do, what he shouldn't do but the robot doesn't know and that allows us new ways to go and new situations to explore and use too much color on a paper. So these works were created in 2070 with rice paper from this particle generator and they were hung up in a room in different ways. And then you have the opportunity to see and you can walk past and look at the elements and you can experience and discover the process of the production and the whole thing was developed into the question of how you can experience it how you can imagine it and then it was decided that the robot was hung up in a room from two sides and then you can walk around and there's a room where you can learn more about art and look at the finished object. That's where we are now but the project goes on and it's a special area for me is painting techniques and we have one style that we're good at but it would be better if we could do something better because people don't just stand in line and make single lines but that the robot can mix more between different techniques and to get the machine to do that you need more input from different movements and I already mentioned that we want a database with different movements and with different brush strokes so that we know with this movement we get this result and then you can say to the robot that we can define the style a little better also in our system design you can see that we've always used a plastic cup with water to clean the brush and we try to build better hardware so that the robot can use better tools, safer and maybe even better results it's also difficult to integrate other codes in our project for example if other people wanted to have their code with our machine and it turned out that we even have participants in the University of Turin our robot will have a new system and then we have two systems in different places and of course there is also machine learning and we try to find out if there is any sense to use machine learning if there is a simple way to integrate machine learning is it possible to use some packages Google Deep Learning and the question is who gets the copyright when the machine does what it does we had a one professor in our lab and he asked how much we can do with it and how much we are still involved because as soon as the machine does enough and gets higher production then the question is who owns the email that is produced ok so the work that we are being done still working on how can I use it to imagine myself how is my perception of a picture so when I look at a picture I look at a surface and my abilities are limited to the points that I can reach and the question is how can I use this tool to generate larger works how can I use different data to generate art maybe you can use different data for example from movement information that you can translate human movement into brush strokes also one more question is how can the machine become more organic in their movements at the moment it is very deterministic you give it a picture and you can reproduce it or it is determined by an algorithm where the machine will integrate what is considered to be a mistake it would be great if we get to a point where the machine makes mistakes in painting also the question of interaction with the machine with reaction and action because then space would be opened for creativity because then new aspects come that you can ask yourself that is kind of what we have to say so far that is what we wanted to tell you if you have more questions we have a website if you want to see more art works what we presented today you can find it in our publication in mdpi self-improving robotic brush stroke replication but you can find it easily you can find it online that was our talk and if you have questions that would be the time I am actually wondering how you make a robot really at a certain moment you have something like hey come on let me get into this because that kind of communication would be interesting with an assistant you just pick it and he or she goes away but e-david it is not just an assistant it is really... we have a diamond cutter sometimes devices are a bit out of control question to the microphone if I understand it correctly it is not a printer because you get visual feedback my question is how can you scale it? what would be the consequences with social feedback? how is it feasible? the question is can we introduce some other feedback not just visual information yes it is possible we don't have that yet but if we give the robot an overlay and say this corner isn't good it is kind of... hey this corner is not good then we need a function to improve the picture and then we can introduce this feedback so the problem is to find a function and how to correct it with black and white it is easier if it is too light or too dark you add more lighter or darker color it is always a question of perception the visual feedback we get now is from a photo where everything is scaled for example the darkness and it doesn't have the option to have a semantic connection so for example to see that hey here I painted the tree as part of the sky where there should be two different areas so there are still people who can bring the robot to understand how to interpret the feedback I would really like to see a live demonstration thank you for this very inspirational talk I have several questions but I will limit myself my first question could you please elaborate a little bit more then question number two what was for each of you like the most surprising what was for each of you the biggest surprise in this project and in this collaboration the first question was what to do with cleaning I only mentioned that because my head hurts the most at the moment when the robot dries a lot and sticks to the brush it is really difficult to get out and then you contaminate other images and everything tends to a grey-brown clarity I think it has something to do with it like computers, like informatics who used to work under laboratory conditions and when I first came here I thought oh, that's a relatively clean studio and of course when you work with machines and in a scientific environment you have to measure everything that means keeping the environment clean is important so that the results can be fulfilled then it would be better not to go to the studio but they are in two different cities that's too far away we can't move it and if you see the table with the colors then there is also a magazine where the robot can change its brush and once there was too much color and when the robot tried to change the brush then it almost turned the table so that's a mechanical problem when you are there you see a lot more when you see a lot more what was the most surprising part? the second question was the most surprising part for me when I started we had a robot and we knew what we wanted to do I had no connection to it and it was surprising to me to delve into this world and that's part of the project I also want to have Leah's opinion I think from my side it's very similar to the parts of Marvin we come from a completely different world how can we discuss similar problems and it was a surprising surprise to see that we have very similar questions we see the world from different perspectives and the most interesting part was the analysis of the structure and how similar and close to each other those two are and to see how close we are to each other entropy entropy please sir can the robot make use of some tactile feedback can the robot use tactile feedback for example the pressure on the brush we don't have that and it's very difficult and the forces which are very small and they can change as soon as we install a brush the result will change but we think to watch the brush live with a camera but then you have the problem that you have a green brush on green and it's very difficult as an artist experimentation is very important for artists experimenting is very important to do things that you didn't plan to do what happens when he gets a brush that he doesn't know if he takes a new brush and brings it to the canvas we tried that and you just let him keep painting with the visual feedback and the machine thinks it's wrong and tries to correct it it's not the same learning skills as a human the machine has its tools and knows how to use them to reach their goal but sometimes they just wear it or I show it to them what you say is very interesting because of course it's dangerous because the machine has little feedback but if we do it in a controlled way where it has limits then we can think of methods where you just paint randomly and draw it and use it in future experiments Thank you, I have some artistic questions First, the concept of painting is a concept that has a lot of features like gender and as an author the question is how can you work with robots and with different media and maybe in a way that changes the concept I was wondering about people like John Tingerley people that were dealing with machines and I think about people like John Tingerley people who work with machines but now as sculptors and there was painting like a painting machine and of course what you're doing is different since it's a robot but like if you relate to that tradition so you use the machines to create things and how do you relate to that tradition So the first question I'm a painter and for me it's important to stay even if the academic system is no longer considered as a painting because it's made by a machine even though I work with paint and brushes I was sent to the video and audio department even though nothing has to do with it The project I'm working on in Italy has used digital visualisation and I think it's interesting because the interaction between the materials is the most important and the machine is a extension of how you can use technology for your own creativity I try to establish a new aesthetic to use it to create a new aesthetic because of course when the camera was introduced it was like in the beginning the camera in the machine was introduced as art and for me it's this freedom that I have now because I don't have an old master that you have to imitate it's new, so that's why I can do what I want and no one can tell me that's right, that's wrong and that's a big privilege For the second question about the way of presentation I would still say I'm a painter and I think there is a certain machine fetishism where people don't care what a robot does because they think it's cool and it's nice and you can there are different levels of interaction but I don't really like it I'm not a performative artist I just want to see how you can represent movement in a still way and how the challenge we have today can be counteracted I as an artist have to know who my contemporaries are in Hong Kong and Paris that's why I look at them on my cheap phone that's why there are different types of information and I want to use the machine as a tool to play back the information that comes from the computer and to continue to communicate with real materials almost in the factory but a microphone please and that's a few minutes Thank you also for my side very interesting talk and I'm very interested in the machine learning side you have to define a cost function or a loss function to define how good the system is and I would like to know what you think about defining a function exactly what I'm working on right now exactly what I'm working on right now and exactly what I'm trying to do because it's very difficult you try to reduce the problem and then you already work with enormous restriction so you could be reinforcement learning where the machine tries to create a kind of situation or rather to get closer and then it becomes very vague and very difficult we will try to find that out that's the reproducing side but how is the artistic side how do you define such questions it's a question of what is right and wrong it's a problem I think if we would find something if we could say that this painting is 85% aesthetic and I think it's a bit scary how many studies are there how many studies want to quantify things there are no absolute truths and on the context that it's a problem that we have today in our society everything is flattened everything is flattened and I think that's very dangerous that's why this work with the machine is not about giving the machine autonomy but maybe you can learn to work with the user and the machine learns the properties of the person that you use I would say I really have to apologize because you have to close this session I really have to apologize because we want to close this lecture and I'm really looking forward to the new debates that are being opened I'm looking forward to the new debates that are being opened but I'm thinking that you will put the interview in a different place and I would like to thank Liat and Marvin and E. David and Professor and thank the professional professor