 Hi everyone, you hear me? Yeah, okay. So, how many of you play video games? Great. How many of you develop video games? How many of you would like to develop video games? Better. Okay. Again, how many of you know this tool? For those of you that don't know this tool, it is the engine, the framework that has been used for the following titles. If you are a video game player, you should recognize at least one of them. Years of War, one of the masterpieces for the current Xbox. Tekken. This is the seventh installment. Street Fighter, I hope everyone knows it. The recently announced Dark Siders 3. This company started with its own engine with the previous two installment, then moved to Unreal Engine 4. This is not so famous, but it is a Kickstarter, a successful Kickstarter, made by the designer of the original Castlevania series. And even the remake of Final Fantasy VII, that is one of the most famous and beautiful games ever made, it is in work with Unreal Engine 4. And a lot of more titles. So, before starting working with Unreal Engine 4, you need to get it. For those of you that don't know it, don't know it, Unreal Engine 4 is open source. It's not free software, but it is open source. So you register on the arealengine.com site, and then you get access to the GitHub repository, where you can clone the source code of the whole engine and editor. You can eventually build it for Linux, Windows, Mac, or download a binary release. Our Epic, the company that makes Unreal Engine, makes money. Basically, Epic asks you for the 5% when you start earning more than $3,000. Only for games. If you make simulations, or films, or whether it can be considered a game, it is free for use. Once you download the engine, you get something like that. On the left side, on the left side, you have a list of the actors. An actor is something you can drag on the world. The world is a representation of your game. Each actor can have its logic, so you can program them, and when you hit play, your game starts. If those two chair a table and I don't know what it is that strange shadow on the table does not impress you, I've prepared another screenshot. I think it's way better and can show you how much powerful is Unreal Engine 4. We are programmers, so this is the most interesting question for us. By default, you can program your game logic with C++ that I suppose everyone knows, and blueprints. How many of you know what a blueprint is? OK. Everyone knows C++ right? OK. This is a blueprint. It is basically visual scripting. On the left, you see an event node. It means when the spacebar event, so when you press the spacebar on your keyboard, do something, in this case is jump. So your character will start jumping. Even if you are not a programmer, it should be easy understandable as it cause effect. This is another example of blueprint. We have another event, left control on your keyboard. And then we spawn, so we generate a new actor, in this case a missile. So we basically shot a missile from the position of our player. The get actor transform on the lower left is to extracting the transform in the gaming world and the linear algebra world is the description of position, rotation and scaling of an object in the space. This is another example of blueprint. The same one as before, but with a delay node added. With only that node we can implement some form of cooldown for your weapon. So you don't start shooting missile one after another, but you slow down them by 0.2 seconds. Then you may want to start working with vector map. It is very common in game development to start working with vectors. And then you can easily start with this kind of spaghetti. Or you can do even worse. Now, do not get me wrong. Scripting with blueprint is effectively programming. Blueprints remove only the syntax part of programming. If you respect all of the good programming rules it is very hard to make this kind of mess by following both common sense and good design role. Blueprints are effectively really cool. This is a debugging session. So in real time, while your games run you can see the nodes highlighting. You can stop and you can make a breakpoint. You can see the value of variables to change and so on. Before starting programming a real engine you have to know a bunch of concepts. Your object is the main class in the sense of C++ that is used by the engine. Basically, anything is in the editor and in your game is a subclass of your object. An actor is a subclass of your object that can be spawned, so generated on your word, so it is effectively been seen on the word. A pawn is a subclass of an actor. A pawn is a special actor because it can be possessed. By possessed it means a controller. A controller can be a keyboard, a mouse, a joypad, everything that can abstract the human player or an AI controller for artificial intelligence. A character is a subclass of pawn that has all the same characteristics of pawn but for anthropomorphic figures. So you have a human with a skeleton that can jump, run, crouch and so on. Then we have component. Component and reusable pieces of code that you can attach to all of the previous classes. If you ever really used some other engine like Unity or Godot you may be used to the tick paradigm so you basically write a single function update tick, it depends on how the editor use. And in this single function you make all of your game logic. That function is run 60 times per second, 30 times per second independent of the frame rate of your game. In a real engine 4 using the tick is considered an antipattern. You have to rely on events. On the left you can see a bunch of events, of example events that have defined, so explode, move forward, past and so on. So you should always follow the event paradigm. Why? First of all for code organization we have already seen how easy is to generate spaghetti with blueprints. So thinking in an evented way should simplify the organization of code or visual code. Events are network friendly. A real engine includes a network stack out of the box and whenever you use events that events can be propagated on clients on the server, broadcaster to all of the clients, to a bunch of clients and so on. As you have already seen you will get time management for free. Events can be triggered by animations too. For example if you have a running animation you can trigger an event when the foot of your character is on the ground. I don't know for spawning particles or for doing noise. Most important, game development is done by a lot of different figures from programmers to artists to some designer, level designer, architect and so on. So there are a lot of non programmers in a team so following the event pattern should simplify the explanation of your gaming rules to non programmers. When and why should they choose blueprint over C++? Unfortunately not all of the C++ low level API is exposed to blueprint only a little subset. On the other side writing shaders materials in the aerial engine jergo is extremely easy with the blueprint extremely hard with C++ or with the native languages of shaders. Interfacing with C or C++ library is basically a master. It is very rare that all included library in the engines will be enough for your work. Good design the blueprint are really easy to read even for non programmers. This is a personal thing. Blueprints are sometimes too much silent for me during error. Then this is the central topic. In addition to C++ and blueprint we can now program our game logic or eventually script the whole editor with Python. It is an open source project you can freely download it. It is supported on Linux, Mac and Windows. Why I did it? First of all my dream was to code AAA level game with Python instead of C++ or eventually blueprint. Unfortunately I miserably failed. Why? After the first release of the project I obviously started checking comments on Reddit, Facebook and all of the channels where I posted the announcement of the release. I didn't realize that for the game industry coding in C++ is not seen as a problem. As something that slows down your work. Obviously there are a lot of more or less myopic programmer really welcome the plugin but the vast majority of much of programmers throw shit on it without thinking twice. So instead of stopping working on it I started asking my colleagues and the other company doing games how I should change the direction of the development. So after a bunch of chats with these people I came up with this list of tasks. First of all the main project in the C++ industry is managing tons of assets. There are a lot of artists musicians that throw assets in the project and these assets must be managed by a programmer assets can change and the programmer have to change again what is mapped to assets and so on. So scripting, doing orchestration of your development pipeline it is something really wanted by the game industry especially before for artists in the 3D areas modeling, animating and so on. Python is already present in their development pipeline. A lot of animators scripts their working pipeline. So having the ability to use the same scripts, the same code base even into the game engine would be really really interesting. Unit testing as well as functional testing really diffuse practice in the gaming industry. Most of the time because writing tests in C++ you really are. If you fail on a test generally it means a segmentation fault or a crash, you have to restart your work from the beginning. So even having the ability or running unit test was an interesting thing. There is a lot you know there is a lot of Python in the scientific and academic world and data visualization is another interesting topic for them. So having a really powerful 3D engine will be really useful to show their work. Another almost strange thing is simplifying the versioning of their work. A good part of a game is composed by binary data you cannot version so well. So for a lot of areas it will be very interesting to write code that regenerate the status of your project and eventually write AAA games with the engine. So I started in August last year I didn't have high expectation, it was more of a toy project for me. First step was obviously embedding a Python virtual machine into the engine having a console for issuing Python commands and obviously running code and getting output. If you do not know I am the author of the YuWizgi project so I have a really huge experience in embedding Python in application so it was pretty easy for me to reach this level in basically a couple of hours of development. Here you see a console built over the native Unreal Engine graphical user interface I issue the import command and I get the output. I was already really really happy that the hardware starts. First challenge I already told you that the most important class into the engine is the UObject1. It is the class from where all of the other ones is. So I needed to find a mapping between a UObject, native C++ class and a Python native class. Following these rules the mapping must be fast. In this time of the development I was still trying to use this plug-in to write AAA games so performance were really really important. This mapping must take into account that both Unreal Engine and both Python update garbage collection. C++ has obviously no garbage collection but Unreal Engine implemented one. A funny thing in the Unreal Engine instructor is that properties of classes and functions of classes are built like UObject. So I needed to find a way to map this particular behavior even to Python. Obviously even on Python both classes and properties can be objects. Those are the solutions. For managing the Python UObject mapping C++ plus 11 map if you do not know C++ only added in 2011 standard the ability to add dictionaries or ashes. When the related Python object goes out of scope the garbage collector on Python will very probably destroy it I prevent the garbage collector to destroy the UObject mapped to it because Unreal Engine make a different decision by the Python virtual machine. So I must ensure that UObject are not destroyed without Unreal Engine knowing it because it will mean a crash of the whole editor. Whenever from Python I access the Python UObject mapping I need to always check that the mapping UObject is still valid because if it is still valid it means Unreal Engine has kicked out that UObject so the Python mappings maps to something that is no more valid. And then automatically expose properties and function as Python equivalent. After a couple of days I came up with this. So I can directly call Python script for Unreal Engine added event. Next step there are I think hundreds of C++ classes into the editor. I think there are thousands but let's say hundreds of classes into the Unreal Engine editor, an engine. Mapping each of them to a Python class will be a huge work that I don't want to do. But Unreal Engine exposes a reflection system so I can basically get a reference to a class using a string. My objective was to do something like that. So import the character class like it is a native Python class and spawn on the last line a character so an instance of a character into the editor. So let's take care of the slide. This is how you write extension in C for Python. The interesting part is line 150 that find object camel case function. This is how we access the reflection system of Unreal Engine. Basically we find object we find a reference to a class with that name. In 155 we build the mapping within that object and its Python equivalent. So now I have a way to get a reference to Unreal Engine a native class using a string in Python. What I do with this, I don't know if you can read it. This is basically a hack for faking the import system of Python. Basically, whenever you import check the last line from fakingPorter.classes the getAtR method of figure class is called. So I use this technique to fake the import system. So whenever you import from Unreal Engine.classes a special getAtR method is triggered returning the equivalent classes. Incredibly I get unexpected results. Positive unexpected results. The same approach work even from structure. In Unreal Engine there are objects and there are structures. Structures are passed by value in the vast majority of languages and even for in-hounds. So with the same approach I can import the three main structures of Unreal Engine. And without knowing it I would have already reached that kind of coverage. 80% of the Unreal Engine for reflection system was ready. So with my pattern plugin I was already able to do all of the things we can already do with blueprints. So it was a huge success. Third challenge. Scripting the development pipeline includes even generating blueprint programmatically. I know it could be something strange. It's like generating a language with another language. But this is what people or game developers ask at me, especially animator. In Unreal Engine animators generally add logic to their animation using blueprints. So for them it's way more easy to use Python to generate those nodes. The nodes are not really pretty for a Pythonista but this is how you can build a blueprint graph with Python. What I needed to do to reach the point I needed to implement Python classes, abstraction classes for graphs. The gray background is there. Nodes. The big blocks. And pins. The arrows and the thing that generates spaghetti every time. The current state of the art can be seen in this tutorial. Basically with only Python you can generate a whole monster with its animation, its brain. I don't remember. You can do really a lot of things in this. It is a pretty huge tutorial. As before I get unexpected results. This time not very positive. I get the ability to script animation graphs for free. Beaver tree are blessed in a real engine to write artificial intelligence. They are like state machine on steroids. Unfortunately I started getting crashes all over the place. What happened? I was obviously working in a way not expected by epic developers. So I started messing with their code. I started playing at the same level as C++. So every error results in a brutal crash of the engine. I was playing as a virtual editor. So basically everything I do from Python should be doable even into the editor. If it is not possible, expect only pain from a real engine. And especially the reflection system does not like messing up with their internal structures. After about a year this system has become really solid. But it requires one year of full time development. First challenge, unit test. Here you can see a unit test checking that when I create a new material a new shader for your graphics card. And I give a name to it that name is effectively put into the engine. It is a pretty easy test case. First of all, when we run tests from the command line with PyTest, however you generally do from PySharm and so on. A new Python interpreter a new full Python interpreter is generated on your system. So when the test send data generated by them is erased and destroyed. We cannot generate a whole new Python interpreter into a real engine. It will mean generating a whole new real engine editor for running a single test. So my solution was generating a Python sub interpreter. The Python CAVI allows you to generate a new copy of a Python virtual machine that can be destroyed without destroying the main virtual machine. So when you run tests written in Python a new Python virtual machine is spawned without destroying the original one and it runs the test. Standard output and standard error are mapped to the real engine Python console the window we seen before. And at each run of tests we do rollback. This is one of the feature of the real engine that shocked me most. It is not documented so well it is really hidden. I discovered it for a case I was checking sources and I found it. Basically each operation in the real engine editor is a transactional one like in a database system. So I can build my test and rollback everything that I do in the editor. So I am free to generate actors into the world, check that they are good and then destroy them. Obviously the gameplay is still something I really want to do in Python. So I've started having these abstraction. They are subclasses of actor-pwn character Python component and PyHard. We will see later what is it. They are wrapper for their native counterpart but the ability to call Python code at specific events. After a bunch of tests and crashes all over the place I decided to use a proxy pattern for mapping Python classes to native real engine object. So basically you write a native Python class a special attribute is injected in this class so you get access to the native Python object. This is an example of an actor fully written in Python. The begin play method is automatically called when the game starts and the tick method, albeit it is an anti-pattern, is called a very refresh of the game. The interesting part here is that self view object you can see here right to the location. The self view object attribute that is injected by a real engine when you map a native class to an already existing object. Why I chose this approach? First of all for almost safe reloading at each play iteration once you load a module or a class into the Python virtual machine that module is resonant that code is resonant but during development you want to change code. When you do web development generally you restart the application server to see code modification we can't restart the whole real engine editor because it requires a bunch of seconds to spawn. So in this way I can call a simple imp.reload from Python to reload the code of a proxy class. In this way I have a clean separation between a real engine 4 and Python especially from the garbage collection point of view. As the class is not really tied in I can destroy one of them without bothering too much the other one. This is something I found really interesting but others don't. Once you have packaged your game that is executable with all of your Python scripts into the project directory you are free to change that Python files and basically modify all of the game logic. Logic I understand is not an interesting thing to do for a game developer or something that it doesn't want to do from its customer. Obviously you are free to include not the .py files but the compiled one Another advantage of using proxy pattern is that I can change during gameplay the class mapped to an object. A character could be mapped at some time of the game to a class and then remap to another one during gameplay. This is another challenge I'm still working on and I'm still failing at it for those people that don't like proxy pattern. Basically I want to allow the developer to subclass native Unreal Engine classes with Python classes something a bit strange and immoral for some people. As you can see as Unreal Engine uses C++ and it secretize Python 3 annotation for specifying to a Unreal Engine what type of value this function take. I'm talking about on C++ Those are the steps I did for reaching this broken result. I use the Python annotation for enforcing types. Python became something like a data description language so I use Python to inform Unreal Engine on how to build its internal classes. I needed to rely on metaprogramming a lot and a lot of pain for reaching the result. Why Python metaprogramming? Because character the class from each monster is an object, it is not a class. So I need to write logic to inherit from an object that is not a class and generate a new class. This is how you can fake the inheritance system. You can see the funny sub class is a child of fake class that is an instance of fake base class. It could blow your mind and understand. Unfortunately as I already said it is really buggy. It could crash your system. Incredibly as different behaviors on different engine version I really have no idea why it happens. It is absolutely a good exercise to know how a real engine internals work. I think it is not so worthy because the proxy paradigm is way more safer and solid but it is obviously a lot. So I think I will try to improve it. What I still need to do to improve the project? I would like to stabilize this sub class in EBI. I would like to introduce mobile integration because currently you can build games Audi for Linux, Windows and Mac. I would like a stronger integration with PDB for simplified debugging of Python script into the engine. This is a really huge task covered in C++ API. I still have a bunch of doubts. First of all should I follow in the C part integration of Python should I follow the Python C coding style so underscore lower case or the C++ 11 with full camel case. In the code you have seen before the implementation of reflection you have seen two different styles in the code. It is something really really ugly but currently I have still not made a decision. Should I invest more on threads? Threads are supported out of the box. The gil has become not a huge problem. I added a bunch of tricks to reduce its impacts. By default if you download from the sites have no threads enabled there are builds with threads enabled. When I started the project a lot of Python developers are really used to Maya or Studio Max want to use QTE in their project so I invested a lot in easy integration between Python and PySide into the engine. Nowadays I have exposed the native graphical user interface of a real engine so it is called Slate so it is a good solution instead of QTE. Slate is based on preprocessor C macros so you can write some kind of readable interface in Python trying to give to the developer the same taste of the C++1. This is how you can generate a window in the real engine with a button on its center when the button is clicked the cell yellow function is called into the button there is a text block with a simple string. Currently I'm dreaming about more orchestration in the game industry there is still a lot of manual work I would like to integrate Gherkin for a behavior test another dream is build some kind of domain specific language for writing shaders so material and or blueprints and obviously someone using it for to-go-lay gameplay. These two companies had a lightning from Los Angeles and good thing it's from Toronto, Canada they heavily sponsored the development of the project there are at least dozens of company already heavily using it but these two are the ones that put money into the effort. If you are into the scientific academic world and would like to start drawing your academic research results into a powerful graphical engine this is a really gentle introduction for pedonista that includes matplotlib into your game so basically whenever the player work over one of that colored cube the graphs on the carpet it is updated to generating a new pie charts. I need to switch laptop because unfortunately a real engine is able to melt down my MacBook Pro so can you see it? This is the third person template you can generate from the standard real engine distribution. I have added on the level a bunch of phases, of famous phases so when I play the game I can move into the world and see spark gillen Anderson Wolverine at 800 Sheldon my objective now is to allow the player the mannequin to see and recognize phases using facial recognition included in opencb open computer vision all in python first step is adding a python actor into the scene that will implement the the site of our player so in the list of actors I search for pie actor and I drag into the scene on the right you can see the fields where you can specify the module and the class to map to this actor we will start with this simple code into the init method we better you initialize a timer of one seconds in the gameplay you simply print some message into the console and then this value of timer is decreased at the retic and when it reaches the zero it means the time has elapsed and restart counting it is a very simple code just to check the python actor is so the module is eyes first and the class is site this is our python console hate play and on the bottom you should see on red the elapsing time so our python actor is already working let's go on second step is adding a special component to our actor the scene capture component 2D basically you can see it as a virtual camera that track anything it sees and save it into a texture a texture is a memory area into your graphics card that can be drawn on the screen during the initialization we generate a special texture known as a render target it is where our capture component will write data I've called it what I'm seeing I create a texture 512 for 512 you should always try to use the power of true texture when doing game development it is a transient object it means it is not an asset on your pipeline when the game ends this object is destroyed this special look is called automatically defined before initializing components in your actor so we want to add a new component so we use this look to create it this is how we add a component in our code the scene capture one we specify in which texture he has to write data and we specify that we want the linear values of the colors it is something I will not want I am just playing you it is not the topic interesting for us so I go there and change the model to eyes second I hit play and here you can see pay actor with a new component scene capture on the right it is already tracking what is seeing unfortunately we cannot see what the player is seeing before doing this we have to fix another problem this pay actor has no position into the world our objective is to attach it over the head of the player here all is the same once the game starts we get a reference to the pawn used by the player in our case the mannequin the robot working on the screen and then we attack ourselves this is the injected field into the proxy class and we attach it to the mesh component of the mannequin into its head bone what is a bone here you can see our player and on the left you see what happened to its mesh bones are basically objects mapped to a group of vertices so whenever I move a bones vertices follow it so our objective is to attach our eyes to the head so if we go back you see the third person character that is our mannequin is pi actor 7 as its child here you can see the axis of the character and here you can see the axis of our python actor as you can see it is moving because the head of the mannequin by Wally Nidol to some kind of breath next step we want to develop a head up display some kind of 2D graphics onto the screen we have added this code so we spawn a new actor into the scene head is another actor directly specifying the python module in the python class to use we get a reference to the proxy class mapped to this head so we can start injecting values into it and we specify into the proxy class which is the texture to draw and finally the engine that is new python head should be the player one so the default created one is destroyed so our python one will be used this is our head HUD proxy class expects you to define a drawHUD method where you can call graphics primitives like drawing textures, lines, rectangles and so on until texture to draw I remember you it is defined in the player class it is not defined it exit from the method if it is defined it will start drawing it at the 00 position so top left with these sides so ok on the top left you can see what the player seeing you can see even the balancing when it runs texture containing all of the pixel the player is seeing next step for good results OpenCB needs a grayscale image so we need to generate a grayscale variant of this texture this is pretty easy this is the fifth one this time we import so it is the wrapper for OpenCB and NumPy because OpenCB uses NumPy for all of the data in the tick method of our eyes we get the data as a byte array of the texture of the data we are seeing we inform we generate a new NumPy array giving special dimensions so we have a three dimensional array with four components RGBA and the sides of the texture and we use this OpenCB function to convert this texture from VGRA to grayscale so basically we have a byte for a single pixel of the image on the other side the head class is changed it is now in the head second module we create another texture with a single channel for color another transient texture this time we need to cooperate with a real engine garbage collector because this texture is never used by a real engine it even does not know it exists so after a bunch of time it will destroy it we do not want this so we inform with this method that we are responsible for destroying it when the Python object is destroyed and in addition to drawing what the player is seeing in full color we draw another texture below it without ignoring transparency channel because we have a single byte for each color so I needed to add these additional values so we have the same view but in grayscale now OpenCB is able to recognize cases from the below image this is the last version of the code these remain the same all of the OpenCB logic has been done in the HUD so we set up a cascade classifier it is the way OpenCB uses it is the algorithm OpenCB uses to detect something in this case I'll feed it with information about frontal phases and then at each iteration I try to detect phases with this function detect multiscale and it will return a quadridimensional array with the information about where the phase is you should see green see it? let's see if we can recognize Spock here it is it is even recognizing Dana I push Spock away people with beards are really hard to recognize but this time is taking it good we have a... I don't know if it is a good thing but it is not able to recognize a terminator I don't know if it's good or bad it is interesting that it is not able to recognize Sarah Connor I think it's because of the goggles of the battle goggles that are really strange to recognize for a machine last part by default Unreal Engine has no features for using the webcam of your mobile phone of your laptop but OpenCB is able to do it so I've created this cube and I've added it a Python component mapped with the Utils webcam class that initialize a video capture create a transient texture with the size of the texture my webcam is able to to manage and I assign the texture to the material to the shader assigned to the cube at every frame I try to capture the image from the webcam I convert it to a suitable format for my graphics card and I assign the data to my texture I think you should see me Thank you Who has a question for Roberto? I have been working for Crytek for 5 years and I know what game development is and I know what pain it is to code game logic in C++ or in Lua script and how ugly look unit test Come on, Lua script is not so painful as C++ Yes, yes but anyway so I think integrating game logic in Python is a really great idea and it's absolutely brilliant so it's Bravo Yes and I wanted to ask you did you have any performance issues especially in tick methods calling it in update loop in every frame Performance issues? Yes, yes I've started having performance issues and I started doing it too much with the gil acquiring and releasing the gil is one of the most heavy part into the Python virtual machine so the default build you get from the site of the gil completely disabled so you cannot use Python threads and the impact of Python is not so bad Last year Blueprints was an interpreted language faster than blueprints now blueprints are compiled to native C++ code so unfortunately faster than Python Another question Just a quick question How do people interact with Python inside Unreal Is there only the repel or is there also an ability to maybe extend the ribbon bar there on top like putting little scripts into I cannot feel you Sorry, I try again I was wondering how people would interact with Python inside Unreal So I saw the repel obviously Is there also a thing planned to put code snippets into buttons on the top of the ribbon for users that are maybe not so much also deep into coding Just wondering No, no Yeah So I saw the repel so people would interact with Unreal in Python via the repel and maybe there are some recurring tasks people have while working with Unreal I was wondering if there was also planned maybe to have some of these buttons that I see on the top to incorporate Python snippets in there Yeah Yeah, I imagined Cool, thanks Another question? I don't need to run That's fine, thank you Roberto