 Le prochain speaker est Aurélien Gérond, un expert AI à KiwiSoft, qui s'est formé, il est aussi autorisé par Best Telling Books à Aurélien, mais aussi TensorFlow et une team de vidéo classification sur Google. OK, juste un moment, qui ici utilise TensorFlow professionnellement, ou qui n'a pas utilisé ça ? OK, pas tout le monde. Je vais essayer d'aller plus loin. Merci pour l'introduction et pour l'inventation. Je m'appelle Aurélien Gérond. J'ai déjà mentionné que j'ai travaillé comme le leader de YouTube de la team de vidéo classification en Paris, ce qui signifie que mon travail était de tagger chaque vidéo avec ce qu'il s'agit de, ce qui est généralement des cats sur YouTube. Et puis j'ai écrit ce livre « Hands-On Machine Learning » avec « Second Learning » et « TensorFlow ». Et la deuxième édition est venu. En tout cas, aujourd'hui, le parler est de TensorFlow 2, ce que vous avez peut-être entendu n'a pas été réellisé, mais l'alpha version a été réellisée. Et je vais expliquer ce qu'il s'agit de nouveau et comment vous migratez de TensorFlow 1 à TensorFlow 2. OK, c'était juste annoncé, je pense, dix jours plus tard, à la Sommet de TensorFlow Dev et ils annoncent l'alpha version avec beaucoup d'improvements. Et le candidat final de la rédition s'est décédé pour le Q2 2019, c'est-à-dire que ce sera probablement la fin de juin, peut-être. Mais on verra. Et pourquoi ils ont créé TensorFlow 2 ? C'est un changement majeur, donc ça va impacter beaucoup de gens. Et les changements majeurs vont vous permettre de faire beaucoup de clean-up et d'improvements, même s'ils ne l'entraînent pas. Et la motivation initialement est cette. C'est le nombre de citations dans les papiers de machine-learning. C'est en 2018, donc les graffes ont changé, je dois les updater. Mais vous pouvez voir que TensorFlow est, par far, à l'époque, le plus cité de la bibliothèque dans les papiers scientifiques. Café, piano, torch, nous sommes tous stagnés ou en passant. Et puis vous avez TensorFlow & Keras en tirant. Mais vous avez aussi ce PyTorch en passant. Et beaucoup des users de PyTorch que vous avez entendu migratez de TensorFlow 1 à PyTorch. Tout le monde dit, c'est beaucoup plus simple, c'est vraiment beau et c'est vrai. Donc, TensorFlow a basically had to react et le premier improvement est sur la simplicité, pour faire ça beaucoup plus facile. Donc, on va regarder un petit peu de PyTorch code. Si vous voulez compter comme 1 plus 1,5 plus 1,25 et ceci, c'est ce que ça ressemble. C'est très, très naturel. C'est un code Python. Vous pouvez juste créer votre TensorFlow X, Y equals 1. Et puis la situation, X equals X plus Y et Y equals Y divided by 2. Et vous pouvez le faire 50 fois et vous convertir à 2. C'est très naturel. Maintenant, let's look at the equivalent code in TensorFlow 1. It would look like this. You would have the first phase, which is the construction phase, where you actually build a computation graph. No computation is actually executed here. It's just building a computation graph. And you create all the operations that you'll need to manipulate plus this thing that says, you know, I will initialize all the variables. And this is where the actual computation start. You need to create a session, initialize all the variables and then run your iteration where you explicitly say I want to execute this part of the graph and this part of the graph. So if you think of computation graphs or TensorFlow graphs as kind of a language, basically here you're writing a program in that language and here you're executing that program. So it's kind of metaprogramming and it has a lot of advantages. I'll get to it. But clearly, you know, for most simple work, you don't want to be coding like this. It's a little bit too complicated and verbose. The other thing is that it can hide subtle bugs, like for example, if I decide to execute add up and divide up in one shot. Well, TensorFlow is not or TensorFlow 1 is not going to see any dependencies between these two operations. As far as its concerns, those are completely separate and independent. So it's happy to launch them in parallel and therefore the order of execution is not guaranteed. And as you can see, the result is no longer two. En fact, if you execute this code multiple times, you'll get different results every time. And that's kind of hard to catch. It's not super frequent, but it does happen or it did happen in TensorFlow 1. It's fixed in 2, as I'll explain. And the other thing is it's a little bit hard to debug. Say you have x and y is a function of x and z is a function of y. Well, if you execute z and you get a value that's actually not a number, well, you're kind of stuck. Why do I get a NAND value here? If you could debug, you might end up saying, oh, wait, I'm dividing by zero over there. That's the reason. But as far as Python is concerned, this is just a function call that directly gets executed and in C++ it doesn't see the details. So in the stack trace, all you get is, boom, there's an exception on this line. I don't know any more of that. In TensorFlow 1, there actually is a graph debugger, but not many people use it and it's a little bit tricky to use. The other thing that's harder is to profile for the same reason. Like if I try to execute, time it on running the z here, I'll get some number that's how fast it runs, but I won't actually have the detail in terms of this operation, that operation, that operation, which one runs faster, it's not clear, for exactly the same reason. And again, there are tools that were developed that allow you to profile graphs, but again, not many people use them and it's not very natural. So the solution came in TensorFlow 1.4. So it's not TensorFlow 2, this actually, the solution came way earlier. It's called eager execution. When you start your TensorFlow program, you just activate it and magically for now, it's not in graph mode anymore. So any operation you execute actually gets run right away and you get the answer right away. So as you can see, it's almost the same code as in PyTorch. There's this extra line, but this line actually disappears because in TensorFlow 2, eager mode is the default. So when you look at a TensorFlow 2 program or PyTorch program, they can look very, very similar. This last thing here is because these are tensors, if you wanna see the value without having a long line here, you can just call numpy, it gives you the actual value. All right, so now that we've seen that basically eager execution is much, much simpler to use, you might wonder why did we use graph mode in the first place? Why would you wanna use it? And the first reason I've hinted to earlier, if you do b equals a plus three and c equals a times five in graph mode, this will lead to a graph that looks somewhat like this, actually exactly like this. And it's easy to see when you look at this graph that this addition operation, this multiplication operation are completely independent, they can be run in parallel. That's the first thing you gain from having a computation graph is that TensorFlow can actually run automatically things in parallel without you having to manipulate any threads or anything. It parallelizes everything for you. So if you have multiple cores, different operations will run actually in different cores. The second benefit is that if you actually express everything in a graph, that graph can actually be run entirely on an accelerator like a GPU or a TPU. And without all the back and forth with Python, with the CPU, I mean. And this can speed things up a lot. If you're manipulating operations of huge matrices, then it might not make a big difference because the overhead of going back to Python might not be that huge. But if you have lots of lots and lots of smaller operations that need to run really fast, this can make a big difference. Another benefit of graphs is that since you have this kind of representation of all your computations, TensorFlow can actually analyze the graph and possibly optimize it. So of course, like getting rid of nodes that you don't use or in the case of XLA, it can do things like find pairs of operations that for which an actual optimized implementation exists and it will swap out the pair of operations and replace it with the optimized version. So it can speed things up that way. So that's a benefit of having something symbolic that you can play with. And another benefit, and I guess to me this is like one of the biggest advantages of TensorFlow over the competition. It's the portability. You take a TensorFlow graph and you can just run it on a mobile device. You can run it in a web browser. I invite you, if you haven't seen it already, to go to tensorflow.org slash js slash demos. You'll see TensorFlow js directly running the browser. Here I played this little game and neural net was actually trained in the browser, but you don't have to train in the browser. You can train it in Python, export the graph and then run it in your browser. So portability is also un key element of having graphs. And then of course you can run the same graph on your servers and go to production and this will be the same graph running in all these environments. Okay, so the good news is in TensorFlow 2, you can actually have the best of both worlds. By default, everything is eager, meaning every time I execute a TensorFlow operation, I get the result right away. So it's easy to debug. It's easy to profile. It's natural to code. But if I want to speed up this little function here, I can do that very simply. Just add this decorator at tf.function and that's about it. Everything will happen under the hood. This function will now be a graph function, if you will. So the way it works is the first time you actually call this function with actual values like two and three, well the function, the Python function will be traced. And by tracing, I mean it's actually going to be executed but not with the values two and three. Instead, it's going to be called with so-called symbolic tensors which if you know TensorFlow 1, you can think of as placeholders. Okay, so it's basically calling this function that you wrote with this thing here that says I'm a tensor, I'm a scalar, there's no shape, like shape is empty and it's a float 32. And then the operations don't actually compute anything, they build a graph, right? So this will be run in graph mode and once the function is traced, well you have this nice little graph in memory that's cached in this TensorFlow function. So the next time that you call this same function with different, you know, scalar values, different floats, it'll reuse the same graph. Of course it won't create it every time. And so you can get, you know, there a lot of speed up with this. So the graph might look something like this. With X and Y being placeholders, if you can actually still go and look at the graph that was generated then those would be actual placeholders in the graph, okay? But you don't need to handle them yourself. You don't need to feed anything. Basically, you don't need to manipulate graphs, sessions, placeholders. All of these are handled automatically now. So, right. You get the power of graphs with the ease of programming and all the benefits of e-guerre execution. Okay, so you can mix and match eager in graph mode. So for example here, I'm coming back to the one plus one half plus one quarter thing. And you can see that this is the whole code and I've decided for some reason that I just want to speed up this little piece here. And this is the graph function or the TensorFlow function as they're called. But I have this external loop and it just calls this on and on and on and I get the right result. By the way, the problem that I mentioned earlier that the addition and division operation might run in parallel and you don't know which one will run first that has been fixed in TensorFlow 2. Anything that affects the state or a stateful object such as available will actually automatically be run in the right order by the graph. Okay, so TensorFlow ensures that this will be run first and this will be run second. It won't try to paralyze them. This is fixed. Okay, now also notice that I didn't need to create like a global variables initializer. I just create a variable and it's completely natural. It's like object oriented. This is where the state is. It's nowhere else. There's no like default graph or default session or anything. Everything is linked to a Python object and so it's pretty natural. Don't need to handle the controlled dependencies. I mentioned that already. Now look at this loop here. I have this for loop that's outside of the TensorFlow function. So it's just running on and on and on and calling my TensorFlow function but I actually put it inside another TensorFlow function if I wanted to. You can do that. And this loop here will actually run in the graph. It's not gonna run during tracing time at tracing time because of some magic called autograph. What TensorFlow does is it captures the source code of this function here, analyzes it and notices, hey, I'm actually calling tf.range and not range. And since I'm calling tf.range here, this means that this should actually be ported to the graph itself. So during tracing, it'll just skip this and this loop will be executed when you execute the graph itself. So it makes it very easy to write like dynamic models like RNNs and things like that using this kind of code. I could actually remove the tf function decoration up there because here I'm calling this function run loop. It's a tf function and it depends on run step. So it's actually recursive. When this function gets called and it tries to turn this into a graph, it's gonna see, wait, it depends on this function and it's gonna turn this into a graph and so on. So it's recursive. You don't need to sprinkle tf function everywhere. You can basically put it at your entry points like your main functions. Everything good so far, right? Yes? Right, good question. So if you have, like if you actually wanna port your code and install it in like ported to a web browser and so on, it needs to be exported to a format called save model. And of course, this only supports TensorFlow operations, right? It doesn't support like arbitrary Python code. If it did, I mean, you need to have like a Python environment in the mobile device or so on. So you need to basically, in order for it to be portable, you need to export it to a saved model and this only accepts TensorFlow functions. So you would have to basically convert everything to tf functions in order to be able to export it. But usually what, here I'm showing like the low level API. Usually what you'll use is tf.carus. Carus is, you all know carus, but I think people think of carus as a library because it started out that way, but now it's more like an API and there are several implementations of the carus API, one of which is the original and reference implementation, also called carus. I call it carus team, so there's no confusion. And this implementation is actually less used than the carus implementation inside TensorFlow, which is called tf.carus. And usually you would use tf.carus for most of your work, like it's so flexible now that you don't really very often need to go to that level unless you're really customizing things. And at the carus level, it compiles everything for you, so you need to make sure, if you just use standard layers, everything will be okay. If you write your own custom layers, they need to use only TensorFlow functions if you wanna be for it to be portable. Does that answer your question? The yeager mode is basically in a sequential fashion, right? Exactly, yeah. So when we are moving to graph mode, just changing the add to tf function, will it still be good or we have to do one more level of debugging to make sure all my executions are happening in order and all those things? Right, right. So usually, I run a bunch of tests, right? And usually you always get the same result. But in this particular example, for example, since I'm running a loop with tf.range, this will actually complain in yeager mode. So what you would do is, when you're testing or debugging or you're programming, you would have this as range, right? And then once you're happy and it passes all your tests and you'd add tf.range and that's where it would be ported to the graph, right? So I can't say that it's 100% automatic and you're guaranteed to give you the same result, but usually you'd be programming and completely in yeager mode and then at the end trying to transition to tf functions and normally a lot of the constructs are just taking into account just add tf and you're good. Like if you have print statements, replace them with tf.print and it'll work as well. If you have asserts, you replace them with tf.asserts, okay? So yeah, so this will actually put the loop itself in the graph. If you have range of n, then it would actually run when tracing the function and so you'd end up with a graph that actually calls the function 50 times, for example, right? So make sure you call tf.range if you actually want the loop to be in the graph. All right, so the control flow that Autograph automatically handles is things like if statements. This also ends up in the graph, all right? So it's captured and ends up in the graph. Return statements are handled gracefully in the graph as well. You have loops with for loops. You also have break and while and tf.print and tf.asserts. Autograph captures a lot of the very, the most common constructs for control flow. And if you just use tf operations plus these simple control flows, if you just call functions that you wrote yourself, you're good to go. If you call like an external library or even the Python standard library, then it probably won't work because it's not calling TensorFlow operations, right? So that's it, yeah. So the simplicity of eager mode and graph mode is, that makes a huge difference with TensorFlow 2. If you run away from TensorFlow 1 because of the complexity, try it again with TensorFlow 2 and I'm sure you'll love it as I do. The second improvement is that it's much more pythonic. Like one of the problems with TensorFlow 1, it was that a lot of things were kind of name based using global scopes and things like that and this has been improved tremendously. So one example is sharing weights across layers, typically in assymes networks, you'd have some architecture like this where these two layers are sharing weights. And in TensorFlow 1, you'd implement this somehow like this where you'd have these two layers here and if you look closely, they have the same name and that's how you would share weights between layers. It was name based. That's kind of brittle. It's easy to break this. If you have multiple libraries creating different pieces of the graph and by chance, they actually happen to use the same name, you're out of luck. If you could use namescope, that was a whole mess in itself. Notice the second one says reuse equals true. That's how you would share weights in TensorFlow 1. On TensorFlow 2, it's, yeah, so you'd use variable scopes and get variable and a lot of constructs actually came from the fact that it was this logic of being based on names. So instead in TensorFlow 2, what you do is you actually create keras layers like this. If you use keras before, this is very natural keras code and except we import it from TensorFlow because we're using tf.keras. This is not the pip install keras thing. This is the keras that comes with TensorFlow. You don't need to install anything else. Well, I create one layer here and then I call it twice with two different inputs and that's how I share weights. It's very object oriented and natural, okay? Everybody get this? So that's one thing. Another thing is where in TensorFlow 1, there was some global state that was used. Like you create an optimizer and then you say I wanna minimize this loss. Well, you wanna minimize the loss by tweaking some variables but I'm not telling you which variables to tweak here. So how does it know? Well, it knew because actually internally it would call tf.trainable variables. It's as if you had called it like this and tf.trainable variables would actually look up the default graph and in there would look up a collection of variables and so it assumes it's think people had inserted stuff in there. It's like using global scope. It's not very clean. Instead, it's like global to the graph. It's the same kind of problem in the way this was thought out using global scope and names. So, yeah, it assumes one model per graph. It's only available actually in graph mode. So you don't, yeah, collections basically in TensorFlow 2 are dead. They're removed, variable scopes don't exist anymore. All this mess with names and so on. That's global scope that's gone. So instead, what you do is you create a keras layer like this, model like this and each layer actually handles its own weights, right? There's no other object where we're sending the variables to. No, it's like this component which owns the variables itself. And since this model contains several layers, if you ask the model itself what are the weights of the model, it recursively ask each layer and it gives you the whole list of weights for the whole model. And it's kind of clean, right? It's just in one place and you don't need this global thing. And so now when you call the optimizer's minimize method, well, you can pass very easily all the variables that you want to minimize. So the changes are variable scope is removed. Get variable is removed. Minimize now requires you to pass in some variables but it's actually easy if you've wrapped them in keras layers. Yeah, and that leads me to the next point. It's everything has been cleaned up a lot. Another complaint that people had about TensorFlow 1 is that it is starting to be very cluttered with a lot of stuff. Duplicated APIs and a lot of unused things in tf.contrib. So like one example of duplication is tf.layers and tf.kerras.layers. Well, that's gone. Now everything is high level and it's in the official high level API for TensorFlow 2 is Keras. So everything's in there and there's no duplication here. tf.contrib, this is like a snapshot of what tf.contrib contains in TensorFlow 1. And in there you have a lot of things that haven't just never been used. Like if you search on GitHub you won't actually find anybody using them or almost nothing. Others are just not maintained anymore. Some have actually already been ported to core API, but they're still there. So it's becoming a big mess and a maintenance nightmare. So they just decided to completely remove it. There will be no tf.contrib anymore where anybody can just push some code in that just won't happen. And the important parts of those have already been moved to TensorFlow 2 core API. And the other ones that were less important or less used, some of them have been moved to the add-ons. We have one of GDDs here, Jason, who handles that or one of the SIGs that's around this. And a special interest group is what I just said, SIG. And yeah, and so this get removed and some of them are just not used and so they disappeared. All right. The API has been cleaned up as well. Like there were lots of, a lot and a lot of things just at the root namescope. And so now a lot of packages have been created. So instead of, you know, tf.asserts, well tf.asserts actually is the wrong example but a lot of things have been moved into sub packages so it cleans everything up. And there are some aliases for the most commonly used functions like tf.exp for exponential. The actual function is tf.math.exp. It's so used that they kept the aliases in tf.exp. But overall it's really very cleaned up. And a migration tool has been provided. That's where, you know, you're pretty happy at this point. You're like, oh, this is too much change. Well, this is a migration tool. And so my last point will be about migration. I'm almost out of time. My recommendation would be if you have a TensorFlow 1 code base that exists today, try to migrate it first to Keras as much as you can. A lot of the functionalities that are available in TensorFlow 2 with Keras just work the same way in TensorFlow 1. So if you can migrate as much as you can to Keras, the transition will be much easier. The second thing is anywhere you're using tf.contrib, I'll try to migrate to something else because tf.contrib is just out of the picture in TensorFlow 2, all right? So no migration possible if you're using tf.contrib. And lastly, this is how you use the upgrade tool. You just give the directory of your source code and the directory of the output and it just converts it. So mostly it will actually just rename things and put them in tf.compat.v1. They have this compatibility module where basically all of TensorFlow 1 is still supported except for tf.contrib inside this package. Of course, it doesn't give you nice and canonical code, but it allows you to have production code still work in TensorFlow 2 using this. Now, some things like tf.losses.logloss actually has a perfectly good function in tf.keras, but it's not migrated to the Keras version because it doesn't exactly do the same thing. Like there's subtle differences, so they're being very conservative just pointing you to the compact v1 function. So in some cases, you might have to check that you're not impacted by those little differences and then migrate to the appropriate function. And random uniform now is underscore uniform is at random.uniform. Like it does some of these things very nicely for you. It also takes care of all of the function arguments if the order has changed or the names have changed. So yeah, it does this. There's a nice tool you can try which is called tf2up.ml, which you just give it like a Jupyter notebook or online. If you have a GitHub repo, like I do with some notebooks, you just change the URL instead of github.com. You replace that with tf2up.ml. The rest of the URL is the same. It'll automatically convert it. You can download the upgraded file and you'll see all the diff and everything. And lastly, and I'm just out of time now. Lastly, TensorFlow was already open source, but I feel that the mentality of the TensorFlow team has opened up with TensorFlow 2. And for example, there's this community project where anyone can submit RFCs, request for comments and proposals for what TensorFlow 2 should be. This has run since last August and worked pretty well. There were lots of proposals like these and people contributed this way. So I think in the spirit of this conference, it was already open source, but I think it's even more open now. You can go to these discussion groups or developer groups if you want more details. So yeah, I think to conclude, it's cleaner, more object-oriented, Pythonic. You get the ease of use of eager execution, the power of graph mode, which gives you speed and portability. You have super rich APIs to handle. I didn't talk much about Keras, but it's a beautiful API that you can use a data API for efficient loading and everything. This was already in TensorFlow 1, so that's why I didn't cover it now. The great API, a huge community, like in most use framework out there and machine learning. Documentation has improved a lot and many projects have been built on top of it. You also get like TPUs on the cloud, which right now is, this is the only library that can actually benefit from TPUs. PyTorch is actually coming soon on this. And it's even more open. So if you want to play with this, I have some Jupyter Notebooks that are open source on my GitHub account slash Ageron. There's a whole TF2 course with exercises if you want to go to this. And I'm writing the second edition of my book. This was the Notebooks for the first edition. And the second edition is here, so I'm like at chapter 13 or so. And it goes into all the details. You can try that out. And with that, I think that's it. Thank you very much. If you have any questions, I'm happy to take them. All right, let me repeat that again. So were there any changes with respect to the input of data and the output of data, specifically with respect to turning that into a online prediction, the online prediction component? Yes, that's an actual question. So one of the difficulty people had with TensorFlow 1 is, okay, sure, I can train a model locally, but how do I deploy it to production and in different environments? How do I train it on multiple machines and so on? And how do I prepare the data and everything? So this is one of the announcements that came with TensorFlow 2 Dev Summit, was there's this platform called TFX, TensorFlow Extended, which includes several projects. They're not included in TensorFlow, but you can download them on the side. And they all help to productionize TensorFlow models. So you have, for example, TensorFlow Transform, or TF Transform, which allows you to write pre-processing code once. And you can run this code, it's written in Python, you can run it in batch mode over all your training data, and it'll just convert it using Apache Beam into your pre-process data set, and then you can train your model on that. And then it also generates a TensorFlow graph that you can put in your model once it's trained. You just plug it in and put that in production, and so it will be able to also pre-process, using your same function, pre-process incoming data on the fly as well. So it includes, TFX includes also TF Serving, which is like a very powerful server. That's what they use internally at Google to serve multiple models, it handles multiple versions. If you switch from one version to the other, it works fine. It actually supports having multiple models at the same time. If you're doing experiments, for example. So fantastic platform. There's also things like model validation, another module TensorFlow validation, and I'm missing a few, but to check out TFX, and this is like the way to productionize TensorFlow models, and they just released, they had already released some components last year, but now you have the whole framework. I was still related on the deployment model. Are there plans, or is there an initiative already for TensorFlow to support ONIX, or is it like TFX is the only way to go? Right now, I don't think there's, I'm sure there's no ONIX. I don't think it's even in the roadmap, or so that's a good question, like why they're not using ONIX. One of the feedback I got is that ONIX is kind of a talking point right now on, the argument is, who actually in this room uses ONIX to deploy, it's not that use is the point. And so investing some efforts on that doesn't seem productive for the TensorFlow team. I'm speaking for them, so I'm not entirely sure that's the logic, but that's what my feeling is. Any other questions? But in terms of portability, if you think of ONIX as a way to save your model in a kind of portable way across multiple platforms, TensorFlow actually includes Keras, and you can, if you want to have portability across frameworks, you can save your Keras model using Keras format. And once you do that, you can actually take that model and run it on any Keras implementation, and this also includes like CNTK, this includes, you know, all other piano and other libraries, it doesn't include PyTorch, which doesn't have a Keras implementation. So in terms of portability, even at the model level, it's arguable, you know, arguably the portability thanks to Keras is higher. But yeah, it's debatable. Yeah, yeah, oh, but you can. Keras allows you to save a trained model with all the weights and everything. Yeah. So, yeah, try it out. Now, obviously, I'd be happy if they had ONIX support, right? But right now, I don't think it's a plan. Well, thank you for your talk. Thank you very much.