 Our next presenter, Shagun Sadani, is going to present Non-Pied Parts, which it's about that he's going to present next week in Canada. Thank you, thank you for the introduction. Hi, I'm Shagun. I'm a master student at the University of Montreal in the Mila Lab. And I'm going to talk a bit about PyTorch and Non-Pied. So how many people here have used PyTorch? Awesome. How many people here have used NumPy or use it on like a regular basis? Cool. So I think a lot of people are going to love the talk. So what are we basically going to talk about? Introduction to NumPy and PyTorch, which is going to be very quick because most of you seem to know about it already. The second part would be why PyTorch at all. And when I say why PyTorch, I would not be talking about the machine learning part of PyTorch. So I'm a graduate student. I use PyTorch for machine learning, but there is this other side of PyTorch where people who just use NumPy can also use PyTorch without having to worry about the deep learning or the machine learning hype. They can just use PyTorch to augment their NumPy use cases. And this is what the talk would focus on. It would not focus on the machine learning aspect of it. Then some pitfalls. So if you're using both PyTorch and NumPy at the same time, certain things can go wrong and we would see what those pitfalls are. The next part would be what is missing in PyTorch? Basically, can I use PyTorch as a drop in replacement for NumPy? If no, then what things am I going to miss? And what's the future roadmap for PyTorch? Disclaimer, I have made a few contributions to PyTorch, but I'm not at all affiliated to the core development team or anything. So I'm just a neutral user just like you people. So what NumPy gives us on top of let's say standard Python data structures that like list, instead of using a list, you would use a NumPy or the so-called ND area or the n-dimensional area because it gives you speed, because it gives you a lot of built-in baked-in functions, functions which are optimized functions which do exactly what you want them to do. So you do not have to write those functions from your hand. The standard example when talking about NumPy is you have an example where you're iterating over a list and then you're iterating over the NumPy array and then you show that NumPy array is up every 10 times faster than a list. So that is the basic idea behind using a NumPy array, which I'm sure everyone already knows about. Other benefits of NumPy include things like it allows you to easily integrate C++ or Fortran code. So you can write your loop in Python and then you can actually learn the loop in C++ which makes your loop much, much faster than had it only been in Python, like the benefits over the inbuilt Python sequences. And yes, so this was the example that I was talking about. So you have a list and given this list, you can easily convert it into a NumPy array by just passing it through np.array. The point is given a list, it's very, very easy to migrate to the NumPy ecosystem. The point which I'm trying to convey here is when you had this standard Python list and you were thinking about migrating to NumPy, there were a couple of things that were in your mind. Would this transition be easy? Would I have to rewrite my code? Would the transition actually give me the kind of speed-ups that I'm looking for? Now, when we'll be talking about PyTorch, we would be raising the same questions. We would ask whether going from NumPy to PyTorch is easy or not and is it really going to give me the computational benefits which I get when I move from a Python list to a NumPy array. Right, PyTorch, the open-source machine learning framework which is being developed at Facebook AI Research, commonly called as FAIR. What's similar between PyTorch and NumPy is the entire NumPy ecosystem, entire NumPy framework is built around this thing called as an n-dimensional array. Analogously, PyTorch has something very similar which is called as a tensor and we'll talk about it in detail. The thing which Tensor offers on top of NumPy is basically GPU acceleration. So if you have a GPU and you have NumPy, you cannot use NumPy with GPU but if you have a GPU and you have PyTorch, you can just take a tensor which is basically PyTorch equivalent of an array and you can start running it over a GPU and we'll go through all these things in detail. The second part is that it offers you something called as reverse mode auto differentiation which is more important for the machine learning or the deep learning part of it so we would not be focusing about it but it's something on top of or something extra in addition to what NumPy already offers you. Just a slight bit of discussion about why PyTorch is preferred over other machine learning frameworks because there are plenty of them available right now. PyTorch is kind of a Python-first framework. It has a hybrid front-end which means you can write your code in Python as well as in C++. At the same time, it's not a wrapper over a C++ library. I mean, of course it's a wrapper but it doesn't... The kind of errors you get are Pythonic errors. You can look at the error and you can make sense or there's a shape mismatch or something. Other benefits is that it supports distributed training both over a CPU and a GPU which would be something useful for us even when we are just using it as a replacement for NumPy and not really talking about the machine learning story and it has an ecosystem of solutions which we would talk about towards the end. Oops, yeah. So PyTorch versus NumPy... I'm sorry. Cool. Right. So PyTorch has this base computational object called as tensor which is equivalent of NumPy's ND array and tensor has a lot of operations or APIs which are very similar to the way NumPy APIs work. I'm sorry for this. So for example, if you look at the creation operations which are basically the operations that you use to make a NumPy array. So np.once would make you a NumPy array of the given shape and the exact same thing with Torch, like instead of np or NumPy, you would write Torch.once and it would give you the exact... like the API remains exactly the same. If you are doing something like np.i and making an identity matrix of a certain shape or you are doing things like take a matrix of once and multiply it by two, you could do the exact same thing again just by replacing np or NumPy by Torch. So these two pieces of code are exactly the same and I have a collaborative notebook alongside in the repo where you can run the code and you would see that the results are exactly the same. Other than the fact that at one place you have a tensor and at other place you have an nd array. But other than that, the results and the APIs are very, very similar. In fact, when I was creating these examples, I just wrote the NumPy code and for writing the Torch code, I actually picked up the NumPy code and replaced np with a Torch and most of the time it worked perfectly well. Like I did not even have to look up the APIs to see whether the APIs are consistent or not. This is the extent of similarity they have. So just like you can take a list and pass it to an np.array function call to get a NumPy array, you can do the same thing with Torch.tensor and you would get a Torch.tensor array. Again, the point is the creation operators or a lot of pieces of API are very similar. So if you are used to NumPy syntax or you're used to the NumPy way of doing things, picking up on Torch would be very, very easy for you or picking up on PyTorch would be very easy for you because the APIs are very similar. Then some people here would be using this other style of using NumPy where instead of using the equality operator you pass two operands to the function and you use the out variable of the function. What I mean by that is when you're doing something like this so you have two arrays x and y, you can either add up the arrays and then you can assign it to y or you could call np.add x, y and you can write output into y. So the two pieces of code are exactly the same. You could do this kind of syntax in Torch as well again without any changes. Just that your np or your NumPy is replaced by Torch. Again the take away messages, the APIs are pretty similar as long as like at least for the creation parts they are exactly the same. Another very useful feature of NumPy is the advanced indexing which it gives you. So you can take an array and you can start indexing it in weird ways in the sense that you have an array which has got four dimensions and then you are slicing across one dimension and making strides and so on. Again the exact same advanced indexing is supported in Torch so I have some weird advanced indexing code and I could run the exact same code again just by replacing np by Torch. Again the point being the APIs are pretty similar plus using things like negative indices and so on you can do those things again with Torch. So far the things we have seen I'm showing you a toolkit which is very similar to NumPy and this is the only point I'm trying to make so far this tool has an API which is very similar to NumPy. So now the question comes up if this toolkit has an API which is very similar to NumPy and I already love NumPy, why should I consider to move? What's wrong with NumPy that I should consider something like PyTorch? And I would use the same kind of example that we use when we talk of lists to NumPy. I would show an operation being done in NumPy and I would show an operation being done in Torch and we would see the difference in the amount of time it takes. The basic argument here is NumPy cannot use GPU acceleration whereas PyTorch can and in certain cases it's going to give you a huge speed up. So here I've got an array, this array is a huge array it's a 10,000 x 10,000 array and I'm doing a multiplication of this array with itself so basically I have this x array and I'm multiplying it by itself. Doing it in NumPy takes around 36.6 seconds doing it in Torch on GPU takes less than a second it takes 800 milliseconds. So this is the key benefit this is the key benefit that you get when you start using Torch instead of NumPy and the API again remains very similar np.matmul has been replaced by torch.matmul so it's not that you have got a whole new learning curve but you can now start using better hardware without having to give up on the syntax and so on. So this is good but then the question is do I need to rewrite my NumPy code into this Torch code even replacing things like import NumPy as np to torch is going to take time, you don't really have to you can keep doing all your workflows in NumPy and whenever you reach a point where there are heavy matrices or large matrices involved and you're going to take things like their product or matrix multiplication and so on that is the time where you can move to Torch and moving from NumPy to Torch is very easy all it takes is just one function call and the other way around as well so if you have a NumPy array and you want to convert it into a Torch tensor it's just one function call and if you have a Torch tensor and you want to bring it back into NumPy it's another function call so here I have a NumPy array all I do is torch.fromNumPy and I have a Torch array so that's it, that's all it takes and the other direction where I have a Torch array and I want to convert it into a NumPy array.NumPy this is just a sample code which shows that you could start with a NumPy array convert it into Torch and then re-convert the Torch array into NumPy array and you would see that the two arrays match so you started with NumPy, went to Torch and then went back to NumPy and everything remains the same so this is what an idea workflow would look for people if they do not really want to rewrite everything in Torch you have your NumPy logic as it is then there comes an operation where it's going to take a lot of time because the matrix is huge so at that point you move your data to the GPU you use the GPU for the costly operation and then you come back into NumPy so really only that one piece of code which is the so called bottleneck that bottleneck is what you send over to the better hardware you let the hardware take care of that part and then you bring back the computation into your NumPy code and it can go the other way around as well I haven't talked about those cases yet but there would be cases where you cannot use PyTorch because PyTorch is not feature compatible with NumPy PyTorch is not a drop in replacement for NumPy so there are APIs there are things which are still missing which you do not have in PyTorch but you have in NumPy so there would be cases where you are writing your entire code in PyTorch and then there comes an op which is not defined so you bring your data into the to NumPy you apply that operation and then you again take it back into Torch right to be clear the benefit that we have seen so far is coming from the GPU and not per se from PyTorch but the benefit is PyTorch makes it very easy for you to use the GPU so you should not be confused that PyTorch is much faster, no it's not it's probably as fast as NumPy definitely not faster than it but it lets you use GPU which NumPy does not let you and that's why you see those boost in performance by default all the tensors are created and they live on CPU and they can be easily moved to GPU between GPUs depending on how many GPUs you have so we have a notion of something called as device so you can make a CPU device by calling torch.device and sending the argument as CPU or if you have a CUDA device a GPU device you just say torch.device CUDA and this index says which GPU are you talking about the indexing starts at 0 and then you can have 0, 1, 2 and so on and once you have a tensor sending it to a GPU or to basically any device is very simple the tensor. whatever device you want to send it to and your work is done again going back to the NumPy workflow you have your NumPy code, you make a GPU device you send your data to the GPU you do whatever heavy computations you want to do there and you bring back your data that's it now when you are using PyTorch and NumPy at the same time there are certain pitfalls to look out for there could be certain things that can go wrong the first thing is I'll show this with the example of the NumPy array I put it to torch and now I'm changing the torch array I'm not changing the NumPy array what happens is it changes my NumPy array as well because when you use something like dot from NumPy the memory is being shared and this is one reason why this transformation or why this transition from a NumPy array to torch tensor is very inexpensive because you are not really creating a copy when you do, when you call torch.fromNumPy and you take a NumPy array to torch you are not creating a copy of the exact same data again so this is something to look out for and this is, this gives you performance benefits in a lot of cases thus yeah so if you actually want to make a new copy and not just share the underlying data then you call, then you use the tensor function not the dot from NumPy function the second thing that can go the second thing that can go wrong is whenever you are moving data to a GPU you need to create a copy that is something you cannot avoid because GPU does not have access to your CPU memory in that sense so instead of moving your data back and forth between CPU, GPU multiple times ideally move your data to GPU once do whatever computations you want to do there and bring back your data to the CPU because there is some cost involved when you take your CPU data to the GPU remember there was no cost when you were taking a NumPy array to a torch tensor because underlying storage was exactly the same as this CPU-GPU thing this computational cost kicks in oh yes, and the third thing is if you have a tensor on GPU and you are directly trying to convert it into a NumPy array that's going to fail you have to explicitly first bring it on the CPU by doing something like torch array dot 2 CPU device and then only you can call dot NumPy the reason for keeping it like an explicit error and not doing it internally is because sometimes it can lead to confusions as long as the data is on CPU it's being shared between the processes it's being shared between the torch library and the NumPy library so you could just implicitly assume that if this conversion was done implicitly, you could modify the CPU copy hoping that your GPU copy would also be modified which would not be the case so you cannot directly convert a GPU tensor to a NumPy array what if I do not have any fancy GPUs would PyTorch be slow for me? not really, so I don't have any benchmarks but I have been using PyTorch for quite some time mixed with NumPy and sometimes without NumPy and I have not really seen performance differences there have been cases where performance differences have been reported and the core team the fair team has been very responsive on those things so if there are operations which are slower they very quickly roll out a fix for that what are the other benefits of PyTorch other than GPU acceleration well, at the end of the day NumPy is a tensor manipulation in the array manipulation library which gives you this array and allows you to manipulate this array PyTorch is something more it allows you to manipulate the array and at the same time it lets you build up this entire deep learning framework on top of it the point is if you know NumPy you know PyTorch and if you know PyTorch it becomes much easier to transition into other things like machine learning and deep learning PyTorch ecosystem also has a set of other libraries which provide you a lot of flexibility in working on machine learning kind of use cases which I do not want to go in this talk yes, the other benefits well, as long as you have a NumPy array you can very easily convert it into Torch and vice versa this means other machine learning libraries in python which play very well with NumPy arrays would play well with tensors as well or with PyTorch as well so for example, scikit-learn and scipy which takes NumPy arrays as their primitive or the main data structure with PyTorch as well so you don't start to lose out on all these libraries just because you are using Torch tensors in some cases, specific wrappers also exist so you have something called a scotch which is basically scikit-learn plus PyTorch and then there's another side to it as I said, you can use NumPy to augment PyTorch as well so it is possible to write down functions in fact, it's possible to write down optimization functions in PyTorch using just scipy, NumPy and so on so it's not just really one-sided thing using PyTorch and having GPUs gives you this accelerated runtime and having NumPy allows you to augment some use cases or some APIs which are missing in Python which are missing in PyTorch so what is basically missing in PyTorch? one thing is there is no feature parity it's not a drop-in replacement NumPy uses this parameter called as dim for a lot of matrix operations whereas PyTorch uses access so this is one place where your code would break down PyTorch does not really have a good support for sparse tensors right now so if your use cases are where you have huge tensors but they are sparse then probably you would not really see the speed-ups neither on CPU nor on GPU but this is something which is on their roadmap then there are other tools like Qpy and a lot of other tools which allow you to which are basically drop-in replacements for NumPy and then let NumPy run on GPU the only benefit which PyTorch gives you on top of these kind of libraries is the following when you are using NumPy and you start using Qpy you are basically using better hardware and there is no denial about that if your use case is totally around NumPy and you just want to use better hardware probably Qpy is a better thing to go but if you are looking for better primitives better data structures, better abstractions to build your models or build other things on top of them then PyTorch gives you that flexibility as well so on one hand you can use Qpy using GPU without changing anything in your code but if you at the same time want better primitives or better functions which you can build upon then PyTorch is something to try out yes so the support for installing both the stable version as well as preview version is very well given on their website it's very easy to install there are no complicated errors or anything and if you prefer cloud there are built-in cloud disk in some sense for AWS Azure as well as GCP what's coming up December 2018 is when PyTorch 1.0 comes out which is supposed to be the production ready PyTorch and the API though it's not exactly in sync with the NumPy API but it's going to get closer and closer to NumPy API another benefit which I did not go very well or very deep is that PyTorch makes it very easy to run distributed code again even for basic tensor operations PyTorch has a pretty good community of developers as well as contributors and they run this discourse forum where the main developers are very active and they're very actively reply to your queries I'm thankful to Adam Adam is one of the core developers of PyTorch for reviewing the slide and giving me some useful feedback and yes so most of the stuff is taken from NumPy.org and PyTorch.org the slides and everything are available on GitHub in case you want to check out there's a library notebook where all this code is given and you can actually run it and see the kind of numbers I showed are real numbers and the API similarities which I showed also exist so you can just run all the code that we went through in the slides thank you Thanks Adam, any question? Thank you