 So, I have to say, I'm kind of in awe of the statistics around PyTorch, and I think when we... Why is there a number here, that's weird, huh, weird, let's get rid of that, I'm incredibly inattentive. You know, when we were, oh, weird, actually something happened here, there we go, aha. When we were conceiving the PyTorch foundation, honestly, it's amazing, I would say I pinged Jim on, I think, LinkedIn messaging, and it was like, hey, we're thinking about doing something with the foundation with one of our projects, right, we didn't tell him which, and of course he's like, oh, it's a PyTorch, please let it be PyTorch, please let it be PyTorch. I can't tell you, but we need to talk, let's have a conversation. And so, yeah, honestly, I'm so pleased at the way it worked out. We had no idea if it was going to be successful or not, if anyone would show up to the meetings, if anyone, you know, if it would grow, and I think it was... The results have been spectacular, so thank you, thank you so much. So I'm Joe Spezak for those who don't know me, and I'm going to talk about PyTorch, a project that has near and dear to my heart, no matter where I go, even if I go to Google, I'm still going to love PyTorch, which I am, of course, at Google now, and I'm going to talk about how PyTorch became the foundation of this crazy revolution in Gen AI and LLens that we're in today, and I think it's like the overnight success is like one of the ways that, you know, we think about it, like in the last six months, this craziness has happened, but I want to talk actually about the foundations of it. I want to talk about like leading up to this, the last five, even ten years of where all of this came from, because it didn't happen overnight. It was actually born on the blood, sweat, and tears of open source developers, people building libraries, people merging pull requests, fixing issues, which is obviously near and dear to this conference here in open source. So before we jump into that, very briefly on myself, again, I'm at Google working on Cloud TPU mainly, but also going to be investing, of course, in LLM and large model platform, as well as working on some applied ML work in my next, I guess, increase in scope, so to speak, I also do a lot of work in the VC world. I'm an executive resident at Canvas Ventures, if you're familiar with them in Porto Valley. I also angel and as well as my advice, companies like Lightning and Thropic is a couple of stealth mode companies I invested in recently, like contextual folks coming out of fare, as well as Run House, which is a former PM of mine who actually worked on PyTorch. So really awesome group of companies I'm investing in and partnering on. Of course, I was a founding founder of the PyTorch Foundation, where Sumith and I worked really closely to build it, and then previously I was at Metta for about less of five years, working on PyTorch, building partnerships, built a large model scaling team, but then also built an organization in fair research focused on math and science. So I have a bit of a full stack, I would say, background in the space. So this large model wave, I want to actually put this to some numbers really quick to actually give people an idea. So the old timers, I guess, like myself, who kind of look back in the 2013 time frame there, machine learning was like this less than 20% Google search term, where it was like, okay, this is kind of cool. And I remember at the time I was actually at Intel, and I attended a Berkeley retreat, and it was a free trip to Tahoe, so I couldn't turn it down. And I came back and I said, holy, you know, actually I'm on camera, yeah okay. I said, holy cow, this is gonna be crazy. ImageNet had just came, you know, I was surrounded by academics that were just pushing the state of the art. I met Yanching Jia, who was building Cafe at the time, which ended up being a foundation of Facebook's machine learning, of a lot of other companies, Yahoo's machine learning at scale. And you can see, you know, like it was kind of, it was perceptible, but it wasn't where it is today. And then you fast forward 10 years and you can see it's everywhere, of course, but what's really interesting in this data is if you look at large language models and you look at generative AI just in the last six months, it goes from basically being esoteric to just absolutely everywhere. So I think what's really happened here is we've brought it into the mainstream, chat GPT, and, you know, we've now kind of hit virality to the point where my parents, who are the least technical people on the planet, know what it is and they actually play with it. My father's a criminal psychologist for the record and he knows nothing about machine learning. He just knows we have free food at Google and he asked me about it every time I talked to him, but he's 75 and he likes these things. So and if you look at it in another axis, you look at the VC finding, which is interesting. And you look at the scale and just in 2022, you know, $2.6 billion invested in 110 deals just in generative AI alone. And this is going to scale even further because there's been dry powder in the VC world the last couple of years with the pandemic and, you know, holding people holding back on investments and so on. It's it's going to start scaling even crazier. And of course, the amount of compute actually these days, semi dictates how ambitious you can be. So we actually see this kind of feeding frenzy with startups trying to get compute deals and trying to, you know, secure the ability to train really large models. So this is actually poised to even go crazier. And I think one of the things that's interesting, so obviously being at Google, I have a different perspective now after kind of being a metaphor while, you know, the attention paper, we all talk about the attention paper is actually the semi inception point. It's wild to look at the start the founders of startups have just come off that paper alone. You can see, you know, well, I'm technically she should Nikki have actually left adept, they're actually starting something new as a few months ago. But you can see, you know, Noam, obviously a character, folks leaving for cohere near open AI. It's, you know, inflection obviously just came public with with pie. But just the amount of money being raised and the scale of these companies and just the the dispersion of the talent from that one paper has been incredible. And obviously it's given a lot of heartburn to Google as a company because we've lost a lot of great talent in the process. But what we've learned is scale actually creates these emergent properties. This is from our palm paper, which is now available as an API. So if you go to vertex or go to our offerings now, you can actually play with this. And this is in a lot of this progress is actually being pushed by the startups and obviously large companies as well. And, you know, I have to talk about parameters, even though parameters don't mean that much these days. They it is still one way we look at model scale. But what I'm kind of proud of is my old team that at Meta creating the Lambda models, they prove that you can actually do a lot. You can look at the 540 billion parameter model from Palm or you can look at GPT three. But with the new scaling laws, you can actually do a lot with even under, you know, 100 billion parameters or even the models are being released by say mosaic and others in even in the single digit, parameter billions of parameters, they're still quite capable. So I actually firmly believe there's going to be a bifurcation of this. And we're going to talk more instead of parameters, we're going to talk about capabilities. And those capabilities are going to be, you know, there's going to be a smaller parameter count or smaller size models. But then there's obviously going to be these world models that only big companies or people that can access a lot of compute can can actually train. And so we're in for an interesting few years, I guess. But we're here to talk about PyTorch, right? So let me let me give back to PyTorch for a second. So I want to give everyone a little bit of a history, historic perspective on where things came from. So we started out, I guess, who is a user of Lua Torch? Anyone? This is going to date you. So be careful if you raise your hand. Because, you know, yeah, okay, so no one's no one's brave enough. I tried it like when I first in the early days, when I, I tried them all right, I tried the piano, I tried Lua Torch, you know, obviously cafes I had to. But really, PyTorch was born out of the Lua Torch project. So we're not in Colbert who's at Apple now, if you actually one of the more interesting things as inside is if you go to the PyTorch GitHub repo, look at the licensed TXT. It's really interesting. It goes back like a couple of decades, actually, there's a lot of people who actually have copyrights on that code, including many folks you wouldn't expect. But this really all started, I would say in the research world, I was at that ICML, where I think they were hosting a workshop at Microsoft, and, you know, they announced, you know, support for Python on Torch. And and that was that was it. It was boring. It was it was kind of Python on on Lua, basically as a binding. And the goal there was to, to basically be one with the community, right? The Python community was was really growing was was diverse had a lot of libraries. How do you connect with that, that community? Well, John, the Coon will still say that Lisp is the best, right? You should go and use Lisp even though no one uses Lisp. But I think Sumith is right. Sumith was right at the time. And so was Adam Pashka, who was actually the actual developer who created PyTorch. And I'm not sure if you can actually read that tweet, but that that is the sun will tweet from under carpati that goes back, you know, I've been using PyTorch for six months now, my my skin is clear. You know, the air smells fresher, you know, I have more energy, right? And that kind of started off as kind of a silly tweet at the time, but it kind of set off this this virality. And and people got really excited and started using it. And then he fast forward a year or so here. And this is kind of what I joined from. From the other side, I was actually at Amazon prior. And we started to really build the community around it. And, you know, hugging face was tiny at the time, they were less than 10 people. Fast AI was just Jeremy Howard, and, you know, whatever time he could get from Sylvan, Google, and a few other folks, the open mind community around privacy, they were very early days for a lot of these communities, but they were building on PyTorch because it was easier to build on them because it was Pythonic, it was defined by Ron, it was just easy to integrate with Pyro out of Pyro team out of Uber. And then, you know, boom, we announced PyTorch 1.0. This was our first first year we did a DevCon. And we started to really in earnest build out our partnerships. So you can see some of the partnerships there. TensorBoard, I remember bringing a case of scotch over to Google at the time and after they landed all of our pull requests, so you didn't have to have, you know, a TensorFlow dependency, you could just, you know, from torch.utils and port tensorboard and, you know, specify your log deer and you were like up and running on TensorBoard. Beautiful. Torch serve, we built with Amazon, MLflow, we built with Databricks, Kuflo with Google. Obviously Lightning was one of those projects that came out of FAIR actually as an intern, Wil Falcon. And then obviously we started to see the demand for the production ecosystem start to take hold. So it wasn't like we were going into Amazon and Google and Microsoft anymore and saying, hey, please support PyTorch, they were coming to us and saying, you know, help us support PyTorch in our platforms. And you fast forward all the way today and you see what's become of that. If you go to papers with code, which I highly recommend that site, by the way, you can go to something, I forget what it is, it's like trends, I think it is. If you go to paperswithcode.com slash trends, it'll actually show the framework usage and how many libraries and papers actually get generated by framework. And you can see that big orange blob in the center is PyTorch. It's just incredible. ChatGPT obviously built on PyTorch. We built that partnership with OpenAI, they went all in PyTorch, stable diffusion, built on PyTorch. The foundations are there. It's pretty incredible. PyTorch basically is the revolution. So I do want to actually talk a little bit more about why we invested, though. So I get this question a lot. And it's always, well, why did Facebook, or why did Meta, why did they do that? Was it altruism? To some extent, yes. I think we built PyTorch with that in mind. But there were six reasons, I would say, that we fundamentally, when we actually had to go to our C suite and say, here's what's successful, which you have to when you're burning hundreds of engineers, and you're, at one point, I was having to stand up in front of our exact team and say, we are actually prioritizing the bugs that Uber is finding instead of our own production teams. Those are hard conversations, but we actually were able to convey the long game. And the long game was when you have a common platform and framework with the community, you get things like a timeline edge. You get to see, in some ways, the future. People were building libraries on a daily, weekly basis. I would get, you know, it was a very small team at the time. And I would get these, you know, pings over chat or pings over Slack, people would want to get onto our ecosystem page. And you would see all kinds of new things happen, whether it was around computer vision, whether it was on NLP, whether it was new training loop abstractions, different tools and things that would make things more productive to use PyTorch, take PyTorch into the world of reinforcement learning, whatever it was, they were all built on a common framework. This was incredibly powerful. So we started to pull these things actually into Facebook. We started using them. Corny, I was one of them, for example, in the computer vision world. PyTorch Lightning today is used at scale for a lot of the models. Obviously, software leverage. We took these, you know, I mentioned geometric lightning. We took these internally. Hardware. So one of the things that, you know, being kind of self serving a little bit is by creating a ton of demand in the ecosystem. Every hardware vendor was forced to integrate with us, right? So this was a nice little Trojan horse that said, okay, well AMD or Graphcore or all, you know, and then it becomes much, much easier when you're on a common framework. Then you can benchmark, then you can evaluate. So there's, you know, I'm not sure we actually ended up taking any of that hardware internally and in front, but we had the option because we were on that same platform. And then obviously there's other pieces that are more, I would say, like vanilla open source, but controlling our own destiny was actually pretty important. And the rest is, you know, recruiting and mind sharing all that stuff. But controlling our own destiny was actually a big reason why we did what we did. And, you know, truthfully TensorFlow, for example, was on the list of things, you know, one of the options we considered. And it was at the time, it was growing like crazy and we considered it and we looked at the governance, we said, okay, what happens when we're in a situation when our production team needs to land code and we are at the mercy of some other team at potentially Google, right? Not a good place to be in. So I think controlling our destiny, we actually governance, if I had to boil everything down, governance actually was a very, very important reason for the decision. But there was one big thing that we need to think about for this to actually all work. We had to take a community first approach. And I think we probably took this to heart probably more, more than I would say than most believed. And so if I kind of boil it down to a tagline, I would say PyTorch needs to be seen as a neutral brand, giving first class participation to every major stakeholder investing in the ecosystem. So this wasn't that, you know, PyTorch was an open source project that we kind of owned and held and there were these contributors out there and they were nice and we just kind of gave them, you know, the ability to access the code because it was an open source. No, we, they were literally on the same level as anything that was coming out of the company. So whether it was a library coming out of fair, whether it was, you know, a new capability that was getting launched, a blog, whatever, everyone was equal. And we fought on a daily basis to make sure that happened. And we took that really to heart. And I think that's a lot of the reason why we were successful when we were. I mean, I think if you look at some of the projects over time, I mentioned Open Mind, CAPTOM, we went to the point of almost detriment. Lightning, for example, you know, if you look at the training loop abstraction library, I mean, who uses lightning today? Anyone? A few folks, okay. So how many, who knows, like, about all the other training loop abstractions like Ignite and Poutine and Catalyst and no, no, okay. There's a bunch of them, right? At one point I was getting at least one or two a week where people would find a cool way to abstract the training loop because we never had a model.fit in PyTorch and still don't today. Like there's no Keras like model.fit. And that was actually by design. And we had to hold the line there because we wanted the full kind of exposed functionality for researchers. And if you want to use an abstraction, you can use lightning or you can use Catalyst or you can use something, you know, that is maybe a little bit simpler and easier to use. That's totally fine. That's where the community comes in. With projects like Open Mind, for example, you know, we were releasing our own libraries around differential privacy. There was a project called Opaques, which was a great differential privacy library. We actually went and shared it with the Open Mind community even before launch and let's figure out how to integrate. Let's figure out how to collaborate versus going in community and competing with the community. So we went to great lengths to make sure that we did take a community-first approach. And I think it was actually really successful. And I guess, you know, the results kind of speak for themselves. I would say, you know, network effects is what I would boil us all down to. And, you know, things like Lama or things like Open Mind or Lightning or, you know, these are the kind of outcomes we were looking for and we were lucky enough to get them. But it came with a strategy and it came with a vision and we had an operating model and we had, you know, air cover and we had a vision and, you know, we were able to execute it. So why is this guy at Google talking about PyTorch? So, by the way, there's a few folks here from my team definitely hang out with Milad and Shaheen. You'll want to chat with them a little more. They're definitely, they've been working on PyTorch a long time, so definitely chat with them. So at the highest level, Google is investing in every layer of the stack. We build Silicon, we build servers and racks and data centers and all the way down to, you know, to infrastructure as a service, both internally and externally. We have an ML platform that provides basically, you know, your kind of traditional things like notebooks and MLops and all the way through to models as a service and to kind of agents that you can then deploy in your applications. And you can see it's kind of a full stack. So depending on what you're talking about, we're deploying more and more obviously models as a service these days or providing APIs for models like Palm and providing BART as kind of a platform for experimentation evaluation. So you can basically, you can say we're doing pretty much everything you can in the AI space, I would say, every kind of niche of it. And if you look at PyTorch, especially down in these two levels, this is really where we're investing. But what is it concretely, I guess? So this is a project that we actually started back in 2018 when I was at Metta and actually working with, you know, Vishal and Shaheen and Zack Stone and it was it was a collaboration between between the two companies with the single focus goal of basically supporting cloud TPUs kind of cutting edge hardware on PyTorch. And I think if you if you're at the first DevCon we did in 2018, you saw glimpses of it, right? We teased it up on stage and then the following one in 2019, you saw, you know, myself and Vishal up on stage, you know, talking about the more broad availability. But you can see basically we made the project was available, it started in 2018, started to become available in 2020. It basically the core of this was a lazy tensor idea. So, you know, you kind of think about the continuum of ways to execute operators, you can compile operators. So you have a defined by run, which is kind of the eager mode, you know, way pythonic way that PyTorch executes operators at the whole other end of the spectrum, you have something called AOT or ahead of time, full graph compilation. And then we found this basically this approach that's kind of sat in the center called lazy tensor. And lazy tensor is essentially you're really only executing executing ops when you need to execute those ops. So it's kind of a it kind of it's in theory, it brings you really the best of both worlds. It brings you the performance of the hardware, but it also gives you the near kind of pythonic experience that you're, you know, used to kind of running an eager mode. Is everyone familiar with like the lazy tensor or eager mode or your mode in general? Okay, cool. And then, you know, in 2.0, we started to integrate that with torch dynamo. Hopefully folks have kind of heard all the compiler noise that came out with PyTorch 2.0 dynamo and inductor. And you can start to see, you know, really the performance for both TPUs, but as well as GPUs. So this project really I would say sits kind of just below the PyTorch layer. It provides a pathway to something called OpenXLA, which is a new open project by led by Google and many others, Waze here from Intel. And this basically provides you kind of this open compiler back end, which then can, can then connect to any XLA device. And it doesn't need to be a TPU. It can be a GPU, it can be XPU, whatever type of device you're looking for. In fact, it could be even a mobile device. And then on the other side of it, you kind of have your, your, you know, your ecosystem of PyTorch libraries, just like you always have. You have Ray, Hugging Face, Lightning, you have the whole thousands of libraries that are out there. And so a lot of people ask, like, what do I need to do? I'd like to get all this great performance. Well, actually, the code changes are pretty minor. So if you look at what I kind of highlight here, if anyone's familiar with, you know, setting a device in PyTorch, right, this CUDA devices, GPU, CPU device and so on. So you're going to set an XLA device on this case that then would would propagate through your, your step all the way through, you know, through your model. Basically, your model code may change like a couple of lines here and there. But the main thing is you need to import, say, the Torch XLA library and then basically set a device and everything kind of works from there. So pretty minor changes. But what we get is actually a lot of performance. And so I actually expected GPU data on this Milad, but I never got it. But this is all TPU data. And this is all latency. So we've actually been doing a number of optimizations for Lama, for example, working with the meta team, even at 175 billion parameters, which is of course, the GPT three scale, the latency is incredible. And this is, I think, probably an order of magnitude lower than what the original latency was. I think it was around 200 milliseconds. And now you can actually unlock the ability to actually serve, you know, these models, these large language models at scale. And of course, latency for 7 billion is like negligible, right, three milliseconds. So incredible performance for both TPUs and GPUs. So I'll cover a little bit about ways to get involved. But honestly, this is all motherhood and apple pie for an open source person. So I'll go through this. And by the way, feel free to ask me questions. Actually, don't mind being interrupted. So I want to call out a couple things. So one Jeremy Howard is awesome. I've known him for ages now. He continues to build his courses on PyTorch. And so he just actually released fairly recently the practical deep learning course. He moved in his family all the way back to Australia. He's but he's still advising and still heavily involved in community. I highly recommend taking his course. So way number one of getting involved, take a class or take a take a MOOC or play with tutorials. PyTorch has incredible tutorials. When I was there, we worked really hard to make them great. The 60 minute blitz is still probably the most popular thing. I think on the PyTorch website. And that is, you know, at this point, quite a few years old. The coolest thing I think that we actually did on the tutorials when I was with PyTorch is we actually had co lab integration. So who uses co lab? Okay, I'm going to see some hands this time, hopefully. All right. So co lab was amazing. So one of the things that I did with my team at meta was we taught a class at a master's level course at Georgia Tech on deep learning that we then brought into something called the AI learning alliance to drive more diversity into AI. We taught that course almost entirely in co lab. And then the students would actually go and pay the 50 bucks for co lab pro and get you know better GPU and a longer runtime. And it was it was absolutely incredible. So you can actually click through on most of the tutorials on PyTorch or right into co lab. And you can start playing with code right there. You don't need to spin up a VM or pull out your credit card or whatever. It just works. It's beautiful. And there's of course GPUs available for free. So I would definitely check that out. Another thing you can do is you can compete in a Kaggle competition. So this is kind of a cool. Well it's a bit bittersweet but mostly cool graph on the left here. This is actually deep learning usage on Kaggle a PyTorch versus TensorFlow. And you can see that dark blue by the way is PyTorch. So PyTorch is everywhere in Kaggle which is pretty awesome. I was actually looking through the challenges when I was putting these slides together. There's a million dollar challenge. It's got a baseline in PyTorch if you want to play with it and try and win a million dollars on the Vesuvius challenge the ink detection. Go for it. Kaggle has really incredible statistics have 13 million users million monthly. They have up to date PyTorch 2.0 Docker images 36,000 monthly PyTorch users on Kaggle. So it's pretty cool. And the challenges are awesome. And obviously it's a great badge of honor to win a competition on Kaggle. And I think lastly, just again, motherhood never pie, host a meetup or join a meetup. I saw Imad, its stability is going to be hosting one, I think in two weeks in New York over at the Lightning AI office. So, you know, join a meetup, get involved in the community, come to these events, right? Talk about your work. I think, you know, we've been blogging a lot obviously with the LF. It's been awesome. The I hate to say how the old blogging used to do. We basically it was a pull request that I used to just merge myself, which is dangerous. It was a loophole in the system. But talk about your work. You know, reach out to the LF folks. It's been actually pretty incredible. We've been blogging from the Google side on pytorch.org. Tell your story. There's actually a community stories page as well. And you can, you know, basically talk about your work, talk about, you know, things that you built and so on. And if I can help in any way, either personally or if it's from Google, whatever, I'm happy to help. I love building this community. And I'd love to help tell your story as well. That is it. Thank you. And I can answer any questions you have.