 So I'm going to suggest the question because this comes up once in a while, especially since as of last year as we're onboarding the project into the Linux Foundation. And the question is, you know, what is the best way to contribute to the project coming from a developer who is not an existing developer contributing to PyTorch, you know, where to start. It's a big project and sometimes it can be intimidating for developers. There's, you know, thousands of developers acting, you know, participating in the project, a lot of companies, a large code base. So if anyone would be kind of insightful on, you know, what would be your advice for any, for a developer coming into the project, where to look for things to do and things like that? I mean, I can start, I guess. I mean, I mean, if you're brand new to the community, brand new to the code base. I mean, some of the most basic ways you can, you can contribute are on docs, for example, like there's the docs are plenty messy. There's always an opportunity to improve documentation. There's, I think one of the things that we with PyTorch early on wanted to do is identify like, like small issues, or small features that people could kind of onboard. So there's actually like, if you actually go to the code base, there's always, you know, like starter, I think we call them starter issues or starter features. So it kind of gets people warm and, you know, warmed up and kind of familiar with the code base. I think that's, you know, that's, that's one way. I think also like RFCs. So there's a fair number of RFCs that come through. And so, you know, following on those commenting on those, and then when it comes time to implement, right, jump in, like actually, you know, reach out to, to whoever wrote the RFC or group of folks that are reaching out to RFC and see, you know, where they want to help. Because I mean, everyone puts out RFCs, but no one realistically has all the time and bandwidth to actually follow through on the implementation. So those are probably the few things that, that off top my head. Thank you. Not for PyTorch itself, but for like hugging face, like you can text me. Personally, I have like multiple people who through like LinkedIn have like messaged me and been like, I want to work either in machine learning or at hugging face specifically. And like, we're like very helpful with like trying to shepherd people to do like more than just like even intro issues will help you with like big code changes. And I like give out my personal phone number to people and like, you know, assist them with stuff. So we're, we're like super pro like anything hugging face open source that people want to be contributors. And we'll give you jobs. That's how I got a job. So I have a funny story about why I attempted to contribute to PyTorch and ended up not being able to. My first ever thing I tried to do machine learning was that I marked as a intro issue on PyTorch that was make like we try to convert a PyTorch graph to Onyx. Like, you know, looks not every operator in PyTorch is compatible with Onyx. And so like singular value decomposition is not a Onyx operator for very good region. But this was not, but this was like an intro issue like, hey, like, this operator is not compatible with Onyx. And so like I ended up looking into it was like, I have no clue what PyTorch or Onyx is. And then like it ended up being like a multiple month rabbit hole of proposing a like SVD is an operator to the Onyx like SIG. It has to like implement like, like the actual like gradient of it, which is actually not very clearly specified anywhere. So this was like a funny like, it's an intro issue and ended up being like a multiple month thing of like, oh, actually, this shouldn't even be like in the like, there's a reason we're not going to convert this to Onyx or whatever. So yeah, I've personally had experiences like that as well. And on my repos, I very aggressively police things that are marked as as, you know, good first issue and regularly, like, I've probably like, for every good first issue, we have a probably untagged like four. Because yeah, like, I think that's a really big barrier for a lot of people for PyTorch for large and sophisticated co-basers in general. So there are a couple of things to maybe point out. At the end of May, for the first two weeks of June, we are hosting a docathon. We do a lot of posts on our LinkedIn. So if you're not part of the PyTorch LinkedIn or Twitter, please join because we will be continuously promoting that. It's a really great way for you to initially get involved. The other thing we'll do a repost is we did a Ask the Expert series when 2.0 launched. Those videos are really, really good. And again, those are all available. So we'll repost those next week for those of you who maybe want to get more familiar with it. But those are two great ways to get involved. Any of you maybe had a question, and then I'll come back to you. Yeah, so great work from all of you guys. Just have a simple question. In the brave new world of general AI and large language models, what gap do you guys see from the PyTorch site that we wish you had today? Actually, the challenges that we had with generative AI was lack of support for deployment in production, like with Totscript or something like that. So it was very hard for us to take generative, mostly encoded decoder architecture to production. Because the amount of effort needed to optimize is much, much higher. If this was there, like one year ago, we could have moved much, much faster. PyTorch can support better operators in their IR framework, so that branching conditions and all those things. That's what we are just starting to look at Dynamo. We don't have answers yet. But we have to take it to production and also fund research to do the future work. So we believe, that's again the belief, the advantage I have is I have one of that PyTorch text lead sitting next to me. And so I can actually have a conversation with him to understand what's the future. So one thing which I also would like to see as a developer is what is the future of PyTorch looks like? Where should I invest and where should I adopt? That's always been a tricky question for me as a lead. Like should I build, should I adopt? Adoption, what's the timeframe? Because we have deadlines and all those things. So if the roadmap is much, much clear, my developers can also contribute back. So coming back to generative AI, that is for me deployment to production. And generative AI inference started being more primitive than the training. So the cost of inference is pretty high for generative AI models. Hi, guys. Thanks everybody for the great talks. So my name is Alexey Kravroff. I lead open source science at IBM. It's an initiative we do with non-focus. And that is to bring together open source developers and scientists to accelerate science. So my question is, do you know of initiatives where you help scientific labs researchers advance fundamental questions of science, right? So for instance, we have a high-file lab project which will contribute to PyTorch community called Fuse-Med-ML, right? It's used for research in medical imagery. So I'm really curious, like does any of you track or have in mind some very interesting example where PyTorch, like your customers are scientists and you help them solve really hard problems which are very difficult to do without skilled machinery. And how can you help, like how can we work together to help advance science with PyTorch? I'll take a stab at this one. It's actually an area of interest for me personally. I'd say two things. You know, I had a team at Metta that focused on AI for science and I personally believe this is actually so, you know, while we're all looking at chat GPT and we're all looking at kind of, you know, generation of videos and that I actually think there's a lot of values can be created in the sciences. So areas like protein folding and synthetic biology, AI got a material discovery. There's, to me, there's a lot of real values can be created. So one of the projects that actually, I'll talk about a few projects, but one obviously is the protein project at Metta. We open sourced all those models. The NVIDIA BioNemo project is actually based on the ESM models. We're seeing biotech companies picking those models up, doing drug discovery. They don't share that, unfortunately, which hurts me on the open side because they kind of take models and I never really talk publicly about it or you have to twist their arms to talk publicly about it. The other is AI got a material discovery, which is kind of a, it's a project called Open Catalyst, but we took that same technology, which is essentially using large scale graph neural networks and looking at big universes of like molecules and being able to, to find the right molecules based on a select set of properties. So in this case, we were targeting, you know, a new type of catalyst material. Today, catalysts are based on very expensive materials. It doesn't scale. You can't make cheap green hydrogen, for example, based on current catalysis that's available. We took that same technology though and started applying it to things like direct air capture for carbon, and which is obviously important for climate change. And now that project actually Meta is actually building clean data centers. So basically all the carbon that's getting generated is getting sucked up and actually read in the carbon, the assortment material that's used to capture the carbon is actually being regenerated based on the waste heat from the data center. So it's fully, you know, fully closed loop system, basically. So I think that's to me, I think, and that's just a couple of examples. There's a lot more behind that. I think personally AI for science is huge green field. While everyone is focused on a bigger GPT-4 and GPT-5 for chat GPT, there's a ton of value being created elsewhere. Yeah. So from the HPC side at AMD, I've been involved with many deployments over the past five years of these big supercomputers. And almost every case is that they have now an AI preprocessing or post-processing of whatever math they were doing. So whether it's science and space or molecular side, sparsity, all that comes into it. But really understanding the data, it's all about AI. So it's not just large language models. So to just say that we do see AI in science as being very impactful, right? There's a larger human impact to this. And at AWS, we work in closely with universities such as Columbia University to train PyTorch-based protein folding models. For instance, OpenFold, which was an AlphaFold equivalent model, but it was a PyTorch-based model and it's completely open sourced from the model weights to the training code. All of it was open sourced. It was also a part of CASP-15. Similarly, we are also training a single sequence version of it, similar to the ESM-Fold model. But it's also again, based off PyTorch, and it's also intended to be completely open sourced. We also have some easy-to-deploy solutions that we have created and published on GitHub that you can then use to run inference or just use any of these models, right? But whether it's ESM-Fold, OmegaFold, OpenFold, RFDiffusion, you can run them pretty easily on AWS infrastructure. Great, thank you. I think, yeah, we do have several case studies on the PyTorch website. That said, there's definitely a lot of feedback we're getting today that we want to increase that. So, we'll take that note back and try to work particularly in that area. Hi, I'm Vinny Jaeswal. Thank you all for great talks and great work. So, now, since a lot of AI is being given to public and it's easy to use now, there will be a demand which is unforeseen. So, what is being done to keep up with that demand? Do we see shortage of hardware anytime soon? I know you all talked about optimization techniques, but how do we keep up with the industry demands as well as demands from people actually integrating GPT and other LLM models into their own businesses? So, any thoughts? Derek or Ashak, since it's kind of in your space? Yeah, you know, I think one of the challenges we have is to make sure, you know, if you look at the CPUs, right, there are a lot of CPUs and you could technically use them for a lot of the use cases. And I gave you examples in today's talk. So, our challenge is to make sure we educate users and research community to adopt CPUs more anywhere they can. And to follow on that, it's definitely there's a hardware shortage across the industry. You see it just by the number of devices being deployed and there's sort of our backlog. But I agree with the whole CPU side. It all comes down to what the power efficiency is. And so, that's usually what's driving because you have some cost to provide your service. So, you have to be able to, you know, provide the hardware at a cost that's affordable for the service. Yeah, for sure. Like across the full suite of devices. Yeah, I mean, there's definitely, I mean, there's a quote-unquote shortage of infrastructure right now. At least the demand right now seems to be infinite. I mean, inside Google, it's largely infinite because everyone has ambitions to do things, train large models across all the product areas outside and the cloud side. We just can't deploy infrastructure fast enough. This is, I think we're definitely in uncharted territory in some ways. But I also think this is a huge opportunity, I think around the optimization side of thing or we're utilizing, you know, the resources in other ways like CPUs, for example. And I think this is one of those things that the AI community tends to rally around when there's a problem and there's like clear value that's being created. I mean, think about how much faster like we're able to train a GBD3 model today than, you know, two years ago or three years ago. So, when, you know, when there's, there's a problem, typically the community rallies around it, finds ways to solve it, makes things faster, makes things more efficient. I mean, we're doing diffusion models on mobile devices, which is insane. And that's in stable diffusion on the chemo what less than a year ago. So I do think like we work on these problems and we have a smart community that tends to rally around them. So, echoing everyone else, I do think that having a range of underlying hardware to run your models is becoming increasingly important so that you give yourself more flexibility. That said, I think companies will look at how they can optimize performance to get parity with GPUs, whether it's model support, operator support, and even with existing hardware, how do you increase your utilization? How do you get more training efficiency using fewer GPUs to do the training? I think those are some of the areas that we'll start looking at. How do we build out these architectures that are stress tested so you get the best utilization of the resources that you do have as well? I can answer from a developer side, right? Actually Pytorch as a framework doesn't need large hardware. You can kick start even on a CPU. I started first my modeling on an Intel CPU. That's when you start expanding more and when you want to train larger is when you go for the hardware. As a developer, I think Pytorch as an organization doesn't have any affiliations to any specific company. So they are enabling us to start early. And the way I look at even your phone is actually a good, yes, enough compute to actually deploy a deep learning model. So I don't think people should worry about hardware when they kick starting their learning. But when you scale up when you're at 90% accuracy, when you want to go from 90 to 91, there's no other way other than throwing hardware at it. And hardware shortage, it's there everywhere because one side of the industry is going larger and larger models. And the other side is also scaling up inference. So my opinion is you should start small and incrementally get there. That was my journey. Even when I strain all this thing, I started very small. My learning was on AMD, Intel, all those hardware. And always understand there is compute there for you to access. It's that you need to step ahead. And I need this to train my model. I'll unblock myself. That's what I would do. I said I will approach learning something new. Okay. Anybody have anything else they want to add before we adjourn? Got one more question. We'll take one more. Thanks. It's been really awesome here to just learn from all of you. My personal, I'm really invested in like personal AI, running on device. I personally feel like it's a terrible thing if it ends up being all centralized and having to over the internet, you know, because I feel like AI's future is going to be personal, you know, personalized to all kinds of data. And so having centralized, you know, data warehouses running it is just is brilliant. And so I obviously have been very happy to see the last six months, spit stable diffusion and with bloom and other sort of initiatives, people fine tuning lower all these different innovations. And so I'm to ask that sort of to extend that question about hardware shortages and about sort of like just in general, deploying, starting small what do you see in terms of the paradigm shift? Like are there like some really incredible paradigms coming up that will allow us to actually maybe for the most part abandon super large models in favor of small models and aim towards that as a community? If for no other reason than just for the sake of personal AI and data privacy, but also the other is like, do you find things like open AI sort of keeping GPT fours technical details private? And like you mentioned companies like in biotech who are benefiting from this, but then not sort of talking about it. Do you find that to be a significant detriment? Like if you see compared to like, you know, llama and the incredible innovation we've seen like literally mobile devices, as you mentioned, running models, just people independent break lever figured it out. GPT four for some reason needs to be private because of safety. I don't know if you have thoughts about these things. I just wanted to sort of start the conversation on that. I mean, I think, I mean, on the open versus closed, it's pretty obvious where I stand on that. I think, you know, I look at there are a lot of startups, some of which are not public yet. They're still in stealth, but they are planning to be, you know, kind of going for an anthropocore open AI type of model, but be open and kind of white box about their models, which I think is a good thing. I personally think it's better to be open. I think that the pros far outweigh the cons when it comes to being open versus closed. And truthfully, like if it's closed, like who, who is the paternalistic entity, right? Is it open AI? Is it Google? Is it the government? Like, so I actually believe that openness actually should win out. I think where you get started is, I mean, the rate of open source models at this, I would say quality has never been, it's never been like this. I made the joke of on LinkedIn, I think yesterday about this being a real model zoo these days, because, you know, we started out 10 years ago talking about cafes, you know, the original model zoo and those models of pale and obviously in comparison to the scale of things that are available today. So I do think part of the problem or part of maybe part of the solution here to the infrastructure shortage is to iterate on smaller models. I think that's, you know, the even these seven billion parameter models are really capable. And if you tune them to your application, I think you can do incredible things. That's a piece of it. And you're going to end up with more decentralized models running on device. You know, I think that's, I think there's still going to be this, you know, kind of monotonic scale thing that big companies that have big resources are going to go after because they're going to try and push on the fundamental capabilities. And they have the right to do that. They have the resources. And if there's a business case to be made for training something at that scale, some poor product manager like myself probably has to go and make that case. But I do think that the open source world is not going to stop. More open source models are on the way. And so they're all going to be great starting points. There's going to be continued to be a good starting point for developers. I mean, unsupervised pre-training, like not having to pay for that is magical, right? You just take a model and then you can kind of do it. You can instruction tune it. You can do RLHF. You can do whatever you want with it. So we have a whole team that's now called like open reproduction. I think just in the last week they've released two big code models. I think the name might actually be big code, forget. But yeah, like that's our whole, like that's as a company that's like our whole thesis is like open source will win out. Oh, there's the name. There's the name. I'm not familiar. I'm very removed from that world. But I'm, yeah, they've done great works. Yeah. I mean, our whole thesis is like open source will win. So like we're, we're very, we have much more like in the works on from the open reproduction team, like there'll be a bunch of stuff coming out in the next few months, I'm sure. Okay. Anything else you want to say as we close out, Ibrahim?