 Hi everyone, so my name is Grigory Fursin. I'll try to speak up if you don't listen if you come here We just rise your hand So I'm very excited to be here first of all because it's my first for them I was really very cool to see many interesting tools out there many interesting talks And I'm also excited to present you this new community project Which is called collective knowledge and it's I think it's a bit an alternative view from a researcher perspective So we have all those cool tools out there. But as a researcher, I want to kind of use them And I'll tell you what are the possible problems and solutions to do it and In the past year, I was very glad that we worked with some great companies universities on nonprofits and They provided lots of feedback for helping them to use CK and they were providing some interesting They were doing some cool things like performance regression testing crowdsourcing experiments and even generating some automatically generating Reproducible articles, so I'll mention this and again the last year while working with those Organizations were actually validated our approach that it works with most of the tools out there. So it's makes me quite happy But what is it all about so in my past life? I was a researcher and I was working on machine learning systems and compilers and Basically when you when you have some interesting ideas and you want to implement it so I'm trying to look at different papers being published there and Last few years there were about thousands of papers being published on machine learning and actually someone told me when I had those slides that in fact Last year someone mentioned about 10,000 papers being published just some machine learning if you count reports Blocks and so on 10,000 so as a research I want to validate them and most of them actually didn't share code and data So what do I do? So, okay? When I was young I thought like okay, I'll start looking at some of the papers I will learn great tools which are doing which are working on and I learn I look at all the different tools out there Which are really helping you to simplify your life. I continue learning them. I become old I have white hair now and At the end I managed to implement my idea and when I implement my idea I suddenly realized oh TensorFlow API changed my new CUDA doesn't support my new GCC which is out there So what do I do? I start learning again or I just quit I get depressed or some people like Kenneth Try to build to create some cool tools like easy build. She's great. So but it's a thing in minority so we know all those problems and So I heated them 20 10 15 years ago, but last five years everyone was speaking about it So one of the ideas is about like every buzzword. Let's do reproducible research. Fantastic. How do we do it? five years ago we started introducing what we call artifact evaluation at many different conferences and the idea was that when we publish your paper you can voluntarily submit your code and data to some committee and we'll try to validate your results and It worked out very well from the first conference, which we did like about five years ago on people who got Five artifacts out of 25 papers, which was still like a little but it worked We tested methodology and so on last year. We got about 70% of papers that people pass submitting code and data will date them It's it's a good thing. Now it is bad. So when we started looking at it, of course, there is still no methodology about how you share and reproduce all those complex experiments performance evaluation and so on it's really still in infancy and what is really ugly That all past five years. I looked about 100 artifacts and all those 100 papers in top conferences And they all have 100 of scripts to do some experiments to download some models to Hardwire your past if you want to change your past you need to find the script somewhere fix it and so on And it's really so our regulators will spend more time actually on trying to figure out how to deal with all those scripts then actually doing fun stuff relating your result and validating it and At the same time, even though you have all those code and data shared when your lead researcher is leaving or your student Or whatever developer is leading all this no one is using all this code and data So it's really like for me. I'm really It's a shame that it happens because there is so much interesting stuff out there And of course the latest thing is like, okay, so let's use containers and these are fantastic tools as a kind of Solution as a snapshot but again, they're hiding the mess underneath someone still have to solve this mess. So it's not solved at all However, this experience with all those conferences when I was looking at all those artifacts I started realizing that all those hundreds of artifacts they have they do the same things all the time and they're simple things So if you look like what do you usually do as a high level like algorithm, right? I would say that I have my program image classification. I want to compile and run it with some I don't know. I would look at some different images out there I would want to kind of adapt to my software and usually I would try to detect that I have some GCC or LVM Intel compilers available. I'll try to find some or a data set usually get some hardware pass and so on But it's still the same and then I would run experiment collect performance stats Produce some graphs validate them and print you create your paper and this all of them are doing the same So what is happening? So this started like when I discussed it with all our colleagues I was thinking that we missing APIs common APIs for all those tasks. That's all Can we come up with some simple APIs which will automate all those tedious tasks Which you have so that they will reuse those APIs and they must be simple and what is very important Also, you can provide some meta information meta description for all those components which are out there so that everyone could use them and The big point for this is that now if we have APIs and those meta descriptions in some color format We can enable DevOps because this is like why do ops are not there and research because you don't have APIs How would you connect your Jenkins or whatever Travis with all these scripts out there? And that's how collective knowledge came into place So we start let's create just a tiny Python library We should provide you those human readable modules Python modules with huge human readable functions And they will provide you an access so this will be always like program mobile module with function compile run and we'll have Diction of JSON input and all this GCN and dictionary output because very easy to extend them And the same time all the data which you share you would also have some kind of unified meta JSON And what's the key is just your common line front-end to just call those modules for now That's all and if you like I know that when I was talking to researchers, they're like, oh, but it's too simple We don't like it. We want complexity But being in industry for a long time. I said no opposite like this simply it is the better it is And that's why we had many fights, but okay And at there now and you have all those components the more components you have you can now start assembling all the Experimental workflows from those components and do more and more complexity do some fancy stuff again I'll tell you later some more fun stuff that we're doing with that And at the same time when we're discussing with our colleagues We said that okay, let's now when you share like your code and data through github Let's just have some common format and usually it's just very high level information So whenever you see this collective knowledge compatible batch and some github repository around 100 now They have common information so first director usually tells you your module python module API So that you know that your further data is abstracted by this API and then second level directors You have basically what we call CK entries This is your data which can be anything from your soft in your soft abstractions you would Describe how you detect some software in your program would describe what their dependencies and so on Again very high-level late if you're interested you can look at all the information on the website But just trying to give you very high-level idea about what we did However, of course, it's not a magic so someone still needs to implement all those APIs And this is kind of was a tedious task in 2017. We got some First adopters that was arm and general motors and now of course they develop all those like lots of hardware Different algorithms. They need to provide big performance regressions with testing They have money multiple work groups working together So thinking that maybe they can connect all their work groups with the same framework and the same for general motors So we started gradually adding those APIs ourselves always our colleagues And just again, what were the four first APIs which we provided very simple again First of all, what do you want to have you want to describe the operating system? How you're like, I don't know like Compile some problem. How well what are you? How you can find some data there and so on and this can be between different operating systems So it can be Linux my class windows Armors contributing lots of stuff for Android then the same thing what we always do detecting platform information again Notice that our API afterwards they call different tools out there which you develop But we're just providing a common API so that everyone else is protected from the changes in the system The same installing whenever you have we want to adapt to your native environment So provided many APIs and many kind of what we call plugins for detecting software There are about there are around 500 software detection plugins So basically you can just say CK search soft with either a little VM data set and so on It will find you all the installations out there and the same for packages There are great tools out there easy builds back and some other ones So provided an API for those kind of tools so that Anyone can share their even more high-level recipes about which tool you want to use and what are the dependencies So if your software is missing you can automatically call and install missing so missing package for the tools available out there And just the last thing about those like very high-level thing about this CK because it still probably looks like magic We saw 20 years ago. I was working on performance utter tuning and basically last year I was talking with my colleagues from Raspberry Pi Foundation was a benapent and we're saying Oh, why not to like kind of convert and create a program workflow Which would do performance regression testing on different GCC versions on Raspberry Pi devices and we created this workflow, which basically was a Work which I did 15 years ago in my PhD. We did it in about a month. I don't run experiments We crowdsourced experiments so the same workflow now different users from Raspberry Pi's They were running the experiments on their machines and they were collecting data on our Synology to Torx server and suddenly like I think one week I collected so much data Which I think I didn't collect in five years of my PhD, which on one hand was kind of a good Experience for us here on the other hand I was crying that I spent five years on my PhD like collecting just collecting this data And I could do it in five months. So and now I can do it in ten minutes because That's fine, okay, and the cool thing that we created if you look on the website We prep with Raspberry Pi Foundation We created an interactive report which is automatically generated also through CK and if you look at here You will see all this what we did it generates PDFs with all the papers with all the graphs out there And I just want to go down somewhere. So here in fact, you will see all the way how we reproduce experiments You can do it yourself if you look through this paper But what is fun now is that because we have APIs Now we can actually embed some underneath this interactive graph. We can embed some Interactive graphs, which if you click on it in your paper, you now go. Oops, sorry No, I don't know some cool point. This is like performance tuning You'll get to the repository and you will see different information where this experiment ran What was your performance? You can view all the compiler flags, which were there You can replay it on your machine because it's again all down for replaying different War groups can do it and now just I have still I say I have still ten eight minutes ten minutes I wanted to show you a small demo just to show you that you can do it yourself now with a few clicks you can actually start participating in this crowd tuning and From your machine from any machine, maybe there will be some bug or something, but let me check if it will work This I usually don't like to do live demos because they don't often work Can you see the font? Everyone's okay? No? Too small? It's too small So now I'm losing time To get this, but okay By it's better So now I'm on one of our remote servers will actually perform some regression testing from regression testing and we share those CK repositories and Again, what I do is that you would see many CK repositories out there Which I usually get hub repositories which you pull on different work groups or communities work in parallel on those It's a distributed system. They work in parallel and what I will do now is so I have many programs here many data sets here CK list What I will do now is that what if I do the following so all the repositories on the CK BXE K Directory what if I'll do the following What if you do this at your website or for your user in the HPC center? What they will tell you if I would remove all your repositories and I'm doing it now and I hope that people will not fire me now if they're using this Website or this server because now I'm usually they're using my source. I have I think one minute to kind of restore it So now when I do CK last program, I don't have anything anymore. I don't even have this module So what do I do? So I start saying that okay, let's pull the repository with scroll tuning and It should normally go to the github. Yeah, we were shared and I start also pulling all the Repositories all those APIs different APIs out there to rebuild you the workflow to do this interactive paper or this experiment on performance tuning So I still have like 30 seconds. I think to restore it and I'm checking if my phone is known as ringing me, but not yet And I don't well so now we have the secret place so now I gain let's check So I have CK program and now what is interesting that I can just compile any program out there Susan corners that I might have benchmark and what it happens is that it starts detecting all the dependencies out there I don't have time to show you how it looks so you can go on github and you will solve the meta information about those programs But now I'm going for plugins to detect you all the dependencies. It needs compiler. So we detected some Intel compiler go through GCC Still have 10 seconds. So let's install. I don't know like some DC7 which time it's interactively tells you what is available on your system. So we're adapting to your system Clang we get fun. So now it goes see it found all the it detected all the dependency tells you how do you want to compile a program? Let's compile with GCC compiler. We compile it around the program the same program The first time because I delete everything I need to tell that my platform is Linux It can be done automatically. So it's important and I'm running corner detection of some images notice that all those images this again I pulled someone else pull a new repository and I'll have like hundreds of those images from someone else I don't need to substitute all the past CK automatically finds all this for you around the program I ran at 10 times I perform some statistical analysis and This normally works for 10 seconds. Yeah, I Performance I provide some performance formation notice that when you use Python to call CK You will have actually huge JSON file internally with provenance and now just to start participating in this Performance regression. I just do CK crowd tuned program and it ask you all different plugins what you can tune they can do models You can do in compiler flex. You can do whatever you want Just a lot of your things and now it starts tuning your program and sending results for our server back So in fact this reproducible article actually in few days I can even get more results and this will be live article because I will be getting your results I can apply machine learning so something what I was craving for 15 years ago Okay, I'll stop this so Just the idea is to tell that first of all I rebuild all this workflow now everyone still has access to all those Repoisters and it works. So it's very quick imagine now if I delete all my stuff and then it takes everyone like huge amount of time to restore I saw some PhD students who actually quit it PhD because there was like server died and they couldn't reproduce the restore stuff backup was not there and so on so now it's very easy to reproduce stuff and Just going back to I have five minutes. I just wanted to tell you a few things actually of our Companies in universities how we use them and how we now use CK just again to give you an idea what may be useful for you I think oh, I think it's reached sorry Yeah, so I started Last year I went to Seattle and I was talking to my colleagues in Microsoft and the University of Washington And we said oh, wow, but this cool stuff that now you have all those shared components. We can actually enable open science It's again a buzzword, but what do we mean now instead of even publishing your paper validating it and then like accepting it Why not to tell people that you share your CQ workflow workflow for a given task will be validating it will have a real life scoreboard Well, I don't know if you claim that you have performance analysis. We'll check it and there looks a paper and with those guys with Washington Cornell few other universities Cambridge we created this tournament last year at us plus and We said we took very simple task for image classification Everyone knows it and we said anyone can submit any solution for which classification can be software hardware model And just show us what will be your performance throughput accuracy and we'll put it on scoreboard And we'll accept you depending on the results and this was a quite good success because so we got very diverse Artifacts we had some cool stuff. We had submissions With someone was providing a cluster of 10 Raspberry Pi devices and showing that they can do the same speed of image classification as Tegra and Tex machines, so it was really cool on the other side. We had some submissions from Intel provided their submission on Using some powerful little server in AWS and Amazon cloud and just one one One month before they published a paper claiming that they got a record number of record throughput like 450 Images per second on this server And it was 50x speed up over traditional cafe Framework which they use and the validated it so we plotted all these graphs So again, you can go there and you will see in this link all this live data from from this tournament And what is interesting that this result from Intel we validated it, but it was not easy So when we started looking at their code and data was shared it wasn't with CK So it's not validating it. Oh, we're not getting 450 FPS. We're getting like 50 And this is how when you read the paper and you don't know all the data That's what happens to you and then you say, oh, no, I don't trust those guys. They're cheating It's really not nice, but then because it was validation started testing it. Ah, library was missing They use very specific into library with DNN and MKL, which you had to use it. We got it We got like 200 FPS. It's already better, but still not 450. We continue digging in. I don't realize and by the way Those guys were extremely Supportive and working together to kind of fix the problem And then we found the bug that they had this model where it was supposed to be in date It wasn't in date. It was still in p32. So we fixed it. I don't know what 450 FPS We fixed the workflow. We fixed all the dependencies. We shared it So in ACM digital library, you have a paper with those artifacts and what is even cooler that? One month later Amazon colleagues called us and said wow we saw this workflow and actually two minutes We did it in our Amazon cloud and it worked and it actually got to give us the result Furthermore, we ran it another workflow. We had FPGA submission on FPGA and also worked and we had a common Presentation a few months ago. You can see the details where it's kind of now when you publish paper It's very quick to reproduce results on a native environment. You can use Docker. It's fine But you can now actually use it with different Systems and it can adapt to your environment Which has two more minutes. I'm just finishing going through quickly general motors What they say they work on self-driving cars So they said oh they usually have to find the good solution. It's again public information. You can sit on YouTube So I'm not saying any secrets So like any other self-driving car, they need to find the best solution for model software and hardware Which would be let's say less than $100 You can't use like $1,000 cheap in your car because it'll be too expensive It has to budget on I don't know 10 watts of energy Has to be accurate so that you don't kill a pedestrian because there was set cases like this as you know So this is all like very complex task So they say oh but now we can actually use those workflows and we can run them on different software hardware And we can find on the Pareta the best solution. So this is again this helps them I'm a reproducibility vice chair for supercomputing. We are trying to see if we can automate Different submissions and we were doing this proof of concept I Say so application You can look at it how automated and how we can run on different supercomputers and okay, I'm finishing the last thing Because time is up and I have one minute as said well So I did the quantum competition quantum tournaments Just one week ago in Paris and those good turn up and you can see if you are dev room at quantum computing We're actually reproducing work with a BMW get the and we're reproducing some of the results and some we had some interesting new Submissions about improving machine learning techniques so you can see it online Time is up. I have last slide so Now I can say it's all magic. It's also not at all and There is it's like well only the very very very beginning of this journey to tell people that we can do reproducible research And I think that CK is just probably the first prototype I consider it still prototype which allowed us to connect all those different tools together what you are building and Allow researchers to actually take advantage of them without spending that time of understanding what you're doing So providing this like CK API with your tool and hopefully you can plug it in But there is a lot to be done and we're open to collaboration because we're looking at standardization now for API's We're improving like installation glues and so on providing more PIs With that Again, I said I'm open to collaboration contact me and thank you for your attention. I hope to be useful I'm sorry that I was speaking fast because I was like And and we have even time for questions actually go to 1040 Which is like exactly. Oh, yeah, sure Okay, so the framework you call Luigi. I don't know this framework. Unfortunately. So so the question was if we considered using to Luigi, right Again, I will look at it and which apparently can help you if you change small parts It can help you trace it and so on. So no we didn't so far what we do is When someone changes something since it's a common workflow someone else complains and send we had a case when we broke Something an arm told us like in a day like oh my god, there's something broken So it's an open project distributed project So they can be tools like this which we think can help make it more stable now So I will look at it. Thank you and but I don't know this tool yet One more question probably there is one more question. I don't know. Yeah, please. Oh, they are using production now so yeah, yeah, so oh Yeah, do we have any oops, sorry, I'm falling down. Do we have any experience using using those workflows in production? so they use in production so and as I said so I Don't know if I can officially start saying which companies are using it I will prefer not to for now even though you saw some of the which I kind of using yeah We're using it all the time production actually why you didn't hear about this frame for last few years because I was actually working a lot with companies Trying to make understand what they need and they're using it and now I kind of making it like public So but yeah, absolutely. It's my main thing is we're working with companies Okay, so thank you very much