 Hola, hola, buenas tardes, ya funciona el micrófono. So I'll switch to English now. So welcome. So my name is Israel, like the country. But I don't name after a country. The country is named after me. No, I'm not kidding. I'm going to talk today about machine learning. But this is not going to be the typical machine learning talk. So I'm going to talk about how to have an impact in the world with machine learning. You have to do much more than just learning from data. So when we talk about machine learning, a lot of people think of the tragedy of machine learning and artificial intelligence. We will give birth to machines that will be so intelligent that will enslave humanity in the future and will do our jobs for ourselves. And then we will be out of jobs and so on. But to be honest, for this terminator point of view of machine learning, to be honest, I think we have already actually created autonomous killing machines. So I actually want to ask you something. So do you believe the same? So when do you think that we have created as humanity the first autonomous killing machine? So I have here three different dates. Who thinks it was like option A, 1997. Can you raise your hands? And the B, 1982. Well, a lot of people. So maybe you have already seen this talk, maybe. And option C, anyone? So 1997 is actually Judgement Day in Terminator 2. And then 2017 is Judgement Day in Terminator something. It's another movie that wasn't successful. But it was actually in 1982. It was in 1982 when we as humanity created a machine that killed people in an autonomous way and independent way. This machine wasn't created to kill people, but it actually ended up killing people. So I'm talking about the Stedak 25. This is a health industry device. It's a machine that gives radiation to try to cure some types of cancer. It was manufactured in 1982. It was put into production in 1983. And ended up killing some people. So if you want to know more about this specific machine, I'm not going to talk about the machine itself a lot. So there are those links. You can actually search for it in Google. And you will find a lot of information. So this radiotherapy system killed four people between 1985 and 1987. So it was four people, not one, not two, not three. But four people killed by a machine and actually injured more people before the administrators of this machine decided to stop the machine and look into the situation to try to find out why this machine wasn't working as intended. It wasn't a human error in the sense that it wasn't that the operators of the machine decided to give a lethal dosage of radiation to these patients. It was the machine itself not actually fulfilling the orders that it was given to perform its tax. So then why the machine decided to kill these four people, not obeying these orders that it was given? It was actually because of software engineering. Rather because of lack of software engineering. So this machine has a software that controls the machines, a firmware. So when this case was investigated in detail, the investigation team found lots of issues. So take into account that this is a machine in a hospital. So it's a health department. In this case it was Canada. This is a very serious issue. So the investigation was a very thorough process and it was really a big deal at the time. So they conducted a very detailed investigation and they didn't find any problem with the hardware of the machine. But they found several problems with the software of the machine. They found that when the developers of the firmware of this machine were working together in a team, for instance, they were not doing code reviews. Every developer, when they wrote her own code and committed into the common repository that were using for source code, did so without any approval or opinion by any other member of the team. When they were writing software, they were actually not conducting any kind of unitary testing. So they were not testing the code at all. So when they wrote a new piece of code, a new feature, and they put that into the firmware into the code of the firmware of the machine, they didn't test it. They were, so it was a company that had several devices, several, it was a health industry company with lots of different machines, with common parts for different parts of the firmware because there wasn't a lot of shared hardware between different kinds of machines. And the team was copy pasting a lot of code from other projects that they didn't really understood what the code was doing. They were not really aware of all the implications of that code, but they had like a basic knowledge or a basic intuition of what the code could be doing and it was useful for the post-ports that they needed in their code. And so they copy pasted to try to solve the problem in a quick way. And actually the investigation, when it tried to reproduce the situations that led to the death of all these four people, the development team and the hardware team of this company was not able to reproduce the situations that actually led to the killing of people. They were not able to reproduce any situation when you gave orders to the machine and the machine was actually doing something very different to that. It was very difficult corner cases, so it was with the knowledge that they have of the machine, the creators of the machine with the knowledge that they had, they were not even able to reproduce these situations. So when I think of all these things, it sounds very familiar to me. So I'm gonna give you some hints. So I work in now as a cloud engineer at Google, but I consider myself a data scientist. I've been always involved in data science years ago. And when I see the software project with all these features, it's data science. So in data science you have things that don't really follow rigorous software process that they don't do testing and so on. They just focus on getting new code written. So to me, this is data science. So if we compare data science with software engineering, so data scientists would never call itself herself a software developer or a software engineer. I know lots of data scientists that they actually defend that they are not software developers. I disagree with that. So data scientists write software and therefore they are software developers. Of a very special specific kind, but software developers nevertheless. So the problem is that with data scientists we have not been educated in software engineering. So many of us are learned to program by ourselves in a self-taught manner and we don't know, let's say, the tools of the trade of people who do professional software development. But software engineering is very important for data scientists because the role of a data scientist in a company or the aim of a data scientist in a company is to have an impact on the business. So when you are hired by a company that have lots of data, what they want to have is to try to leverage that data to obtain some kind of advantage in the market to know better the market, to know better the customers so they can target better the offers. So you need to have this impact in the world. And if you are not able to industrialize because you're not following the best practices to do a software project, this is very difficult. And if you don't put something in production, in coordination with the rest of the systems on the company, in coordination with all the policies and common situations in your company, if you are not able to do that, you will not have an impact in the business. And then your aim as data scientist is not fulfilled. But data scientists should not be blamed for this. It's actually a problem of tools. So we are in need for better tools in order to be able to do our work. If we focus in the case of software development, so software engineering is now even an academic discipline in the past life. I was a professor at the university doing research on software engineering. It was very fun. So, but it didn't start as an academic discipline. It actually started in industry when people, the first people who were doing software realized that it was a very complex activity and that it led to many different kind of situations to many kinds of incidents and problems. There is a famous video by Dijkstra talking about the Apollo 11. I think it's the program that they discovered back like a couple of days before the launch of the rocket. And when he asked, oh, so how did you discover the back? So what kind of testing strategy did you use to discover that? The answer is none. They discovered it just by pure luck. So the situation is similar in data science nowadays. But the good news is that software engineering is like the realization of all these experiences in the form of a body of knowledge that we can now try to apply to a new situation that is a data science and to avoid committing the same mistakes that the software developers have committed historically in the field. So notebooks. The title of this talk was notebooks are not enough. What is a notebook? So I'm assuming that most of you are familiar with what is a notebook. We're not talking about the laptop. I'm not talking about the paper notebook. I'm talking about a specific tool that is one of the favorites of data scientists. Here in the picture, we have a Jupyter, very well-known tool. You probably are familiar with it. A notebook allows you to write code in combination with documentation, in combination with images. It's a very nice tool. It has very nice capabilities. And you can really create beautiful code and beautiful notebooks with these tools and share it with other people. And in fact, Jupyter is not the only tool that provides these kinds of interfaces. They are now becoming kind of popular. For instance, here are another example of another tool, which is Apache Zeppelin. You see there are some code with some graphs and the graphs in this case are interactive. They are a very nice tool. Because they are so nice, notebooks are very popular between scientists in general and data scientists in particular. So the authors of Jupyter, when they started the project, the use case that they had in mind, the problem they wanted to solve, is sharing experiments in science. So there had been lots of cases where people tried to reproduce the studies of others, like previous studies, previous papers, and they were unable to do so because of the complexity of all the dependencies between software, the separation of software and data. And when they tried to reproduce, they were unable to do it. So thinking of that particular use case, Fernando Perez started to work on this tool and now it has become very popular between data scientists precisely because of that. Sharing an experiment written in front of code, documentation, graphs inside the notebook, it's straightforward. It's just, you send the file and that's it. And if the notebook contains some data inside, like in forms of variables, a data frame, a table, whatever, it will be included in the notebook too. And if you have some external data, you just bundle everything together and share it. It's really straightforward to share. But this tool that was created with science in mind to be able to share things very easily doesn't work very well with software engineering tools. It wasn't thought to be working with this kind of tools. So like testing code reviews, conversion control, all these kind of tools don't work well with notebooks. We have some workarounds for some situations, but those workarounds are not enough. And this is a big problem because this lack of integration within engineering tools is encouraging bad habits and bad processes. And bad habits and bad processes are a big problem in data science. So when you don't do code reviews, when you don't do testing, when you copy paste code from another notebook in due to your notebook because you cannot import easily code from a notebook because it's not a package, it's not a library, when you do this, you may end up creating an autonomous killing machine that will kill people by itself when it decides so. And it will be very difficult to control. So specifically, what are these bad habits and bad processes that notebooks encourage? In a notebook, you have a lot of hidden state. So when you are writing a Python script, think of a Python script in a text file, when you are writing a program in a text form, which is, let's say, the traditional way of writing code, in order for a line to be executed, the previous line needs to finish. This seems kind of obvious. You write the program and the program is executing line after line after line after line. In my experience as a professor, one of the courses I was teaching was introduction to programming for engineers that were not computer science engineers, civil engineers and other kind of engineers. This simple thing, it was the first problem that beginners, people who were starting to learn to program, this was the first problem that they had trying to understand how a program works, understanding that the program is not fulfilled all at once with all the lines defined in different conditions and they are trying to be fulfilled at the same time all at once. It's just that you need to execute one line and wait until the previous line has finished to execute the next one and so on and so on and so on. In notebooks, when you have cells that can be executed in many different orders so there is research about this topic, this is actually highly confusing for beginners. Considering that data scientists are sometimes, even learning to program at the same time that they are learning to do data science, I think this is kind of a very dangerous situation. Other but having some processes that the notebooks encourage, when you cannot integrate a notebook with a version control system, you will end up with this very well known problem of having files in many different versions so these tools will actually generate files with these kind of names by default. So it's very tempting to use this really silly way and a poor way of managing versions as your version control system instead of using proper version control. You cannot do any kind of a libraries management so I give you a notebook, how do you import it in another piece of code? You cannot do that. You can actually import notebooks. There is some evil plugins for Jupyter that allows you to do those kind of things and again, that's also quite dangerous. And testing for instance, can you test code in a notebook? You cannot write unitary testings for each one of the cells that you have in a notebook. That's simply not possible. And for code reviews, because notebooks are not integrated with a version control system, you cannot do a code review of a modification, a small chunk of code to review what it's doing and approving that before it's merged into the code base. So notebooks are actually quite dangerous because of all these. And in data science, because we are using notebooks, we are facing many challenges that the software engineering community in general faced decades ago. We're talking about the 60s and the 17s and that are normally overlooked. So here's a picture of a friend, Ahamed Hassan, research in Canada. This slide is actually copied from one of his presentations. It's actually, if you look at the picture, it's exactly the same thing that is in the picture. You have the link to Twitter where this picture is. So there is no common accepted development process for data science to do software in data science. And this can lead to dangerous situations like producing models that don't perform well in the real world. So evaluating whether a model is performing well. So we have, of course, metrics to evaluate the performance of a model, comparing with some data. But things like, for instance, values and furnace, these are not so easy to test. So a model will easily reproduce the same unfairness and values that is introduced in the data. And this cannot be easily evaluated just with the typical metrics that you have in a machine learning model. When people do machine learning, they try to focus on getting the best model for a particular situation. They try to get the best classifier or the best regressor. So they focus on performance. And in order to be able to understand what's going on in a difficult situation, like a machine killing people by itself, you need to be able to explain what the machine is trying to do. So its planability, it's better than performance in data science. And the things that are doing data science, they normally overlook everything that is related to operation and maintenance. This is not taken into account from the beginning of the project. And this becomes a problem later on. So data scientists are a very specific kind of people. So I've been very happy working in industry. When I discovered that I could do data science, I have to confess that I didn't even know what it was at the time when I was first hired as a data scientist. But I was very well treated in my first position as a data scientist. And industry is treating data scientists very well because they consider that these kind of people are very valuable for business. So one way to be able to make data scientists happy is to try to give them autonomy on the way they work. So when you are hiring chess players, you don't want them to become chess pieces. You want them to be chess players. So companies, when they are hiring data scientists, they try to organize them in small teams with autonomy on how they work, on ownership on the tools that they use, the processes that they use, and so on. For instance, this is mapped very well to agile methodologies in cases where companies are using agile methodologies. This way of organizing data science teams integrates very well with the rest of the methodology. But this has a problem. When different teams of data scientists are working in isolated teams that don't talk to between them very much because each team has its own goal which is achieving whatever they're trying to do. So that's the main goal and collaborating with other teams becomes like a second-class aim. So each team starts using their own practices which are not similar to the practices that other teams are using. So the projects start to be inconsistent between them and the promotion to production is very painful. So normally in a company, data scientists working in these kinds of things are separated from the engineering part. So people who need to put into production the things that data scientists do are not data scientists. They are different kinds of people and they will talk to the data scientists once data scientists have finished their first prototype. They will inherit this prototype and now data scientists are done and it's the problem of the engineers to be able to put this into production. When those projects are not always using the same practices, when each project is a new thing and the engineering team has to start from scratch, has to start over again, then putting that into production is a very sensitive process and it takes a lot of time and many times projects die at that phase just because it requires a lot of effort to try to understand that complex prototype in order to be able to put it into production. And also the sharing of knowledge and the collaboration and sharing of experiences becomes more difficult because people are not actually talking in the same language, even between teams of data scientists. And this is very important in order to do this because it's a very complex activity and requires a lot of studying and sharing knowledge and learning on the go and life of learning while you are working and without collaboration that becomes just more difficult. So how can we fix this problem? So here I have written seven, not ten commandments of how you should do software in data science. And this seven actually, this is nothing new. This has been standard practices in software development communities for decades now. So you should be using version control. A lot of the data science teams fulfill this and you should be using also continuous integration. Every time there's a new change into the repository, you should be testing automatically that, you should be checking that code automatically. That's not so... Not found so commonly in projects. Code should be reviewed, period. You cannot put code into a repository without having someone else having a look at your code. That's a very dangerous practice. And if you don't do that, then you're just living on the edge and you will have a problem sooner and later. If you got new code, you should write the corresponding test. Otherwise, your job is not done. When you find a problem, or when someone requests a new feature or someone has a new idea, you should record that in order to be able to follow up later on, in order to be able to have an historical look about what were the most common problems, in order to be able to write a roadmap for the future of your system. So you should have an issue trying to fulfill all these needs. And it should be connected to your repository. So when you go to a bug report, when you go to a feature, you should be able to recover exactly the code that is related to that bug or to that new feature. And you should agree on how you are gonna write code. You cannot let every person, these geniuses, data scientists, test players to do things in the way they think are better. You should agree on something, whatever, but you agree and everyone should be using these same practices. And this should be checked and tested automatically with every new change in the code. So in summary, data science needs a software process. With the adjusted gravity is a typical software process. And this is a software process that is focused on achieving software quality, on trying to achieve zero problems, zero defects once the code is released. Of course, zero defects in software is impossible. It's like an ideal goal. It's all software has bugs that we know that. But focusing on trying to reduce this number of problems that are found once the software is released will minimize situations like an autonomous machine killing people and it will ensure or it will foster the quality of your code. So you may think that this is really like a pain. It's like any data scientist, I don't really need this. I do code that is so good, that works, that does it. But when you don't do this, you are not doing good science, period. So this is not something that I say. It's something that was said sometime ago by Francois Chollet in Twitter. So he's the author of Keras. When you have code that contains bugs and you do nothing about that, then you are doing bad science. Because source code is the core of any scientific project, including data science. Not your very sophisticated methodologies and machine learning methods. It's the code, it's the main artifact in your project and you should look after it. So in summary, what I'm trying to say when you are doing science, doing science is very fun. So imagine yourself in these nice mountains trying to go around hiking. So you do exercise, you breathe fresh air. So it's very nice. And then you discover how to reach that city through a way that you find in the mountain and how to go through the next mountain, through another way and so on. But now imagine that you have to do that with 1,000 people. So you have to move 1,000 people from point A to point B in these mountains. And you have to do that through the ways that you have discovered using science. Very soon you will have people falling off a cliff and killing themselves in accidents. You will have twisted ankles. You have all kinds of problems. How can you solve this problem? We think you need it. So not only you will have accidents, you will be wasting a lot of resources because people will take a lot of time to reach from point A to B. And we know how to solve these problems. We have invented beaches for that. So here I'm showing two different pictures. People may understand that science and engineering are different things. But they are actually in data science in particular. And in machine learning, they are the same thing. You cannot understand science without engineering. You cannot do engineering without science in machine learning. They are the same thing. In machine learning, you don't do machine learning with pen and paper. You do machine learning with data, code, and a computer. And therefore, in this specific case, machine learning science and machine learning engineering are the same thing. So we are almost finishing. So some of you may be reacting to this talk, like reacting to this idea. It's like, no, no, no, no. So data science, machine learning is a very special thing. That is not just a process that you can follow to do machine learning, to do creative work. Because this is just art. It is not. It is a very complex activity that requires intelligence, dedication, resources, and so on. And probably the process that you need in order to do this in the best way, it's different to the process that has been used for genetic software in the past. But there is a process. OK, so thanks all for your attention. I'm almost done. You can wake up now for the next talk. So I started this talk talking about killing machines, about how when I talk this with my family, my political family, about artificial intelligence, all that we are doing, machine will really enslave us, and so on. It is not true. It is not true. We are not failing because of having utterly intelligent machines that are making us redundant. We are actually failing by having completely stupid computers doing everything in our world. So software has eaten the world, and we have to do it better. So the academic community is already looking into this topic. If you want to know more, I recommend you to read this paper. It's a very short position paper. There is not a lot of research on the topic yet, but they are starting to look into it. So this paper appeared one month ago in the IEEE software. It's an academic half-industry journal of the IEEE organization. And the people who wrote it are very well-known researchers in the software engineering community. So they also know about machine learning, some of them. So thanks all for your attention. One more minute, and we finish. So remember, so tonight, when you get home, and you have to talk to your wife, sister, family, and they ask you about this talk, so these are the points you have to tell them. So remember these points, the rest you can forget. So data scientists are software writers, period developers, very specific developers, very special, but software developers, because they write software. Without software, they cannot do their job. So they are developers. Full stop. Notebooks are a very nice tool. Really, really, you can do very beautiful things. I like a lot using notebooks for things like sharing documents with my students after class and during class and so on. They are a very good tool for some purposes. But then we can, an evil tool when you start writing products in the form of notebooks and then you give your prototype to the engineering team in the form of notebooks. If you are doing that, you are being evil, period. You need a process to do data science. It doesn't matter that it's a creative activity. It is. You need some creativity and intelligence in order to do machine learning and data science. But that doesn't mean that you cannot follow a rigorous process in order to ensure quality. So in definitively, what we are talking about is how about this is the right tool for the right purpose. Machine learning is science and engineering. And for this purpose, we need to use the right tools, which is, in this case, software engineering. Because a stupid software can kill, too. Not only terminators. Terminators were highly advanced machine learning systems. But we are being started to be killed by machines, not because they are smart, but because they are stupid. And of course, this is not an easy endeavor. So we don't know how to do this yet. But the academic community is starting to look into it. And you have a lot of well-known and learned past in the form of software engineering. So you can start by using that in order to write stupid killing machines. So thanks for your attention. So if you have questions, now it's time for that. Thanks. One question there. Thank you. Do you think it's possible to have two roles? So a data scientist who does the research and provides a prototype written not in NodeWood, but in Python, for example, with clear inputs and reproducible outputs, and machine learning engineer who implements the production code? Yes, for sure. So that's totally possible. So it's very difficult to find all these skills in just one person. So it's very well known in the data science community that the best data scientist is a team. So you don't have to have all these skills in the same person. So you don't need to have the engineering and the science skills in just one person. But you need to have that within the same team. So working together. So things like operation and maintenance are like first-class citizens of your project and are taken into account from the beginning, from the zero of your project. But for sure, so you can have different specializations inside the team. Yes. Thank you. Yeah, hi. Hello. So when you develop a machine learning software, you're not only interested in knowing that it will work in the way that normal software works, but you're also interested in knowing that the performance is going to be good. So how do you test that? Because if you do that in a naive way, you can have tests that last for hours because you have to train the model. So it's indeed very difficult to test. I don't have short answers to give you. But it's something that you can test for sure. So if you have a control data set that you always use for the same kind of test, you can do things like that. So you cannot do probably in a continuous way for every change that you do in a program for sure, that it's just unfeasible just because of the time it requires to train a model and so on. But it's something that can be done with some periodicity. So I haven't talked at all about that situation. So that kind of tests in software engineering are called integration tests. When you test the system as a whole to see if it keeps working as expected. You are not just testing small chunks of code just to make sure that that piece of code works. You are testing everything together. So that can be done. It cannot be done in a continuous way, or probably not with the resources that most of the people have available. But it can be done. So you could do nightly tests or weekly tests and things like those and making sure that with a closed data set, you are producing metrics that are within your expectations for the model and that the changes have not affected those metrics. OK, no more questions? Well, thanks all for your time.