 Okay, so here with us are Fabrizio and Marche Who are going to tell us about their python-driven company, so please give a warm round of applause Hello, hello everyone. I'm Marche. That's Fabrizio and here's the rest of a team from TBG The company that we're representing today. So it's not our companies. We just work for them but so what the pie-driven company is about so basically and We were challenged with quite a big problem at some point in our career We're working at TBG and the solution that we found was actually branded by us as a python-driven company so Before too long, sorry Okay, so before we actually start taking you through all of the challenges that we have to meet Let me actually tell you the story that we were fighting with so What was the challenge so we are quite a modest size team it's only five of us three developers it's here and those three guys and Basically three developers one QA and scrum master And then our management came to us and say hey guys by the way So would you be able to provide some tools for the managing management of data producing report helping to Do the statistical inference and help all the bunch of our analysts that know actually nothing about programming They they actually would like to have shiny UIs and there are quite a lot of them There are some people from finance is actually they would need your help as well There are people also even in HR that would be nice if you could help them as well So then we just looked at each other just just doing the simple counting one two three four five And then just realizing that it's actually a little bit too much for us By the way We started quite a lot of new features and new actually projects And we started to move to the area that was kind of unknown to the company So when it's come to specs so we had this incredible requirement this point number two and the second point was actually like We really don't know what the specs are so be prepared for everything Which is quite easy if you have five people and please deliver frequently naturally and of course quality first So when you deliver application that would be nice from management point of view to deliver the highest quality possible So we started to scratch our heads and saying like and should we actually look for a job? And actually should we actually take that challenge? so so now the big challenge so we actually started to think about how we could approach that and The first thing is actually we started to think about the basic foundation of working in any company because we realized that actually So there is no way that we'll be capable of delivering reach UI application that will help automate some of the task of the people That will actually make their life much easier because it's simply not enough of us And simply we don't have enough time and we won't be able to fulfill all the points. That's why I just outlined a few seconds ago So we ask ourselves. All right, so so what's the current? so we ask ourselves what's the current and Structure of a company and then we realize that actually the structure of companies is even though we use all the agile all the buzzwords of the Scrum inside the tech teams. We are pretty isolated units So basically if you think about it, you have a big guy in the middle and surrounded by small managers And then of course you have a teams themselves so basically if you want to automate or make a life automate some of the tasks of The guys that are doing the main work that are doing actually the heavy lifting we should go to them So we shouldn't actually rely on the usual communication channels. We should actually Think about the structure of a company like that So the first impression that you can have is actually so this is like one big mess So please imagine that each color represents people coming from the and different apartment and of course color in The size indicates Importance of a given person in a in a car key of the company So the tech team actually should be blended into the landscape of the company So we thought how about we actually go to people and talk with them and just ask them like all right What do you actually need right now? What's your problem? Like we don't want to know like what is like a high-level specs of actually, you know, like in very advanced Application that that maybe in a few months having the team of this size will be able of delivering So then we started to ask even more questions to ourselves the others In order to define our approach So the first question is like could we find a platform that would combine flexibility of Python Awesomeness of the web tools So why are we actually decided to ask a Python oriented questions because the first thing we realize no UI All right, we went to our management to say you want to have those tools? Let's forget about the UIs All right, no user interfaces not no graphical user interfaces So they say like all right So how can you interact with with with the platforms with the application? We say, you know, we can code and say like whoa But but wait a second like those people Know nothing about coding so like yeah, all right. So that's another challenge. We'll teach them so so basically We just realized how about we use ipython notebook where they can actually, you know Inside those really shiny cells put some of the snippets Share those snippets share the solution automate their work and we can by being blended into the company actually help them at the Every every stage of the development So actually we thought we will provide some high-level tools and they will provide more of them So in other words, we'll go with the mashup kind of approach and suddenly our development team actually Exploded to tens of people because it's not five of us anymore But if they will gain even like a slight knowledge of fighting then they can actually build some apps for themselves and Some of the questions that they were asking and some of the struggling that they were having was mainly data oriented So this is the world of Excel so Microsoft Excel is like ruling the world of any analysis and The biggest problem that they had with the big data They were already calling it big data is that that not capable of opening the exo files because that's just too big So they already arrived to the world of big data and we said like all right So the first solution that we deliver was called the three analysis interface So so here is just very primitive and simple snippet But actually this is all you need in order to fetch quite a lot like around half a gigabyte of data and of course We don't the user don't do not have to know basically What we are using at the back end whether it's mongo whether it's postgres So how we actually transform those data how we actually deliver them? They don't need to know all the primates. They just say like all right So I have this data store which is like name of my clients Let's say company a and they say fetch me data out that and of course they will fetch the data in the form of data Frame so it's already more of Excel World so so basically we said and when you have the data that you just Fetched using three lines of code you can either export them to Excel or Perform your analysis in the notebook itself So and they they actually interact with some of the services and And they ask us like is it possible actually to automate those So we deliver them a few solutions that enable them to just interact with the services Not only to fetch data and do the usual transformations But actually to to interact with the services to post some of the service in the server related Platforms to actually fetch the results to combine them and so on and at the end and of course as always We provided like a back and for storing and the data Okay, the biggest challenge that actually Fabrizio will tell you about was actually to teach them that oh so we provided the high level and Interfaces the programming interfaces so that they could just log into the software that we Exposed to them for hosting their notebooks. They could actually just go there, but what to type So you know, so in order actually to achieve that the first try was like maybe we will try this guy and Being crazy enough to teach them the Python programming or maybe we need someone with superpowers and that's Fabrizio So um, but that was totally not easy So maybe the story that I'm telling you right now is like it sounds trivial And actually when I was presenting those slides, I thought like oh, is it actually a good topic to present it during Europe? I don't because we are saying like yeah, we just decided to teach Programming to bunch of non-programmers, but presumably remember that they have no association whatsoever with anything Programming related like anything So all right So the thing that we had to actually keep in mind all the time was wise and how so in other words It's not only to show them the snippets and show like hey, this would be super useful You need to reinforce this idea that this would be actually useful and helpful all the time every single small step like hey This is how you can automatically take all get your all of your data do the analysis And that's faster than than any other solution that you know then for instance You can use this in this library to draw stuff to actually plot some Things then you can use this and that to actually produce your reports So by simply just introducing those those Elements of their usual pipeline. We were able to enforce those house and wise at the same time And the other problem was and realized that we realized that we got it for free Is that the usual question that we're getting from them? Can you say that in plain English? So in other words like they were coming with some problem They were like outlining the problem like using their hand waving and just high-level terms We were giving them response Answer to that and they were saying oh, could you say that in plain English actually because I didn't get that So then we thought how if we would teach them Python Python is a language. We know it's programming language But let's stick only to the language as a buzzword Let's use it. Let's actually treat Python as a foreign language So if you have a problem draft your problem in the notebook and use the language that we really understand and Then we'll deliver your solution much faster And of course dealing with the big data as I already mentioned Let's use Python again and If you combine their knowledge if you actually introduce the multitasking of the Python that we are all aware of they are just starting to Just discover totally different different world Okay, so naturally there were like a lot of people being like Incredible non-believers while lazy didn't want to actually engage themselves into something new and they will keep challenging us with some problems So for instance like yeah, you know, there's this Notebook and they were using this special tonal voice while asking those questions. So hey, can I pull my data using notebook? Yes use request or whatever like is it possible to scrape websites? So of course they were always Waiting for the snow saying like, okay, our system have limitations. So but They never ask actually the right question. So naturally is it possible to scrape website natural request scraping whatever Can I pivot my data pandas? Can I create plots? Yeah, there is like a whole bunch of Solutions for that kind of work on millions of rows, which is till now like totally not possible so or You could actually use like, you know and the very nice solution of splitting your data into 10 Excel files And then trying to do a big data like my produce and yes pandas again Okay, but I really need to work with Excel because like you need to have Excel Otherwise, you won't be able to survive in the jungle of data So read docs of pandas, you know All right, what was the result? This is a walk in progress. We are still Struggling and fighting and trying to enforce our way of thinking the results is far from utopian, but I think that Lots of people actually learn something new lots of people started to automate their work lots of people started to actually gain and and do something in the Free time this time that we free up by just introducing the solutions and The main question was like, why did we do that? So we have like very huge open space and I'm actually sitting there and observing people and We are like making jokes like the tech team. We are quite happy guys and then I observe in those analysts and Default that I showed you of this crazy Brazil fan so basically this is kind of the faces that doing when I strike them with the big data or whatever and And you can clearly see that those people are not enjoying their day So we thought like we need to give them something that will enable them to express themselves Some some expressiveness and the programming languages are just is actually the tools of the salt If I will give you any kind of problem You will start to actually putting the keywords and the grammar of Python all together and give me the solution They are the slaves of the UIs which are actually just enslaving them and and giving them No, sort of expressiveness. It's about actually, you know the scenarios the clickability scenarios That's what I actually achieving with the expressiveness comes creativity. So I was thinking like I'm a software engineer. I don't know what kind of application you need But if I will give you the expressiveness and and free up some time of you So that you can actually use your creativity with those new tools You will come up with really awesome applications because you know what you need in your industry and And of course at the end of the day, the most important is freedom So I think that those people's the ones that actually leverage this knowledge started to be much much more productive and Hopefully happier. So let's welcome Fabrice that will actually take you through the biggest challenge ever taken in the UK Which is actually training non-programmers to program in Python. So, hello everyone So basically when we submitted the the abstract We were asked to To have a separate section about the educational aspects of this journey that we had So these are my slides about it. You can find them here It's it's it's open and public and See a little bit about me It's a little bit so Don't have much time So why this part of the presentation the short answer is that we they asked us and the shorter answer is we hope it's interesting basically Some people teach Python some people train other people in Python Some people have colleagues that they go. How do you do this? How do you do that? And so I'm gonna I'm gonna talk about a set of guidelines that that hopefully will inspire you in your In your work, whatever you do What we are going to talk about what the trainee should do what the trainer should do and basically about My experience in in Manila have been in Manila two weeks in March to train 20 people It was a very nice experience. I asked to have four groups of five people each as much differentiated as possible Because I didn't want to have like one group of super smart people and one group of less smart people I want them I wanted them to be able to help each other So wait, wait, wait These are the guys. These are my My Manila my Manila guys amazing people So what a trainee should do these two things listen with the capital L Which means you have a trainer at your disposal So squeeze him as much as you can if you are ever going to take training in anything Get everything you can get the knowledge the experience what they think why they think the way they think How do they get the solution to the problems get try to get everything from them use that opportunity and Work hard of course What the trainer should do I think he should have to achieve excellent in the following First of all smile and be patient because if you don't if you don't do that you will put a wall between yourself and the people you're trying to teach something to and They won't be open anymore And you need them to be open if you wish you want anything to pass to pass that to them Never take anything for granted So you can go and teach non-programmers and start Yeah So we're gonna write a function and then a method and then we're gonna use the class and then we're gonna iterate over this stuff And they go like what? Because they absolutely have no idea what the function is what a method is so you have to give them the concept the lingo All the words that we as coders use every day and we were not maybe even aware that this is a special language That we know and we use but they don't know and they don't use so the ideas Why do we need to do what we need to do and the reasons the reasons behind behind? All the techniques that we use like code reuse for example, why why is important? Use the GPS technique. This is this is something that I always say so you're trying to bring someone from point A to point B through a Bunch of roads and at some point you can see by their faces that you you've lost them Because there's some concept that they they are missing so they can't go from here to here in this case It's it's you you have to find another way to explain To them so ask what is not clear choose different words Provide other examples do not repeat the same stuff in the same way because they're not going to go through that through that road Set some goals they should be reachable Start gently so they they're not scared off at the beginning and then gradually increase difficulty and Skip the boring trivial stuff is possible So they don't start wandering off cell phones and stuff and Deliver outstanding quality material you have to put your best effort into it when you prepare the materials The first reason you have to enjoy it as well needs to be a pleasant experience for you as well Give them only what what they need to know Because we're all pressed in time especially For me it was the London people Manila people they're very open they're very hard-working people London people are a bit more more difficult Because they have at least in our company They have a lot of things to do and so if they get the feeling that you are telling them something they don't need They get they get I'm not saying upset, but it's just we don't we don't need this We don't have time for this or just give me what I need so just give them what they need focus focus on their needs Which means if at some point you get you get to you get to a point where you have to explain something again do that instead of trying to Stick to your original to your original plan and refer to the real world Something easy to relate to So this is this is just one example when I when I was explaining how to handle files It's it's all very you you can't just go and say yes You have a handler to a file a pointer here and there You have to open it do something with the file and close it they won't remember it So if you give them the example and you say like a file is like a fridge They all know how to use a fridge So you open the fridge you put something in and you close the fridge And they already know how to handle a file because they have the association with something They've been using since they were little kids. This is very important because they will not forget that and Adapt so adapt speed difficulty and so have extra materials ready if you have a group of People that are actually fast and know what you can cut if you have a group of people that are less fast and entertain them Communicate your passion make them laugh and have a good time. Don't be too strict if they make mistake use them those mistakes To explain things again instead of saying oh you made a mistake And the next one is only for the brave So if you have to take a slide with you home today, just take this one Flash your ego down the toilet This is because you're doing it for them not for you and if you do it for them, it's just gonna get so much better The tools that I used well, Wunto And then a bash shell I Python console and notebook libraries pandas noom pie and and the analytical interface that we that we wrote my paint with the with the cheap Wacom tablet that is that is really really useful because if I have to explain what the function is I just go crazy to someone who is not a coder But if you can draw a box with a with an arrow that goes in with an arrow that goes out And you see it's about you know a bunch of lines of code that take some input and produce some output they all go oh, okay and Skype or googling out when you have to to to go remote and This is an overview of what is it possible to deliver in about 12 hours? So we had an introduction section session for Python I Python Sessions three data oriented session and one QA session at the end. So the introduction session is Giving them all the points all the concepts So they have to they have to understand at least high-level What's looping branching and so on and so forth and it's really important because it provides Content context, so they don't get lost when you when you then start to explain This stuff debate the basics of of Python So code reuse and functions the looping and branching handling files Python data structures the main data structures and the main built-in functions extra materials that I delivered So advanced stuff with dictionaries function arguments list comprehension slicing and the broader introduction to built-ins and The data-oriented sections basically when you work with data you have to do three things you fetch data You work with with it you clean it you mangle it and then you provide some sort of output that can be text statistics or You know those pretty graphs that everybody likes And you have to use of course Jason because stuff that comes we work we work with Twitter and Facebook So we've got a lot of Jason to to work with a reg some regular expression some string manipulation And of course the lovely daytime objects, especially with time zones. They are nice So that's it. I hope this was Interesting for you. You feel free to contact us and before before we close an open question I have a question. I have a favor to ask to all of you There's a there's a there's a lady that I love to beat and she's coming out of Surgery in this very moment. So if we if we could have a round of applause to wish her good luck I would really appreciate that Yeah, a few a few minutes for for questions now Please remember to use the microphones Thanks for the talk So we have a similar problem in my current company, but what would they ask for us normally at just queries basically just How many of these this month? How many of that? So maybe for us it would be even simpler. We could just somehow make it simple to make queries You have any suggestions how we can do that in a way that they don't allow and not allow to make some crazy queries that killed the database or deleted all the records Yes, this is exactly what we've done with the with the D3 analytical interface basically we provide an interface to them and This this interface limits their what what they can do with the database basically shields the database and It has two two basic Advantages they can query the data regardless of where the data is stored or how the data is stored and And we don't have the problem of someone saying oh, you know what I deleted everything So because because they can't there's no delete everything through that interface How do you? Catch up with the questions which comes afterwards after you leave Manila for example Yeah Basically, I didn't after I left for Manila Some people were still asking how do you do this? How do you do that? And of course they always make time If my boss is happy, I I do a remote session if for any reason I'm very pressed I do a bit of overtime maybe an hour or so and Because I'm always happy to do this. I really really love Teaching Python or whatever. So it's it's always a pleasure to help someone Especially when when they are eager to learn. I just can't say no So so yeah, and there is also one more thing because please remember that as soon as you introduce notebooks This is the communication actually Item that is used so whenever there is a bug or the problem. They just share a notebook with you It's not high-level email sent to you like hey by the way, this is the problem They just give you a notebook. You just run it and say alright. This is a bug and It's very simple to actually, you know That's why we said that that python would be used as a foreign language for us Even though we are all foreign as using English English is not enough to actually Discuss their problems. We need to go Into more formal one Hi, thanks of all first for the thanks for the talk you nearly answered my question Could you have done it without ipython notebooks? I think that we actually were at the You know, it was a perfect time to introduce kind of solution because I think that if we would introduce them like a console based You know a solution which was like the bear bones python with even without ipython, you know Like it's showing or giving all the nice colors in the console We will never actually be able to introduce or even push this idea and a third I put a notebook gives this enough UI that you need You can plot things there and this is mainly what they do They want to get data. They want to actually perform some stuff on the data The other thing was that they all mostly work on the laptops and the thing is that the Amount of data that I dealing with are just too big for the laptops to handle So what we did we just bought the super powerful server and just hosted the notebooks there So since they are working in different time zones, basically they could have like 32 gigabytes of RAM of 40 or 64 gigabytes of RAM just like that and just perform and since they were not, you know Building the graphical representation of the Excel file They could actually have handle even more and more data with that and then at the end it was synced with their laptops So so when they were done, they could actually just save their day Results and even carry on on their laptops with much smaller already cleaned up data or plots You said that you were using Ubuntu for the trainings What did you do afterwards if some people were using Windows or some other operating systems? You said that you have a server with no books. So do all the people use just that server or do they have local environments? So they were using their own computers with Windows unfortunately, but everybody uses Windows and The place where we host this technology is is on Debian, right? so the notebooks are on Debian and I use Ubuntu so If you want to be a trainer Ubuntu gives you all the tools that you need You can do anything you want and the beauty of the I Python notebook Technologies that it's not really important what kind of system you're using because basically you do everything in the browser So as long as you have Firefox or Chrome and you don't use the other one Can't even pronounce it So yeah, if you if you are on on Windows though There is a small problem when uploading stuff with Chrome So if that happens don't go crazy debugging just use Firefox and it will work Yes, but how do you deal with people that like want to install packages on their laptops or something like that? Like especially if you do data analysis NumPy and this stuff Can get kind of hard and explaining them like package managers and everything could be hard So do they just not do that and use the server or do they actually want to use Python locally. Oh Basically, they didn't have to do anything on their on their laptops everything was provided on on the server side So where we host the notebooks we have installed all the libraries So the three analytical interface NumPy, SciPy, Macplotlib, Pandas Everything they need is there installed for ready to use so they just open their notebooks We have an authentication system for for each analyst and they go in and they do from blah import And and it just works for them. So they have a slow as low window box or Super nice Ubuntu machine doesn't doesn't really matter because the heavy lifting is all done By us in the server that much was talking about before Are there any more questions? So I understand that the data analysts are quite happy. You were also talking about the human resources people and the accounting department which have quite different needs like Payroll which Was it handle also did you? so if you think about it like one one project that James developed for Finance team was basically, you know, they had this super annoying Thing that they were doing Manu Ali, which was just combining Excel files that were delivered by the sales guys And basically they had to build some repos on top of that So sounds trivial, but for them is just a manual walk. There is no other alternative. It's just wasted Wasted time during that during doing quiz that person will go Irritated and then just one of us in this case James just provided solution for them So so basically we gave them the tools to talk with the data to actually get the data But at the same time we are listening for some new stuff that we can actually deliver on top of that So we just basically delivering them like one magical black box, you know Function that they could actually call like a create report and of course under the hood lots of stuff was were happening But but since we were free up from delivering QI because nobody required that anymore We could actually just focus on on the business value on the business Algorithm that was actually delivering the solution Please use the microphones The use of Excel several such programs decrease after your 20 So basically it didn't decrease because the thing is that they were not able in some cases to use it at all like, you know Just imagine They had this very nice Adjustment mechanism so they were opening an Excel file and if the person was smoking That person knew that it means around 30 minutes to open that file. So that was a break time Basically, so they just stopped opening those so maybe less breaks. Maybe that's better. Actually, I don't know But in Manila actually change things changed a bit there They're starting to introduce I Python they they introduced the I Python notebook thing step-by-step in the process So they were doing a bit and then switch into Excel and do the rest and then another bit and then switch into Excel until it was all I think the most important is that even if you train them Even if you show them all the solutions, you still need to wait for this discovery moment when they actually send you to see me Like whoa, they actually realize that this is powerful stuff Now they will I think alright some keywords something is flying around but then they actually automated a pipeline You know and they actually realize that they can do something creative and actually contribute to something interesting rather than you know Just just clicking through their life Okay, thank you very much much a and Fabrizio