 Okay, everyone give Mike Miller a big hand. He's gonna talk to us about conda Thank you very much for the great introduction. I Would like to talk about conda easy installs and simple builds just a real quick race of fans Who of you knows what conda is? Third and who is actually using anac are using conda Just about 20% thank you very much conda is Quite a few things actually it's an installer similar to PIP, but I think a bit better It's an environment manager similar to a virtual and but still ever a bit better. It's cross-platform. There works on Windows Linux and Mac It's not limited to Python so you can use it for different languages Actually, there are people using for Ruby and follow languages And actually it comes from the scientific community, but I think it can be useful for every Python programmer So it's BC licensed and typically you install it you either install me conda or anaconda I would talk about this different condas here So there's what's called mini conda, and it's a kind of a small bootstrap like version includes Python itself conda and a bunch of dependencies like PIP and wheel and some other tools and It provides access to many hundred libraries so you can install a lot of libraries with conda, maybe even thousands And you just need to say conda install I will talk about this process a little bit and there's something called anaconda which is Different thing. It's a distribution similar to like if you like a Linux distribution so you get Python packaged up with a bunch of scientific applications and This includes of course Python conda conda install and it's about more or less like scientific 200 scientific packages could pop off here So supposed to be there but scientific package about 200 or so and if you install it it takes about two gigs of your hard drive But then I think a lot of scientists need a lot of these tools And you don't have to fiddle around installing the old separately They all come in one thing and then the value pay a little bit price was some this space But this space is cheap and programmers time is expensive And there's something called channel. So when you install something this conda conda looks for a channel and turn it channel is a place somewhere Somewhere on the internet was maybe even your own machine. That doesn't it's also possible That looks for packages And the default is anaconda server, so we don't say anything typically looks there But there's also something called conda forge, which is a channel There's a lot of open-source packages by the pricing community But you can also have private channels, so you can have a channel that's your own name These are company name whatever and then you can install from this channel by and say say install dash see My channel my channel is the name of the channel and then it will grab the package from this channel will be included There's also a lot of configuration options, which channel has precedence and which channels look first You can configure this in a configuration file Okay, a few basic tasks I want to introduce you to what you can do with it That's by no means a comprehensive introduction, but gives you some feeling what you can do so you can install packages you can create and administer environments and you also can create packages yourself so Writing is that a pie as you know and doing a lot of steps to create a package? The first thing you can search for something So you say conda search and you type in the name of the package You're looking for I use pandas pandas is very much used package and scientific library and then Goes out there and it's searching places So I added conda added a few channels you see every dot you see they're fetching packages dot dot dot dot And every dot means looking for a channel So I have by a bunch of channels it's looking at and then gives you a long display I just cut out of a few things and you see the three dots there It's cut out and it lists you those then the package so everything this search looks for everything has pandas in its name And shows you the version that's available Though all these versions are available is a normal one dot a dot here means it is available And it's downloaded though it when you're down when you install something it puts it in a local repository And you install from there. You don't have to download it every time and the one with the star here was asterix That's the one I have installed right now here and you see if it has a dependency on numpy was a scientific focus tells you I have numpy one dot Ten and pysons three five and build zero and it comes from defaults Which is for my naconda, but there's also a version that comes from quanta for sure It gives you the channel here that comes from but it's very interesting can be very long list And but it shows you what you have so it's easy a bit longer here and by the cut out quite a few It can be several pages long easily Okay, the search can be it more fancy so That's a problem. I have a pit when you search for something for church for jungle It just gives you I don't know how much very very very long listing this everything has a jungle inside here You can say for the full name and now it displays only the package It's called pandas not all the packages have pandas in their name and now I get only pandas It's still pretty long list, but it's much shorter already And I know it that's pandas and that that's a good feature and there's more so you can also Limits a search or say I want to have only Even though I did it on my Mac. I can search for all the available packages for Windows Win32 and I also can specify the version. I want now shows him all the available packages for Versions 0.181 and it tells me okay here now it offers everything starting This is numpy 110 Python 27 up to one pie numpy 111 Python 35 So there's no dots and last race because obviously If everyone make it doesn't need to download it here So that's it's not there, but this would be available and would download it and they all come from defaults and there's more options If you want there's quite a few switches you can use and can make the search even more specific pretty interesting feature Installing the package is very easy You just say content store and you name the package and it goes through your channels and you can at the config Fizer the channel that comes first attacks on this channel. See now it provides me with a name I want to install this package and typically tells I'm going to download here doesn't say because I've downloaded ready Typicalist, okay I want you to download these packages and then I install these ones like mkl's mask on a library all the dependencies typically if I have a properly packaged Application in a conduct package and all the dependencies are there installed all the dependencies and everything's binary So everything especially a scientific field. There's a lot of C and fort on extensions and Compiling these things typically works in Linux it might work on OS X and it's right like it doesn't work on Windows So that's why it's and still I think most people still Windows users are the most users of Python I think on Windows as According to all the statistics I've heard about and a lot of scientists and a lot of engineers So I teach a lot of engineers and most of the engineer companies are mostly Some of them have used Linux, but by far the biggest majority in Windows So you need to support Windows users and it's a pretty good way everything is binary and you can Do this okay create an environment So this is the first thing installing something but very often we don't want to install in your what's called the root environment You want to create environment? And that's what this is command up here. You can see conduct create Dash n or dash dash name. So I create a new environment which I call my pie There are three five and I can specify the Python version and I say I want to have Python 3.5 and You can also have if you wanted to could have a 3.4 or 2.7 or even 2.6 so far it supports 2.6 2.7 I'm not sure 3.3, but 3.4 3.5 and They are pretty quick as soon as 3.5 is out. I won't take along 3.6 3.6 will be supported typically a new NumPy version is supported within 24 hours or something pretty fast And that tells me what it's going to do. There's it never does something this out of asking And it tells me okay I need to download these packages and I need to install these packages or the dependencies and then if you say yes Which I did then it starts creating this package and gives you this The car cut off here, but it shows you how long it takes and how fast is doing stuff Okay, now it created environment to in to actually activate Now I can actually list environments. I created environments and I don't have any one I've just a small selection of the environments. I have you say condo and list for instance It's one of the commands that shows you what environments you have and you see an environment It's nothing but a directory in your home folder in my my user folder under condo under condo amps So if you install mini condo will be mini condo amps and here's another condo amps It's just a different directory and these directories contain all the information about environment and you see now the root environment is a The one that's active right now. It has this Okay, you see I gave the trailer the pie data Berlin here Then I made a new for the tutorial I made a new environment and I don't mess up with my route I can install what I like different versions and everything is isolated Okay, activate an environment So depending if you're on a unix kind of system linux in OS X then you have to say source activate name my environment on Windows is just activate so typing source activate is pretty long So I just had defined a short alias But that's to be how we how we activate an environment and just type deactivated deactivated again And then the prompt changes and then you see prompt and now when you list it again, you see now the environment will be Activated you always can see which one's active Okay, and then you can say list and Then it gives you a list of all the packages you have installed You see it tells me again the name of the package the version Here this dependencies of numpy or not the Channel and this one is red because this when you see it's installed this pip So you want you will find maybe a few hundred packages there You can install this conda directly if not you can always install this pip and conda will recognize this one So it's not a contradiction to work. I love this conda pip we can use both And that's what I typically do if it's not available this conda I use pip and install this pip but conda writes these these packages and you that's why I say it's installed this pip Yeah, so you have the version is installed this pip and you can also install builds There's no problem. So there's no contradiction in this and you have this pip installed there also Okay This is if you're a kind of a consumer of packages you can install it can can environments There are way more options. I just show a few but now the next thing is building a package So you want to build a package and there are essentially two ways You can build a package you can just take a package from pi PI and Convert it into a conda package with a few commands or you can start from scratch Bring it with your own package Okay, building from pi PI is pretty easy. There's a skeleton first. You need to install Conda build so conda builds if you have an account that's there already if you have mini conda need to install conda build Which is a command line tool that helps you with the package and then you say conda skeleton pi pi pi pi my package So this one goes to pi PI fetches The package we have there and then you say conda build my package and that's pretty much it So it's not very difficult and it's built a package and the package is a it's a par ball It's a car PC to in this case. It's just one file with everything inside And that's it and now we can install it and per default it creates it for the for the platform You're on so it's if I do this as you can see here's now score as x and You can now install it and that actually two different ways installing it right now I can say conda install use local my package And you just will go to the place because it knows that your fault place We are put it and installed it but you can also if you wanted to if you had somewhere else You could just specify the full pass with me for just file name. This is exactly the same here But it could be a different way just if you download one of these you can always install of this conda install from the command line You can also specify a Python version So even if you have a different Python you can say okay build this one for Python 3 4 and it builds it for 3 4 Yeah, so even if it's pure Python it always has a Python version inside and this is especially useful if you build extensions and see or Fortran, which is very common in scientific field and This one builds everything for this You can also convert to other platforms you see I can say conda convert platform all for instance and just specify my My file and say output directory And if it's a plain Python version it will create a version for each platform for Windows Linux and Mac If it's this extensions is not that easy then you need actually need the operating system because you need to compile it So then I'm I don't think if there's a cross-compile at least I didn't try it I don't know but So so far I compiled on Windows on Mac and I think in Linux is the same to be Linux is easiest to compile it anyway So if you compile extensions it works and I even compiled Fortran extensions Which is to be a bit more involved and see Simpson and the books nicely so far so you can create this extensions also and then if you like to you can upload and You need the Anaconda client and now I Uploaded to the Anaconda and you can also make your own channel if you like So I don't have it here, but you can also create a channel and you just say Anaconda upload and just give full pass of this file and it uploads This thing Anaconda and other people can install it just by saying condo install name of your package and If you put it in a different channels, they have to specify the channel when they install it Okay this is building From a PI PI package you can also do from scratch So if you want you can customize things and you need a few things you need a meta yaml So a yaml file that declaratively Specifies what you want to do and Then you need a build file and there's two different ones if you're on Windows You have a build dot BAT that file a batch file and then Linux and Mac You have a shell file and you just use a normal setup dot py so you probably familiar with it So you just use the normal setup dot py and then you have additional information in the meta yaml Which also has dependencies in it Let's have a look at this file So the meta yaml looks like this is just a yaml file and has a lot of different Sections in there. You don't need all of them So you have something about met information about the package then you have the sources So you can say it comes from some hits URL something like this You can have requirements and there are different requirements your violence for building things and provides for running For instance, if you use science and you might need science and to build it But you might not need to install on the client machine because you just have this to be the extensions you have different dependencies for the tests what you do with the tests and then you have normal information about who the also is what the licenses and Okay, this is just a short part of an Adventa yaml I used so I used just my package name here and you say, okay The source is just depends where all two directories up. There can be a relative part for instance. It doesn't have to be a URL Requirements of Python and setup tools and then for running I need Jupiter and see here This is something you can specify this one I need it only for Windows and the same way you can specify Linux and Mac. So this will be only installed For the Windows version, but not for the version of other operating systems. That's as well Very important and you can use the same syntax for and other places. So you make it dependent on the platform with one file Of course, then you have this MIT license, whatever you want to put in there Then you have this build files and they're typically if there's a skeleton made it just you set it up And you create files that look like this and they're just you see You just call Python and set up to py install and they just call your set up to py the setup device Just the normal one as you know from the normal installation process But you can also add other commands in here, which is very interesting So if something is not available as a contact package in here You can do a pip install something and this will install the package and this will go into this top all So everything you pip install the you will deliver the same package. I had this problem I had the library that was not available as a condo and difficult just installed it and she could distribute it alongside There's quite a few things you can do here in this build file You can customize things or something doesn't work typically have a way to do it in build file some way or the other Okay And then just build it is a conduct with my package And that's it and all the other steps are exactly the same as with the skeleton So you can do upload and all kind of stuff install from the local machine operations Good That's it already so conclusions I think condo is a very interesting package and it's very much used in scientific field But can be really useful for all Python programs because it's a pretty good installer in a very nice package manager You can manage environments and you can also build condo packages It works together with pip so it's not a contradiction You can still use pip if condo doesn't supply the package But as you can see you can always convert a pip installer package into a condo package if you like They might make the installation more smoother for it for your users then you might actually Just convert pip package to a condo package Very well known in scientific community if you go to a sci-fi your sci-fi conference pretty much everybody will know a bit about it I think But it can be really useful For all Python programmers and I encourage you to give it give it a try Thank you very much, and I'm here for questions Because and ask them for me right first hand went up here Thanks, I understand I can use condor for distribution and platform independent package installation, which is great now if I Have I am a maintainer of a tool and I have lots of different dependencies Outside pi pi and inside pi pi and with npm and whatever and it would be nice to be able to just Use condo install whatever for me Would I have to package all those npm pi pi packages to condo manually or is It possible to I don't know say condo just install this and install it from pip because you don't have the package yet Depends there's just some things you can do so if you install a package just that that is condo Then this package will install its dependencies So if you and say I say I depend on pandas pandas has a bunch of dependencies and install numpy for me And so I don't have to do this to some degree you can do pip so far you have to as I think you have to write it in the In this built script you have to say pip install pip install pip install But probably the this meta yaml will be improved That's what I just saw a ticket that you can then even say okay This is supposed to be come from condo This is supposed to be pip and this might be a different channel so it's still evolving and not everything works perfectly. There are some Problems there's no problem. There's a perfect but so far you you can always have a walk around by putting it in the build script I think So what would be your suggestion for this kind of situation? I'd put it in the build script. I had the same dependency thing and I put it put a pip install in the build script in this build SH script and it worked for me so that was was the easiest one the other thing It's just you can always convert the pip Packages into contact package and then you can list them in a meta yaml and just upload them as long as they are BST MLT lesson shouldn't be a problem and anything just upload them to Condo crowd and that's it. It is just you can automate it You just need to list all the packages and then you can update it if it's in a version Just check if the new version and just build new packages. Okay. Thanks Thanks for the talk. So one of the selling point at the beginning that is that not limited Not only to Python right but all the examples have seen all the pattern packages So what do you mean by not limited to Python? So you can use this everything is declarative So people use it for Ruby and other languages so you can use the same thing and you can also have Dependencies that are not necessary Python for instance one package is called pie tables is the interface to htf5 And this depends not only a numpy, but also in the htf5 library There has to be in a different version and all the things can be packaged in there So it you can have depend DLLs char libraries You can include them in your thing that makes it much easier if you have a good if you have a package at the end Then the user doesn't have to do anything about fiddling around this include parsing Oh, it's kind of tough suppose compiling htf5. It's not the simplest thing So I tried this once before condo and you can easily take two days out of your life and you still haven't All right, thanks. Oh, thanks for your talk very to the point Given that recently we have been adopting by PI binary wheels They seem to solve most of the issues that condo seems to solve Other than the fact that you have a single command to manage your install packages in virtual environments. That's good It's nice, but what's the big advantage now that binary wheels are well taking off You write wheels is it's a great step forward as I say that two things not everything is in a real format yet and you might not have the right version and the right platform in the in the wheel and Condo can install wheels no problem. So you say people install and install the wheel So that's not a contradiction this way at this at this as far as I know the story goes like this Travis Elephant he's a CEO of Continuum which is behind this it's open everything's open source But there's a company behind it and he he asked Peter van Rossum a while a few years ago About this packaging and because I just go ahead and do it And that's what it did So it's kind of a little bit a parallel to some degree a bit parallel to PIP and wheels and something like this but it's I think it's more comprehensive and It's better and many has a lot of nicer features and PIP and wheels for instance the search for name I couldn't I don't know maybe I'm not smart enough I couldn't get PIP into just showing me this package It shows me everything shows me hundreds of packages for Django and this is not really useful They have to run it through grab and do your own thing. Why not just say I want it I mean just customize search and those things aren't this the condo so condos nice But you as I said you can use everything that pip and wheels and pipe your eyes You don't exclude it you can still use it and I do it very often because it does only 200 packages Or maybe now 2000 out there, but not 80,000 or 90,000 whatever it's on All right any more interesting questions for Mike. Here we go. Yes Best question yet Okay, so is there like any Standardization effort like are you working with Python packaging authority or something or is it going anywhere? Or will it be forever separate like conda? It's great supposedly, but only in the scientific community and like Everyone else this is bad Great question, but I don't know the answer Would be very nice I would I would like to see it this way that both but actually everything is open source and both communities could use the code from the other Easily no no problem. I think everything is BSD or MIT license So there shouldn't be a big problem to use code and the I don't I'm not really sure why this is not a bit more coordinated efforts I'm I don't know it would be very nice to have one stop and Join forces and do it together For these things and I don't know why it's the case. There might be some reasons and haven't we found out yet More questions more questions keep the dream alive everyone. I Have a question. It's like a two-parter So presumably if I install anaconda it's gonna come with its own versions of Python as well as my system pythons that I might have installed separately And I once had this problem where or you know I don't know if anybody else seen the problem where you upgrade your operating system and you upgrade your system Python and all your Virtual lm's are now broken because something something something which I don't really understand And you have to delete all your old virtual lm's and recreate them is anaconda gonna magically fix that problem for me or sub question If I upgrade, you know, if there's a Python 3.5.1 and a Python 3.5.2 How does anaconda manage the upgrade process there? Does it need to rebuild a bunch of things? And there shouldn't be a problem. So if you work with the environments those environments are pretty much isolated under totally Isolated sometimes if you use command line tools and they're installed like in the scripts directory is from there If not, it goes outside So if you say pile in something and you haven't by installed pile in the new environment It might take the wrong version and gives you a lot of syntax errors because it's a pile in for two seven And you and the three five environment But if you install everything it puts everything in the directory and there shouldn't be a problem and you update if you work with environments and You will see if you work for while you will because environments anyway all the time you will rare Actually, maybe the best approach just have the route just as a helper and then work in environments So or isolated of course you can always if you can always reproduce the environment You can freeze environment it just just specify what you have installed You get a big list of installation and then you can give this this file to somebody else and the other person can say okay Install from this file and install exactly the same versions of the ones you can reproduce the environments and they should solve the problem I don't know if that could be some you can you only find it out if you test it really and do it But theoretically there shouldn't be a problem practically. There might be one. I don't know Is that only me that's ever had that problem of changing the system Python and all the virtual labs are broken So so I guess we're thinking that Anaconda is like kind of even more isolated from system pythons than than virtual end managers if It never has that problem Yeah, there's some voodoo there further research required any more questions. Yes. Yes fantastic. All right No, he wants to hear my voice any longer than absolutely necessary Are the environment relocate table or is there any plan for that? Can you repeat a question? I didn't get it are the environment relocate table. So I Installed somewhere. Let's say archive the environment and Uncompress it somewhere like another Computer. Yeah, so I just said it you don't actually copy the environment You just copy all this every all the packages are installed as a version. So but I showed you this listings here So one of this earlier things shows you this Whatever is installed you have this List somewhere do have a list. Yeah, something like this so you can actually Freeze no, that's a search add to somewhere you can tell tell okay export all the versions I have in the file you just get a file with all the names Equal the version and then you can give this fight somebody else and somebody else can recreate exactly the same environment Exactly the same versions You should work it's just a directory The problem is in not everything is copied though if like I have this pandas version I have may I use it in ten different environments, but it's only once in my in my Repository and not directly in the file It's just a sim link and you have to make sure that you copied it along with visit But I think there should be an option to copy everything just sip the whole directory and put it somewhere else It should work as long as the same operating system But I no guarantee but theoretically should work can try it out But maybe recreating is it's the safe of thing So if you recreate you can also say okay, it freezes everything with except the same version But you can also go in the file and edit is okay It shouldn't be exact the same version should be at least the version or even update the version so you can go in File and customize it and then still everybody who has this file would create and this nice thing is would work I can freeze on my Mac and somebody else can create the same environment or windows I shouldn't be a problem if I sip the file and would only be for the same operating system All right anymore for anymore. It is now or there is one okay You have to make it a question you would talk very very fast and Mike will talk very very fast Okay, so let's say that I have a package Python package on Linux that has That needs separate Installed C library and at and let's say that this C library Expects another C library very so what will come that it will Download this library and all of its dependencies dependencies and put it in the condo repository locally or What will that different ways I had the same problem typically what I did I just included in the file So I had this you can you can specify the Dependency so if you compile something I had for trying dependencies some shared libraries somewhere and you included so I'm not read myself. I'm not sure how to the best way to do it, but that that different ways including Shared libraries in there and you can distribute the shared libraries along and they will put the shared libraries in this environment And they wouldn't clash with your shared libraries because you might have to share live them or maybe different versions Then it works for a while and until it doesn't work anymore because it's a different version something is different So you can distribute shared libraries with it. There's different ways of doing it It had you have you still have to look at there's a lot of recipes out there how to do it And probably you will find one that serves your purpose Okay, let's give Mike a mother big hand. Thank you very much for a great talk and do it break