 So I'm going to talk about two tools, but primarily Hydra. So myself, I work at Intel, but Hydra was developed at Facebook Research, so it's from Meta. And I'm also going to mention a tool called MLflow, which is now curated by the Linux Foundation. So what is Hydra and what are we trying to solve here? So when we're doing data science experiments or machine learning experiments, we have you run into the problem that you have quite a few things to keep track of. You have your data set, you have your model, you have your algorithms, parameters, optimizers, and so on. So even though the Pythonic way of writing things is to keep interfaces very simple, with data science experiments we end up having a lot of parameters to keep track of sometimes dozens of different parameters. And this is actually science. We're doing science meaning that we're making experiments that we don't really know the outcome of. We're making an educated guess that this set of parameters will be the best one, but we really have to test, and only after we train our model, test our model, do we know the answer if this is the right set of parameters or another set of parameters might be better. So because we have to make many of these experiments, the number of experiments we have to keep track of all the different combinations of parameters that we tried in order to explain what worked best and in order to reproduce it. So these problems managing complex configurations, making them traceable and reproducible is what we're trying to solve here. And one of the solutions is this application framework called Hydra. So I'm going to tell you how you can use it. So we're going to start with a simple example. So this is like a minimal version of a Hydra based application. In reality, it's a toy example. It doesn't do anything other than log a hello message and the parameters of an object we called model in the configuration here. But I hope you can appreciate how minimal the boilerplate that Hydra adds to this example is because really what Hydra is adding is just this decorator here. And it's going to inject this config parameter into the main experiment method. So what does this code do? Well, if you run it, it's going to log the contents of a configuration file. So you need to have this configuration file as well. With Hydra, you use YAML files. And if you have a model and its parameters defined like this, then when you run this very simple application, what it's going to print out is your hello and the parameters of the model from the configuration file. So far, very straightforward. What it did was parse the YAML file and pass it in as the configuration. So that's the basics of what you would expect it to do. But immediately out of the box, you get a couple of other features. So the first thing you get is you can ask your very simple application for some help. And that will check the configuration objects that are available to the configuration and list them for you. So you can inspect the different parameters of your application. And it tells you here that you can override anything in the config with a dotted path and a value. And so you can actually do it this way when you can run your script again, saying that model.a equals a different value. And now it will actually equal this other value when you run it. So this way you can inspect your configuration parameters. You can set your configuration parameters at the command line without having written much other than adding this Hydra decorator to your script. But another thing that it does is every time you run your Hydra-based script, it produces a directory of output files. And by default this is set to a date and time that you start your script. The main log of your application is stored in a file with the name of your application.log. But there's also a .hydra directory which contains three YAML files. And the config.yaml file is actually the effective final configuration that was the product of all the overrides that you did, or maybe a configuration file that you provided. Hydra-yaml contains all the settings of Hydra that were used during this run, and overrides are the things that were overwritten by you at the command line. And if you want to, you can actually use this config.yaml file, this effective configuration that was recorded during this run. And to do this, you just specify that Hydra should look in a configuration directory from your output.hydra and then use the config configuration there. And when you run this, you will get your stored configuration and you're essentially able to reproduce the previous run. So thanks to this, you get traceability and reproducibility of your experiment just by serializing the configuration that was used into a YAML file that you can then use as input to your next experiment run. So those are nice features of Hydra. One other nice feature is that it actually comes with a command line completion. So you get tab completion for your parameters as well. You need to execute this valve in your shell, which will generate code for shell completion. And then when you run your, when you're on your terminal and you double tap the tab after the model, it will suggest you the different parameters of the model that you can set. So that's very useful for being able to quickly set a couple of parameters that you want for your next experiment. Another cool feature of Hydra is multi-run and that allows you to run your experiment with a set of different parameters. So for example here, we're telling our script that it should launch the multi-run mode and that it should try to run with model A parameters 1 and 3 and model B parameter values 2 and 4. And it will make all the combinations of these values and start four different jobs and run each of them in sequence by default. And you will then get the results in the output directory of each of those runs and compare them and decide which one is best so you can sweep through a lot of different parameter values this way. So I hope that you're getting the impression that with not a lot of additional code added to your experiment, you're getting a lot of features. And so that's what Hydra does. Now I want to talk about how it works and how its internals are organized. So the components of Hydra are omega conf, which I will get to in a second. It also sets up Python logging for you. There are launchers that start your jobs. Sweepers that are the thing that scans the different parameter spaces that you define by providing the values or ranges of values for each parameter. And it's also modular architecture that can be extended with plugins. So omega conf is actually a separate package that Hydra is based on. It's the YAML configuration manager, which Hydra is based on. It was created by the same main author, Omri, who created Hydra initially. And you can of course install it from PyPI. What does omega conf do? What comes from omega conf? Well, it's the parsing of the YAML configuration files. So you can use omega conf to load up your YAML files. And then this object that is created when you load the configuration is quite flexible. So you can read and set your values either like this was an object or like it was an aesthetic dictionary or even just with the dotted path using the omega conf select function. So it's quite flexible, quite easy to use the configuration after it's loaded. There's also a nice feature called variable interpolation, which means that inside of your YAML configuration, you can use some values from the configuration to define other values of your configuration. So it works a little bit like in the shell with a dollar and curly braces. You can use a value from defined in the configuration as part of a value for another configuration. So if you were to define these three full bar and baz and baz used the previous two, you would actually get the combination hello your python there. Another useful feature of omega conf is called resolvers. And resolvers allow you to add functions that are able to take these configuration values and inside of the YAML file, you can actually have a little bit of logic that will either combine the other values or well in this toy example here, we define addition. And it's very easy to add your own resolvers. You essentially just call the omega conf register new resolver function and define whatever logic you need. So these configuration YAML files, they're quite flexible and allow you to do quite a lot just in the YAML file themselves thanks to omega conf. So that's the basis of HYDRA, what HYDRA is based on but what does HYDRA itself add? Well, HYDRA itself adds all these other features. It's an application development framework with minimal boilerplate and it's focused on managing all these configurations. So the first thing that HYDRA adds is the ability to compose these configurations from smaller files. So the idea is that you don't want to have a huge configuration file for your huge experiment but rather split it up into smaller sections and each section can then define a different model or a different data set or a different optimizer. Whatever object that has a set of configuration values that work together can be in a separate file like in the source code it's defined in a separate file. So you split up big configurations into small files and then you combine them back again through composition. HYDRA has interesting ways of thinking about these configurations. For example, each directory of YAML files is called a package and it kind of works like a Python package or module in that it defines its own namespace and it can be used interchangeably. There's also a concept of configuration search path, kind of like the Python path so you can have these configuration files live in different directories or different modules or they can be even imported from other packages that you install together with HYDRA. So all these things sound abstract so let's go through some examples. So the simplest way to combine configurations with HYDRA is using the defaults directory which allows you to set some defaults from another file. So for example, if we have an experiment configuration file called my experiment YAML, I can define defaults coming from a second file called training settings and if I have a second file called training settles the settings.yaml those defaults will populate the configuration and I will overwrite only the ones that I want. So I can have some global defaults defined somewhere and overwrite only the ones that I want in my experiment that I'm running right now. So the effective configuration that you're going to get here from these two files is the combination of this and of course as you would expect overwriting this one specific value. You can also have configuration groups and configuration groups are subdirectories of the YAML files. For example, you can have a sub directory called data set and inside of that sub directory have different files for different data sets. You can have another directory called model and different files defining different models and you can then combine them by specifying in your defaults list that the data set is named ImageNet and that will automatically, HYDRA will look inside the data set in the sub directory for the ImageNet YAML file and the output will be the content of the file taken from that specific model definition or dataset definition. You can also overwrite where you're going to use your configuration groups. So we're getting a little bit deeper into how you can use these HYDRA settings but you can have one data set file and reuse it multiple times by using this add symbol. So it's actually quite flexible. There's even a package directive that you can set at the beginning of a YAML file that will say that everything in this particular file actually populates the namespace HYDRA.JogLogging and this is actually part of a configuration for a plugin called ColorLog that makes the logs that HYDRA outputs nice and colorful which I also recommend. So the logging that comes with HYDRA is just regular Python logging but it's set up for you out of the box so the biggest pain point of Python logging configuring it is done for you. Another nice feature that I really like about HYDRA is a utility function called instantiate and it works like this but if you add into your configuration file the special keyword underscore target underscore then you can specify a dotted path to a class in your code base or in this case PyTorch code base and if you then call this HYDRA utility instantiate function on this configuration it will look up this module import it and instantiate this class with this parameter so that's quite useful for keeping all of your configuration together and making your application quite flexible sometimes you may have a problem that some parameters that you need are only known at runtime like in case of PyTorch if you want to define an optimizer you need to pass in the model so you first need to define the model and then you have the optimizer but HYDRA has another fun feature if you add this underscore partial underscore keyword to your configuration you can instantiate a partial so you get a funtals partial object it's pre-populated with the parameters that you specify but you can finalize the creation of your optimizer in your code until you have the model by calling it calling the partial when you're ready so there's also type checking possible with HYDRA so because I have not very much time I'm not going to get into the details of this but if you define data classes that have the same fields as the YAML file that you had before you can of course annotate them with types and then by calling the config store of HYDRA you can actually add them as configurations essentially you're replacing the YAML with type annotated data classes and then you can use that for mypy by specifying that the configuration object is this specific data class and at runtime it can even point out that this is the wrong data type for a value if you specify it so there are quite a few things that you can do with HYDRA beyond these basics because of its plugin architecture so launchers are responsible for starting jobs sweepers are responsible for setting different configuration parameter combinations based on the input and there are plugins already for all of these cool workload managers these libraries and these libraries for scanning parameter values so it's actually quite a powerful way to start using also these other tools so I want to talk very briefly about integrating HYDRA with MLflow but first of all what is MLflow it's an application that is very easy to use for keeping track of your experiments so the simplest way to use it is just to start a server locally on your machine for production you might want to set up a database and a bigger server but to get started with MLflow all you have to do is just these two things and then you will get an interface with your experiments where you can view your training progress and results and the way you actually log metrics into MLflow is very simple so it fits in nicely with HYDRA all you have to do is just import MLflow and start logging the metrics it will automatically create an experiment and run for you and how you can actually integrate HYDRA together with MLflow well you can log artifacts into MLflow so files you can upload files into MLflow as well so if you have the HYDRA config the effective configuration that you used for your experiment you can log it as an artifact into MLflow and that way you can keep your configuration that's needed to reproduce this experiment together with the results of the experiments in MLflow so it closes the loop so you can have all your results together and the ability to reproduce them as well okay I think that I have to finish so my takeaways are that HYDRA will make your experiments more easy to configure traceable and reproducible and thank you thank you very much for this great talk it was very interesting at least for me so are there any questions in the audience? if so just come in front to the microphone hi thank you I wanted to ask because I'm personally tracking experiments in Neptune but whatever use I think MLflow, Wys and Wys or Neptune it doesn't really matter it's very similar but what I noticed that during let's say initial phase of experimentation we decided we want to track these specific parameters and later in time and let's say we are training the model always with freeze initial few layers but then we decide okay we want to try new thing and maybe unfreeze some of those and what is problematic for me is to retroactively populate those experiments with some default value and is it somehow possible or doable maybe using HYDRA or is it just something that I would have to live with well if you if you did this and put your configuration into your experiment result tracking software then if it has the ability to scan through your experiments then you could extract any value that you didn't previously maybe add as a specific parameter from this from this log into your experiment okay so just go through all of well write a script for that and it's up and down so the the first example you showed is just a decorator on a fraction yes and then it kind of the further stuff you showed it kind of got more complex is it do you think it's easy to stop to keep this separate from your code and to prevent it kind of getting it's close into your code so that maybe later later times you can swap it out for a different experimental platform so the configuration itself probably is quite easy but when you start doing things like using the instantiate helper then it kind of becomes part of your stack and it might be more difficult to replace it later so I guess depends on how much how many features you end up using right okay but yeah you think the benefits of of using something like a stantiate seems like it's worth the benefit of allowing that into your code well the idea is that it's very helpful and it's not a lot of code so there's not that much of a downside you don't so because it's not so much code that's good thanks for a nice talk seems like an interesting package I was wondering about this multi run functionality you didn't mention anything about parallelizability is that because it doesn't exist yet or and if so is it something you're hoping to implement so the plugins for Hydra are actually solving this issue so by default Hydra just launches jobs in sequence on the local machine that you have but thanks to these plugins that are able to take advantage of backends like Slurm you can actually instead of running the job locally it just starts jobs and they run remotely and then they report back so it's check out these plugins if you're interested because they do the parallelization for you cool thanks thanks for a great talk do you have any experience with PyTorch Lightning I noticed you used PyTorch for some of your samples no I don't I guess that's just it then okay thank you very much for the talk a lot of applause for the end thank you