 Okay, so next up welcome to Balashankar C or value, feeling less adventurous. So value is going to talk to us about the tool Omniverse that he uses in his daily work at JITLAB as a Build Engineer. Yeah. Thank you, value. Thank you. Yeah, I guess I'm audible. Okay, so my name is Balashankar, but you can call me Baloo. That's too many syllables in my original name. So I go by Baloo. I work as a Junior Distribution Engineer at JITLAB. In my first life or in my open source life, I'm a Debian developer and they contribute to other projects like GNOME where I'm a foundation member and Mozilla regarding their localization efforts. So today I'm going to talk about a tool called Omniverse. So Omniverse in a one liner, it is a tool that helps you create full stack installers. I'll be explaining what full stack installers and why they are. So the agenda in a brief is why this tool or why full stack installers are needed and then introduction to the tool, the internal architectures, the technology used and the process that we take while using this tool and outputs and some other features of it. And if time permits, I will show you demo or how we use it at JITLAB. So to begin with, why this tool or why am I talking about full stack installers or why they are necessary, one of the mantras of the current century or the current time that we live in is automate everything. We want to minimize the manual work and we want to automate almost any possible steps while developing a software, while releasing it, testing it, monitoring it, whatever it is. So we want to automate everything. So just how many of you work for a product software company? Not a services one, but create a software product. Anyone? Okay. So basically if you are developing a product, one of your worst estimates is to distribute it having multiple operating systems or multiple architectures and stuff like that. With the arrival of containers or orchestrations, it has been reduced but there is still a pressing need for the good old OS packages like Debian package or RPM package that can be deployed easily to production because it is a familiar workflow for the operations guys. All this is a means. So full stack installers are basically these packages, this OS packages that provides a batteries included feeling like you get every software dependencies or everything you need to run your application in a familiar RK format that you can use with your regular package managers with Debian, it is the depth format with Red Hat family, the RPM one. So why this full stack install is or why this batteries included concept is good is because it safeguarded you from dependency version mismatch. So I do not know how many of you were affected by it but recently there was an issue with LibSSL version upgrade. Basically LibSSL 1.1 is in backward compatible with 1.0 and one example I know is Ruby depends on LibSSL for building. So all of a sudden all the Ruby 2.5 builds started failing because the OSS had upgraded from 1.0 to 1.1 and one fine morning every builds were starting to fail or the production instances were going down because the citizen guy just did a regular upgrade of their system they did not expect something like this to happen. And one other advantage of using full stack packages is that almost all of us using a operating system whether it be a GNOME Linux based one or a Mac OS based one the basic thing we will be familiar with is its package manager. For Debian you will be using apt or aptitude or apt-get for the redhead family there is DNF, SUSE has SIPR and all those. So this OS package manager provide you a familiar workflow whether it be with installation, upgradation or removal any of that. So providing your software so providing the product that you created in the form of an OS package actually helps your customers because you are giving them that they are familiar with. It is not that Docker or Kubernetes is hard to understand or hard to deploy but providing it in a native package format actually saves them from learning yet another orchestration tool or yet another deployment tool. So that is one of the advantages of providing a full stack package and the third one is because you are providing a batteries included format because you are bundling everything that your software needs you are pretty much guaranteeing that what you are providing will work will actually work it does not depend on what machine they are running or it does not depend on what init they are using my software does not work if you are using system D. So such conditions never occur. So basically this is why we are thinking back to full stack packages or native OS packages. So that is about full stack packages. So what I am talking about is a tool for creating these packages. It is called Omnibus. It was developed by Chef Software Ring that was earlier called as Obscote. So this is the repository where they do the development you can visit that repository if you need. It is a free and open source software which is licensed under Apache 2.0 license and it follows a Ruby similar DSL like a DSL that is so close to Ruby. So that is easier to understand and easier to hack upon. So before we start with the project DSL I will briefly explain what this tool does. So in short this creates a full stack installer that means it contains your software it contains all the dependencies of your software whether it be software dependencies whether it be frontend dependencies like Node modules or JavaScript modules or Go libraries or Go binaries. So it bundles all of this stuff all of the software and their dependencies and it converts it into a package that can be installed on different operating systems. So if you are creating it on a Debian machine it gives out a depth package if you are running it on a Fedora machine or a Red Hat machine it gives you an RPM package. So the components include basically the project information like methodology regarding what the project is and what all are its name, version and all those stuff and in addition we are describing the dependencies that these projects use. So as I said it's a Ruby like syntax so this is the project DSL like where we describe what our project is and what it do. So it has a name, maintainer field, the home page replaces so people using OS packages may know sometimes one of your packages may be replacing the older version of another one. So this is an example from GitLab because Debian has a native GitLab package the one GitLab in provides replaces it install path is where the software gets installed to the bundle software gets installed to and you have the version and here you are specifying the dependencies what all dependencies or what all libraries or software your application needs to run. So there is one called preparation there is a PostgreSQL is needed software called GitLab cookbooks is needed so this is basically the project DSL it consists of your brief overview of the project and the method that are regarding it and the software dependencies it has. Now the next step we need to do is we have to instruct omnibus where to get the software's how to build them and where to install them to so similar to project DSL each of the software's gets their own definition or manifest file it's called a software DSL so this is an example of one library called bzip2 the first range is the name bzip2 the default version is the version we want to use with our software the license, license file and again these software's can have other transitive dependencies like bzip2 bzip2 requires the zlib library and the open access to work so we have those described as the dependencies and we will be providing software definitions for those dependencies also then we do some integrity checks we specify the MD file some of the source starboard so omnibus actually validates whether we are installing the correcting it's a source URL and thus at the bottom there is a build block so build block is where we actually specify how you want to build this software for regular libraries it may be normally you have to configure make and make install for go libraries it may be go get the library name for Ruby gems it will be bundle install so it is in the build block that we specify how we want to install each software so this is the build process explained or expanded so basically you specify the environment flags you want to use during the build if you have some patches to apply you specify those patches then at last you perform your regular make functions like make pass the arguments and use this make file and run make install with these environment variables so this build block actually specifies how you want to build your software where you want to install the software like your application may be expecting all the libraries in certain folder or your assets in certain other folder so you can specify all those in the software DSL specifically in the build portion of it so in short these two are the necessary things for an omnibus project to work or omnibus to convert your software to a package like we say here we can provide patches or we can provide overheads and stuff like that if needed like you do not want upstream turbo to hard link a specific location you want a patch to override that you can specify that in a patch file and omnibus will automatically use that while building so that is basically the two method or manifest files that you need so to understand what actually happens when you run omnibus or when you use omnibus so on the left side you see the basic explanation like you have your project DSL you have your software DSLs like what all software you need so the process is like this first it reads the project definition or the project DSL it understands what the metadata regarding the project is it gets a list of the dependencies then once it gets a list of dependencies it has to go and start collecting these dependencies it may be from an upstream turbo it may be from a Git repository or it may be a local file so it fetches the software sources the next step is you actually start building those sources like we specified in the software DSL we have a build process for each software so one by one you start building so the order of building is that the leaf dependencies gets built at first because the software that depends on them can be built so you one by one you start building the software once all the software has been built you generate a license information regarding the all the software that are included in your package and after that we do something called health check so health check is an important step why because the main reason we are doing full stack install is all full stack packages is that you do not want a dependency on your OS libraries like you want to provide a batteries included a concept so you want to provide your own libraries and your own executables so health check actually verifies that all of your software components or all of your binaries link against the libraries that you provided not the system provided one or not the shared library but the once you have bundled in your package so health check basically verifies whether your executables are using the bundled libraries or not if it is not it reports that okay this software requires this library and it was not provided or it was not bundled so once health check is completed and health check is satisfactory it goes on to creating the dev file it is using the default package management systems that different operating systems have like for debian it is using dpkg dev the tool dpkg dev to create a package so on a basic idea this is the entire process like you read the project definition you understand when you start collecting the sources of these dependencies then you start building them one by one you generate a character license information because you should not be what violating licenses of upstream code then you do a health check to verify everything is using the embedded libraries and not the system libraries then you create a dev package and we have some concepts called cache that I will explain later so in short this is the process that goes with omnibus the output of this process like what we get at the end of the process are basically three things the one is the platform specific package why it is platform specific is in the sense that if you run omnibus on a debian machine or if you run it on a debian virtual machine or a debian platform the output you get is a dot deb file similarly if you run it on a Fedora machine you get an RPM file you run it on an ARM board with RASB and installed you get a deb package for ARM machines so it provides you the platform specific packages that you can simply install and use that is the first output the second one is a package method at the file this is like a everything included report like it provides the complete information regarding the packages the included software their name their upstream URL everything the third one is a version manifest file version manifest file is a JSON file that you can use to verify like which version of the libraries were included what were their licenses or what from where they were downloaded all those so in case one of your softwares were pulled from a git repo not an upstream tarboard it will show the git commit that was used to pull the upstream source code so for tarboards it will show the version string so these three are the output that you get out of a omnibus run okay so I mentioned that there was a concept of caching involved this is to make sure because suppose you have a software with 60 dependencies and you made a tiny mistake in writing one of these of your definitions the next time you build you do not want the entire process to be repeated like you made a mistake in the 60th software dependency of the definition and you do not want to wait for the whole 15 and wants to be completed just to verify what you changed work so there is something called caching omnibus two types of cache first one is for the upstream sources in the first step of the omnibus process we see that omnibus downloads the source code whether from upstream tarboards or whether from git repositories so the first cache is for caching these downloaded softwares it is backed by an s3 bucket you can specify an amazon s3 bucket or gcs google cloud storage bucket so what it does is that it caches all the software definitions and in subsequent runs if the versions have not changed all the git commit shas have not changed it will simply pull from these bucket and it will not go such all the internet such as the specified url to download the software cache so that is the first type of cache this the second one is a bit more complex it is a cache that is used during the build process so I said the build order the order of how softwares are built is that the leaf dependencies gets built first so like if a depends on b and b depends on c while building c actually gets built first so that we can use it during its own build then b gets built then a gets built so the problem with this is that whenever you want to change something in a you wanted to change some configuration or some build process in a the entire process gets repeated you start with c you build b then you go to a so each time doing this is time consuming so there is a cache available during the build process what it does is that it uses lightweight git tags it uses git tags to store a snapshot of your build directory so that whenever you are repeating something if the software definition has not changed it can simply use that snapshot so basically unless cache is invalidated it can use that snapshot how cache is invalidated is that if there is a change in the order of build suppose you change it depends on c and c depends on b so the order of build change now b gets built first so if there is a change in the order of build the cache gets invalidated next one if there is a change in the process the entire build process like you are no longer running make instead you have some other tool to do that process so there is a change in the software definition then the cache gets invalidated and the third one is there is a change in the version of the library so unless cache gets invalidated in any of this manner the snapshots that we create that the omnibus create after each build process can be used to speed up the subsequent builds so suppose in our a b and c example you wanted to make a change in a so the next time you run when omnibus checks there has been no change in c so it will simply use that snapshot again there was no change in b it will use that snapshot but there was a change in a so it will rebuild a so c and b does not get rebuilt over and over unless the cache gets invalidated the snapshots gets used and you get faster subsequent builds so this is the concept of caching and in addition to the regular package generation there are some advanced features that omnibus provides first one is gpg signing for integrity you can sign your packages and you can so that your users can confirm that these are the actual ones that you provided there has not been any hindrance or malpractice with it the next one is change log generation this is an experimental feature omnibus can read your git commits and automatically generate a change log file that you can ship with the package the next one is multi-level debugging you get different levels of debugging what went wrong like from where the sources were downloaded whether it was picked from the cache or not what was the output of the build process so there is a multi-level debugging and there is automatic publishing to different back ends like Artifactory or you can push an Amazon or GCS S3 bucket so these are some of the advanced features that omnibus provides you once you create your package okay so I will show you the repository that we use at GitLab where we use this for our GitLab package generation so to summarize the talk food stack installers are still a great idea because they provided a familiar workflow for the CISOP guys all the operations guys so if any of your product developers do take a look at this idea because it can be an easy way to ship your product to different users it is a Ruby based easy to understand and rich DSL it is almost like plain English and you can simply read and understand from the context what it means the third one is there is a single general process for all your operating system needs like for Debian for Fedora for Red Hat anything you can use the same code base and the same process and omnibus will take care of the rest and you get a platform specific build process with caching and these advanced features like GPG signing or automatic publishing and all so let me it is visible right this is one of the repositories that we use at GitLab we use this to create packages of our GitLab communication and enterprise edition so any of you using GitLab personally or at work so you can self host GitLab you can get an entire integrated version controlling system with continuous integration continuous deploying and everything integrated so this is what we use to generate those packages so all the DSLs all the DSLs like the project DSL and software DSL goes into the config directory of the repository so first let us see the project DSL because this is a Ruby based DSL you can use conditionals and all that so if you are building an enterprise edition package we set certain descriptions and certain metadata and for the community edition we set it differently we have the maintainer and like it replaces the native GitLab package because we provide something else there is a runtime dependency on OpenSSH if it is so say the package is called OpenSSH if it is not it is OpenSSH server and we have all these dependencies that we need to make sure GitLab actually runs without any issues so all of these gets a software definition in the software folder like all the dependencies that we specified in the project folder gets a software definition similarly there is a patches folder for whenever we want to patch some software we have a patches folder that we specify in the build process use this patch to patch your upstream source code so basically that is it do take a look at the Chef Omnibus yeah the second one is the upstream project of Omnibus Seth Chisholmora and Christopher Mayor from Opscode or Chef these were the people who actually built the Omnibus tool they actually provided a good presentation introducing Omnibus to it so you can check out that also so yeah thank you thank you very much I suggest we have the next speaker set up yeah