 So we are going to start with our next speaker, Vanak. Yeah. So his name is Vanak and he'll be talking about Akai2 building CLI applications in Python. He's from India and he's an independent software engineer and loves to contribute to the open source world. Welcome, Vanak. So what do you do? Hey, everyone. Are you able to see my screen? Yes, perfect. Welcome to the HHS guide to CLI as in Python. A little bit about me. I'm Vanak, a software engineer currently living in New Delhi. I maintain Camelot and Excalibur, tools that can help you extract tabular data from PDF files. You can check them out at this link. In next month, I'll be joining the RECURSE Center for a three month retreat and I'm pretty excited for that. RC is a self-directed and educational retreat for programmers in New York City. And I'll be looking for opportunities starting November. So if you are looking for a software state engineer to join your team, do reach out. Let's get back to our talk now. In the beginning was the command line. Well, not quite the command line we know today. They were typewriters and we were communicating using more school. So one fine day someone had an idea to connect a typewriter to an existing set of communication bias and the teletype writer was born. Teletypes removed the need for an operator to know Morse code and improved message typing speed and delivery time, making it possible for messages to be flashed across the country with little manual intervention. In this 1932 video, the narrator describes how a teletype takes only a matter of seconds to deliver a message from London to Edinburgh, which is in stark contrast to the earlier idea of One Week. One Week was a time taken by a mail coach to undertake the same 400 mile journey. Meanwhile, commuters were becoming powerful enough to multitask and interact with users in real time in contrast to the earlier batch processing model. So another fine day, someone had another idea to connect a teletype to a modem, which will let them interact with these computers remotely. Among these teletypes was the Friday flexorider in the teletype model 33. Teletypes like these were adapted to provide a user interface to early computers. And this was the origin of the command line interface. Users type commands after a prompt character was printed on paper. After they were satisfied with the input, they would press enter, which would then send the command to the computer. And finally, the output from the computer would be printed on paper again. Teletypes were continued to be used as terminals to modern two computers until video displays became mildly available in the late 1970s. Video terminals quickly became extremely popular. IO devices on many different types of computers. Once the manufacturers moved to a set of common standards, like ASCII in 24 rows and 80 columns of text. Today, we live in a time where physical teletypes and video terminals are obsolete. We instead have terminal emulators, which are software simulation of the real thing. But have modern terminal emulators borrowed any legacy from these old metal beasts? Let's find out. The one thing that is clearly visible is the name. If you take out TNTY from teletype, it becomes TTY, which is a prefix in the names for virtual terminals on UNIX based operating systems. The fundamental type of application that runs on a virtual terminal is a shell. The shell prompts for commands from the user and sends it for execution after the press enter, which is similar to the teledype workflow. So based on intuition, the whole thing kind of looks like this. Keyboard passes input to the terminal, which passes it to the process. The process does some work and gives the output back to the terminal, which then prints it on the display. But an illusion sits in between the terminal and the process. Termios, it's kind of like an interface to some default settings for socket communication parameters in line discipline, which affect how text is entered and printed. The man page for termios lists all these available settings. There's also the STTY utility, which can use to turn these settings on or off. And STTY-A shows all the settings and their current values. For example, the speed of serial communication in the number of rows and columns. Let's see what some of these settings do. I'll show the same examples Brandon Rhodes shared in his 2017 North Bay Python keynote. The first setting we'll look at is called ICANN. It refers to the canonical text editor used for some rudimentary editing of commands before they are sent to the process upon pressing enter. For example, moving the cursor back and forth or removing characters using backspace. Most interactive applications like text editors are turned ICANN off and handle all the line editing themselves. The canonical text editor is on by default and we can turn it off using STTY like this. Let's see what that does. We'll open cat. Since the canonical text editor is on, the input is buffered till we press enter. We can also use backspace to remove characters. Let's turn off ICANN and use cat again. And we press cat, be enter cat. And as you can see, cat is receiving a character as soon as we enter it and printing it right away rather than one line at a time. We can turn ICANN back on by removing the hyphen from the earlier command. Another setting is ONLCR, where NL stands for new line and CR for carriage return. This setting finds new lines in text and adds a carriage return to each one of them. A carriage return makes sure that the cursor moves back to the first column after a new line. Similar to the tele-deck days when the paper carriage would return to the first column after a new line. A carriage return without the new line characters used to make progress bars or modern-day terminals. The program updates the progress, moves the cursor back to the first column and then overrides the earlier progress with the new one. ONLCR is also on by default and can be turned off using STTY like this. We'll look at PS. The output looks very structured. Let's turn off ONLCR and then look at the output again. So we'll enter PS. And as you can see, the illusion is gone. This is a real thing. Each line has been printed on a new line, but the cursor does not return to the first column. A lot of applications are written with this assumption that the terminal will automatically move the cursor back to the first column when they print a new line. There's also echo, which is on by default. Echo directs the terminal to print every character that we input back on the display. What happens if we turn it off? We look at cat again. We can see what we are typing. Hello, world. But what happens when we turn echo off using STTYF and echo? We didn't see cat being typed. And the input strings too, until cat, which was running in the background, printed them for us. Programs turn off echo when they ask the users for passwords. If you're experimenting with terminal settings, you can use the reset command to return all of these settings to their default bandings. And you can also check out the termios module in the Python standard library to turn these settings on or off using your Python code. Another way to change the terminal status through in-band and out-of-band signals. In-band signaling means that you throw in some special characters in your input. The terminal interprets these special characters as commands. It does not print them, but it instead causes an intended effect. One way to do in-band signaling is using control characters. For example, control H for backspace and control C for interrupting the running process. And another way to use escape sequences which can control things like cursor location and text color. For example, printing the first sequence here will clear the screen and printing the second one before and other string will make the second string void. Terminals are also preconfigured with input and output streams where the input stream is mapped to the keyboard and the output stream to the display. This ability to automatically map the input and output to the keyboard and display by default was a UNIX breakthrough. In operating systems before UNIX, programs had to explicitly connect to IO devices which was kind of tedious because of a lack of standards across systems. So here, STDN is the input stream where a program reads its input data and STDN and SDT are output streams where the program writes its output data in error messages. Unless, of course, data is redirected using some operators. The greater than and double greater than are redirection operators which redirect the program's output to a file. The only difference between the two is that the first one will overwrite the file while the second one will upend to the file. Another redirection operator is the pipe which makes the output of one program the input to another. Now that we have an understanding of how the terminal evolved and how it works, let's look at some programs that are inside a terminal. Command line interfaces. The words interfaces, applications, programs, tools are used interchangeably but they won't refer to the same thing at least most of the time. CLAs make it easy to automate repetitive tasks via shell scripting and it's not kind of fun to use. The general usage pattern of a CLA looks like this. The shell displays are prompt as a sign that it's ready to take in commands. The username types in the command that they want to run along with some options and some arguments and finally ending the input by pressing enter. This completes the command line of text going in. Hence the term command line interface. The command is then executed and the output is printed on the terminal. But what are these arguments and options? Arguments are required items of information for a program required in the sense that the program won't work without them. They're often positional which means that an arguments position in the line will help the program identify the arguments type. For example, here's the copy command which can't function without both the required arguments. The arguments in the first position would always be identified as source and the argument in the second position will always be the destination where the source needs to be copied. An option or a flag is used to modify the operation of a command. And as the name suggests, they are optional and may have some default values. The general convention is to have hyphens in front of a character or a word to identify the option. For example, in the copy command, hyphen are changes its operation by asking it to recursively look for files in the source and then copied them to the destination. And one of the criticisms of a CLA is the lack of cues to the user about all the available actions in contrast to graphical user interfaces which usually inform the user of actions with menus, icons, or other visual cues. To overcome this limitation, many CLA programs display some brief documentation around the commands, arguments, and options that they support. This documentation can be viewed by invoking the CLA with the help option. And some of them also have man pages which is short for manual page. And this can be viewed by using the man command. Well, you must be wondering that there's a lot of moving parts here. Each programmer could write this CLA differently. For example, they could use the hyphen x instead of hyphen h to display a help text. Are there any standards to make sure that every CLA follows some basic conventions? Well, yes. There is a standard and it's called POSIX. POSIX makes APIs provided by units-based operating systems uniform. APIs such as command and interfaces. To follow the POSIX standard is to be POSIX compliant. There is also the xdg-based directory spec which dictates how CLA should store different types of files that they need for their function so that everyone doesn't go around saving files all over the place. These files could be your configuration files, data files, or the program cache which should go into these directories on our user's file system. Now let's see how we can implement a command line interface using Python. There are several options to do this, both in the standard library and on PyPairing. We'll use a small example CLA called smallpip and see how we can implement it using these different options. Smallpip just has one sub-command called install using which we can install a package from PyPairing. It also has an upgrade option which will upgrade the package if it's already installed and the package came to quite argument to identify the package itself. Let's look at the standard library first. It has a sys module which comes with an argv variable. sys.argv is a list where the first element contains the name of the CLA that was invoked and the rest of them are the command line options passed to the CLA. Internally, sys.argv uses a get-op module to parse and create this list of command line options. The get-op module is a parser for command line options designed to be similar to the Unix get-op function and it follows the post-extended. Let's look at some sys.argv code. So we import sys and we have a help state and when the CLA is invoked, we get the list of arguments using sys.argv. And since the first element is the name of the CLA itself, we check what the element index one is. If it is hyphen h or hyphen hyphen help, we print the help and do the same for the version. Finally, we check the sub command that was invoked and dispatch control to the relevant code. Up until Python 3.2, the standard library also had the op-pars module which has since been deprecated. Op-pars could only parse options and not arguments, which is something that Stephen Bethard, the author of our parse talks about in PEP 389. This way proposed the deprecation of op-pars in favor of the new and improved op-pars module. It was approved by Google on February 21, 2010, 10 years ago. Op-pars was written because both get-opt and op-pars support only options and not arguments. Op-pars handles both and as a result, it is able to auto-generate better help messages. Op-pars also allows customization of characters that are used to identify options. For example, using a plus instead of a minus or even forward slashes. Op-pars also added support for sub-commands. This is a common pattern in CLIs. For example, PIP install, PIP freezing, PIP search. Let's see how the small PIP code looks like using op-pars. To import op-pars, initialize a sub-parser object and pass in the description of our CLI. And we also add a version option. We then initialize a sub-parser subject and add a sub-parser for the install command. To which we add an upgrade option and a package name argument. The action is equal to store through, make sure that the upgrade option is treated like a Boolean flag. And when the CLI is invoked, we call the parser.parsargs function, which gives us a namespace object with all the command line options as its attributes. Finally, we check the sub-command that was invoked and dispatch control to the relevant code. The nice thing is that we got an auto-generated help for our CLI. Now let's look at some packages that are available on PIP. This doc op, it was written by Vladimir Keleshev and is kind of cool in the way it works. Doc op takes a documentation first approach to writing CLIs. It just requires a POSIX-compliant head string as an input from which it'll infer sub-commands, options, and arguments on its own. So this time around, we first create a head string which shows our CLI's description and usage. And when the CLI is invoked, we call doc op, passing the head string in the version and it returns a dictionary of all the parser command line options, which is pretty neat. We again check the sub-command that was invoked in dispatch control to the relevant code. In all the examples till now, we saw that in addition to parsing results, we also had to write some boilerplate code to dispatch control to the relevant install and upgrade code. If you had to validate the pars command line options, you would need to add some more boilerplate. This boilerplate can grow real big for large applications. There might also be some common features that we might want to add, like progress bars and colors, for which we would usually need to import other packages. Now let's look at CLI. It was written by Armin Donahar to support the Flask project. Click automatically dispatches control to the relevant code based on the sub-command that is invoked. It also supports callbacks, which can use to validate the pars command line options. And it's post-excombined. Let's see what the small pip code looks like using click. So we import click, we add, then we add a function called CLI with a doc string. And since click follows a decorator-based approach to writing CLIs, we add a click.group decorator to the function to make the CLI function a command group to which more sub-commands can be added. We also add a version option. And then we define a function called install with a doc string again. This function will contain the code required to install or upgrade a package. We then convert this function into a sub-command using the CLI.command decorator. The CLI and CLI.command is the command group that we defined earlier. We then add an option called upgrade along with the help string using the click.option decorator. The splag is equal to true makes upgrade a boolean flag. Finally, we add a package name argument using the click.argument decorator. And when the CLI is input, click will automatically dispatch control to the relevant code, which in this case is the install function. We will get the arguments and options as keyword arguments to our install function, which we can then use to install or upgrade a package. Click also auto-generates the help for our CLI based on the function doc strings and the option help strings that we added. Click promises have been multiple apps written using clicker string together. They will work seamlessly, which means that multiple people can work on different parts of a large CLI application and stitch them all together at the end. That's like we are building a CLI way we don't have to define parsers from the start or focus on our help text from the start. It's great for quick iterations. Now let's look at some common CLI use cases and see how we can use click to implement them. We'll use another small example CLI called small get. Small get as an insuggest is a small get clone with six get subcommands, clone config, log, status, commit, and push. We start by defining a CLI function with the click.group decorator. And now we look at the first use case, which is a pretty common one. To display progress bars to the user. For example, we should let the user know about the progress of how many files have been cloned when they invoke the clone subcommand. Click as a progress bar utility that can help us do that. So we define the clone subcommand with the source and disposition arguments. Let's say we have a list of files that we want to clone. We pass that list to the click.progressbar context manager which returns an iterator. And as we iterate on it and download each file, click will show the user a progress bar which will look like this. So we invoke the clone subcommand and as each file is downloaded, the user gets to see a progress bar. Another use case is to persist user specific configuration to a file. For example, we should persist things like username and email in our application folder when the user sets them using the config subcommand. Click provides a function which can help us do this. So we define a config subcommand with the key and value arguments. Let me get the application folder path for small get using the getappdir function. We create the path if it does not exist and finally store the user specific config settings in a file called config using the getappdir function. This function makes sure that the CLI follows the xgg spec and since the function is also cross platform, it will turn the most appropriate path on Windows, backhose, or Linux. And it looks like this. I set my username to Vinayak and if I do a cat on the config file for small get, it shows that my username is savedev. You should also page large CLI outputs so that the user can scroll through it instead of printing it all at once. We should do this for the large commit log that does log subcommand prints. Click supports paged output by calling a terminal pager program like more or less. So again, we define a log subcommand where we can use a click dot echo by a pager function to display the log stream. And it will look like this. Do small get log and we can scroll through the large commit log for this project. We should also call a file names that are added or modified when they are printed using the status subcommand. Click also supports adding color to text and it uses a colorama package underneath to do that. And colorama in turn adds and see escape sequences through text to color it. And to do this, we define a status subcommand. Let's say we have a list of files and their status. We can use the click dot style function to add a foreground color and make the string bold. And finally, we output it using click dot echo. The nice thing here is that click will auto-strip these escape sequences as the output is redirected to a file, for example, a log file. Because when we do that, we would not want to look at incomprehensible escape sequences while figuring out what went wrong with that application. And it looks like this. You can see that a new file called a.txt has been added and it is colored green. And sometimes we might also want multi-line input from the user. For example, asking the user for a commit message when they invoke the commit subcommand. Click supports launching editors for this use case. It will automatically open the user's defined editor or fall back to a sensible default. So again, we define a commit subcommand with a message option. And if the user doesn't use a message option, we, when the subcommand is invoked, we launch an editor to get the commit message. It looks like this. I do small hit commit and it opens VIM, which is my default editor. I enter the commit message and it gets committed to the log. We can also ask users for one line input using the click.prom function. This can be used for the push subcommand to ask the users for credentials to push files to a remote remote. So we define a push subcommand with two arguments, the remote repo we want to push to and the local branch that we want to push. And then we use the click.prom function to ask the user for the username and password. The return values will be stored in the username and password variables. Notice that for the password prompt, we said hide input is equal to true. So the input text that the user enters for the password doesn't get printed. And only click use this get pass from the standard app to do this. And get pass terms the echo setting off using the term iOS module, while the password is being entered. And it'll look like this. I do small get push origin master, enter my username, and then enter my password, but you can't see it. And it pushes the files to the remote repo. And we are done. We have a small get CLI with the help auto-generated. Click also lets us test the CLI set we write. You can use the CLI runner class to invoke the subcommand in a CLI and check the result against the expected output. And these are only a subset of features which click has to offer. You should totally check out the click docs to look at more awesome things that you can do with click. Finally, it's packet small get. To do this, we just create a setup.py file outside the small get module and add a console scripts entry point to it. The console scripts entry point allows Python functions to be registered as command line programs. You can check out the code for small get in the GitHub link at the end of the slides. And once we package a CLI, we can also push it to PyPI so that other people can install and use it. To do this, we just create a source distribution and a wheel using setup.py and then upload them to PyPI using twine. Now that we know how to write CLI in Python, let's briefly talk about the CLI user experience. As mentioned earlier, we are operating in a very constrained design space in contrast to graphical user interfaces, which offer a lot of visual cues and guidance to the user. There are some principles that can help us create a nice UX for the CLIs that we write. The first one is to keep things simple and follow the UNIX philosophy of doing one thing and doing it really well, writing programs so that they work together using the redirection operators and handling text streams. Following the UNIX philosophy makes sure that there are no surprises when the users interact with our CLIs. And the second one is making features discoverable by being forthcoming about them. Kind of similar to the cues that the graphical interfaces offer. Some things that can help us do that are storing the user's command line history as they use our CLI and letting users search through it by maybe giving them a search interface or suggestions for auto-completion based on that history. Amjit Ramanujan talks about this in detail in this PyCon 2017 talk, which you can again find in the resources. He also talks about Prom Tolkien, which is a Python package written by Jonathan Slendis that can help you implement some of the history and auto-completion features we just talked about and do a lot of other cool things too. It's used by Python and all the DB CLI tools. Finally, we are at the end of our CLI journey. I hope you got a lot of touch points to the CLI ecosystem that you can now explore further. I hope you also got an understanding of how terminals in CLIs work and how to write CLIs using Python. This slide contains links to some of the resources mentioned throughout the talk. And the slides are themselves available at the first link. If you have any questions, let's chat on the talk guide to CLI's Discord channel and you can also reach out to me on Twitter at botics.com. Thank you. Well, that was quite an interesting and insightful talk by Vinayak. He covered a lot of core concepts. Now it's time for questions and answers. So yeah. Okay, people are presenting you for your talk. That was really nice. I think we are good to conclude then Vinayak. Thanks a lot for joining and delivering such an awesome talk. I see there's a question. Are there complications supporting Linux, Mac, Windows or supporting different shells like ZSH, FSH? Can you elaborate or can you repeat the question? Yeah, I'll repeat. Are there any complications supporting Linux, Mac, Windows or supporting different shells like ZSH or FSH? So like the first part of the question, I think as long as your CLI tool is written in pure Python, it should work across Windows, Backwards and Linux. And if you just click, like most of the click features are cross-platform. So that should be good. But I'm not sure about the shell part. So I'll have to read up on that and I can get back to you on the channel. Right, thanks for your answer. I think we are good to conclude then. Thanks Vinayak. Thank you and Mohan. Have a nice day. Enjoy the conference. You too.