 In the last couple of months I've made several videos highlighting some of the really cool shell utilities on Linux, and you guys have asked for more, so continuing along in this series of videos, today I wanted to highlight three rather simple shell utilities that you will find very useful. Those utilities are Shuff, Sort, and Unique. So let me switch over to my desktop, and I'm open to terminal and I've zoomed way in, so hopefully you guys can read the commands I run today. Let's start with the Shuff command. What is Shuff? It generates random permutations. Basically it takes whatever input you give it, and it shuffles things around so they're in a random order. For example, if I ran Shuff on my .bashrc file, it takes my bashrc, and it reorders the lines, it shuffles the orders of the lines in a random sort of way, so it just mixes everything up. Now if I wanted to do something a little more useful, I would do something like sequence 10, take a list of numbers, one through 10, right, I get a sequence, and maybe I want a random number one through 10. Well, I could do Shuff, and I get a random order, but I could do Shuff dash in for the number of occurrences, and I only want one random number sequenced one through 10. And there you go, if I ran that again, you know, it would always be a random number one through 10. And of course you don't have to do just one random number, maybe I want five random numbers of one through 10. Or maybe I want five random lines from my .bashrc. And then I just got five lines randomly selected from my bashrc outputted here in the terminal. Now of course you can run the shuff command on a file, and then have it output to a new file, for example, let me create a file, I'm going to get a sequence of numbers one through 10 again, and let me write that to a new file, I'm going to call it test.txt. And then what I could do is I could shuff test.txt, and then give it the dash O flag for output, I want you to output the shuff on the test.txt to a new file, test2.txt. And then if I cat test2, you can see that it wrote the output from this to that test2.txt file. Another example of shuff is shuff dash in for number of occurrences, I want three occurrences this time, and then I'm going to give it the dash I flag, and this is an inclusive range, and the range I want is one through 10, and then I'm going to give it dash dash repeat, meaning I don't mind if we have repeated instances in the shuffle. And if I run that, it takes the numbers one through 10, it takes a range of numbers one through 10, it shuffles them and gives me three occurrences, and you can see we can get a repeated occurrence, if I didn't want a repeated occurrence, if I wanted only unique numbers, I would get rid of dash dash repeat, and run that same command. The next command I want to talk about is the sort command. What is the sort command? It sorts lines of text. So an obvious example, let's sort my bash rc file, and it does this alphabetically of course, so it basically took every line of my bash rc and reordered it alphabetically. And if you wanted to, you could give it the dash r flag for a reverse alphabetical order. Now the end of it is a bunch of spaces, but if I keep scrolling back up, you can see it reversed the orders of the lines. Now if I ran a man page on sort here, you can see there's several flags with sort. Some of the more common ones include the dash f flag, that's the ignore case flag. So it folds lowercase to uppercase characters. Another common one is the dash i flag for ignore non-printing characters. So if you're outputting something that has a bunch of non-printable characters, the dash i flag will get rid of the non-printable garbage from the output. One interesting thing you can do with sort is you can give it a field separator. Kind of like you do with awk. You give awk a field separator, and then based on that field separator, you pull out a certain column. With sort, you give it a field separator, and then sort by whatever column you tell it to sort by. For example, if I want to do this command here, I can do sort, and then dash t, and this is for field separator, and we're going to use a colon as a field separator. And then I'm going to give it the dash k flag, and then I'm going to give it a number of the column that I want to sort. I want you to sort by the third column, and then we need a file, and the obvious file to do this on would be slash etsy slash pass wd, because that particular file is nothing but colon separated values. So if I run that, it sorts that file by the third column, which the third column is nothing but numbers. It starts at zero, and then goes all the way to 65,000 here, and it sorted them based on the third column. Not the first column, if it sorted by the first column, it would be by name. And you can see it definitely sorted by the first column. If I wanted to, I could sort by the first column just to show you the difference. Or any other column, if I could sort by the second column. But the second column is all the same value as all the x. Probably one of the most common uses of sort is sort with the dash u flag for unique. So if I did a sort on my dot bash rc one more time, but this time I'm going to give it the dash u flag for I only want unique lines. So there's several things in my bash rc that probably appear many times. Things like if and fi, for if statements, that's the beginning and ending of a if statement. And if I sort these lines, and if I go alphabetically up, I see one occurrence of fi. Now I know in my bash rc there's probably a dozen different occurrences of fi at the beginning of a line. So it only gave me one instance of it, and instead of a dozen instances of it. If I ran that command without the dash u, you'd actually see this in action. Let me scroll back up to where fi is. Yeah, you see there's actually four appearances of fi, three appearances of spaces with a fi. So the dash u flag just gets rid of the duplications. And just like the Chef command had dash o for output, you also have dash o for output with sort. So if I did sort, well sort my dot bash rc and then dash o for output. We're gonna write that to test2.txt. Remember that was a random order of 1 through 10 earlier. But now if I catted test2.txt, now it's actually my bash rc that was sorted. Another cool thing you can do with sort is you can actually do kind of a chef with sort, you can do sort with the dash r flag for random. So if I did sort dash r on my bash rc, it shuffles the lines randomly. It's not necessarily random because you do get groupings like this, where you get a lot of fi statements together. So it's not exactly like chef. It's kind of close, but if I did man on sort to see what is going on with the dash r flag, you can see it's a random sort, but it does group identical keys. So it's not quite as random as chef. So now we've talked about chef. We talked about sort. Let's briefly cover unique, which is UNIQ. And typically with unique, you use unique in conjunction with sort. Typically you run sort and you pipe it into unique. So if I wanted to sort my dot bash rc and then pipe that into unique, what this does, it sorts alphabetically my bash rc, and then it removes the duplicated lines by piping it into the unique command. And if I give you an example of how that worked, remember I had four different instances of fi in this file, but now I only have the one line returned because again, with that command, we piped it into unique. We got rid of the duplicated lines, only unique lines are returned. Now typically this sort and then piping it into unique with no other flags. Typically you don't want to do this because honestly you could just do sort dash u because remember the unique flag with sort, it actually accomplishes the exact same thing. Remember we only have now one instance of fi. So again, you only get the one instance of each line, it gets rid of the duplications. But some of the flags that you will often use here with unique, typically you'll sort something and then pipe it into unique and you'll give it the dash u flag. And what this does is it only returns unique lines. And I don't mean it only returns unique lines one time. I mean, if the line is duplicated, it doesn't return any of them. So now when I sort this, I go into the Fs because it's alphabetical. There is no fi anymore, right? It got rid of that because it appeared four times. It's not unique. It doesn't return any of them. It only returns absolutely unique lines. And if I wanted to do this in reverse, I could give it dash d for only return duplicated lines. And these are the four lines in my bash RC that appear several times. You got spaces and then the closing brackets or braces there and then fi with the spaces in front of it and fi at the beginning of the line. Another very useful flag is the dash C flag with unique. So if I do unique dash C here, what this will do is it will give me a number of occurrences for each line. So it prints out every line, but it gives me how many times that line appears in the document as far as the duplications. You see fi appears four different times. Fi with the spaces in front appears three different times. Now that's really useful, especially if you wanna sort by the most frequent number of occurrences, how you would do that is you would sort it and then you would pipe it into unique with the C flag and then once again, pipe it back into sort and give sort the dash n flag and the dash r flag for reverse for number of occurrences in reverse. And if you do that, now it should sort by the frequency. So if I get to the very top here, you see a blank line, an empty line was 48 times. And then the next most frequent line was the fi line. And let me clear the screen. So that was just a little bit of what you can do with Shuff, sort and unique. They're very simple utilities, but very powerful. They allow you to do a lot of cool things and manipulate texts and they're very useful in shell scripting. Now, before I go, I need to thank a few special people. I need to thank the producers of this episode. I'm talking about Absi Gabe James, Mitchell, Paul, Wes, Akami Allen, Chuck, Kurt, David, Dylan, Gregory, Haiko, Mike, Erion, Alexander, Peace, Archon, Fedora, Polytech, Raver, Red Prophet, Scott, Steven and Willie. These guys, they're my highest tiered patrons over on Patreon without these guys. This episode about Shuff, sort and unique would not have been possible. The show is also brought to you by each and every one of these fine ladies and gentlemen. All these names you're seeing on the screen right now, these are all my supporters over on Patreon because I don't have any corporate sponsors. It's just me and you guys, the community. If you like my work and wanna support me, look for DistroTube over on Patreon. All right guys, peace.