 So, welcome to this talk. Let's start with a simple question. How many people in this room are C programmers? Raise your hands. So, let's start with the quiz. What does this code do? It prints something. What's that? Unix. The answer is Unix. And if you don't understand this code, it's fine. It's tricky. It was the winner of IOCCC competition in 1987. And when it won the competition, it was down as the best one liner ever received. It was written by David Korn. Trying to maintain KSH code base is like trying to debug this type of code. It's written in a very hand-optimized way, and that's what this talk is about. So this talk is about AT&T conchshell. When I say KSH or conchshell, I mean AT&T conchshell. I don't mean the variants, which are MKSH or PDKSH. And it's going to be a brief status update from new upstream maintainers. That's me and Curtis. And I just have 20 minutes. So I will avoid going into the deep details. So this talk is going to be very shallow. And I gave this talk at 4th damn this year. So if you have seen this, my talk before, it's pretty much the same talk. My name is Siteshwar. I work for Red Hat. I'm currently the maintainer of Bash and conchshell at Red Hat. In the past, I have been involved in a number of projects that was like fish shell, sealfish OS. And currently I'm a swim in KSH. So what is conchshell? It's a very old shell. It's development started somewhere in the early 80s. Initial implementation was based on born shell. Even if you look at the source code today, you will see mentions of Sifon born in source code. So let's start with very basic question. Why try to keep the shell alive? What's the point? So let's have another quiz. This is very simple Bash script. And there's a bug in the script. And the bug is specific to Bash. What's the bug? Yes, Lukash is right. Because he has seen this code before. So the bug lies here. Bash is going to fork the while loop in a separate process. And you are going to lose the value of bar variable. Why I'm showing you this code? Because if you run exactly the same script under KSH, it will work as expected. And it turns out that even though it's an old shell, it still has a better language specification. It's much more powerful than Bash. It has better position compliance. And it is still the fastest project shell. Why do I say this? Let's look at an example. This is very simple script which runs a for loop like 10,000 times and it runs a sub shell inside the for loop. And when I run this script on my system under Bash, it took like 2.9 seconds. On G shell, it takes 2.4 seconds. With conchal, it takes like 78 milliseconds. It's so fast. No one uses it today. Why is that? Last year I was writing tests for Vance built-in and I came across this bug. And I reported this bug in GitHub upstream. And when I reported this bug, someone commented on my bug that exactly this bug was reported to previous maintainers 20 years ago. They never fixed it. And this is not the only bug. There are many such bugs hidden there. And I got involved in KSH somewhere in mid-2017. And this was the situation back then. The previous developers have left. Either they have retired or moved to some other companies. Codebase has origins going back to the 80s. Very old codebase. The tooling is old. We didn't have a revision control history. Like the whole source code was taken and dumped on GitHub without any revision control history. There are a few meaningful comments in the source code. I have some examples of them. The code was tricky. The test coverage was bad. There were lots of old bugs like the one I showed in my previous slide. And this is a legacy codebase so you can imagine the situation. So how do you deal with the situation? How do you maintain the un-maintainable? The answer is simple. You don't change anything. And this is like, it looks a bit funny, but it actually works. This approach works. If you care about stability, the code is tricky and your test coverage is bad. If you try to fix a bug, it's very likely you will introduce more bugs. So how do you end this chaos? You just stop changing things. You only fix critical bugs. And this approach sort of works, but it comes with a side effect. It kills your software. If you keep saying no to your users, eventually there will be a day when no one will ask you to do anything. Am I speaking too fast? So let's start by choosing some goals. Our primary goal is to create a modern but backward compatible version of KSH. We want to make it easier to contribute. We want to have better tooling around it. We want better build system. We want better CI. We want better documentation system. We want to improve POSIX compliance. KSH is largely POSIX compliant, but there are still some corner cases. We want to fix those cases. And we want to have better test coverage. In short, we want to bring this code base to 21st century. Now let's start working towards the goals. This is my favorite comment from Ken Thompson. One of my most productive days was throwing away 1000 lines of code. He followed this advice. We threw away more than 500,000 lines of code. There's a lot of code. Let's look at the statistics. The last release that came out from Bell Labs had more than 660,000 lines of code. Currently in KSH we have 134,000 lines of code. That's a significant reduction. So what did we drop? KSH actually exists even before POSIX was defined. So it has support for non-POSIX operating systems. And to support those non-POSIX operating systems, it had re-implemented many POSIX functions. It had entire subsystems for efficiency purpose, like VM, local, local. We just dropped all this code. It turns out that KSH was not a single package. It was part of a bigger package, which is called ASD. And what we did was we just took out the KSH code and we threw everything else. And we threw also through the old build system. The old build system was a combination of NMAC and if feature exists. Anyone heard about NMAC here? Actually, there are two different build systems with NMAC. One is the Microsoft one and the second is AT&T one. So I'm talking about the AT&T one here. And it was extremely complicated. No one understood it. It had bugs. The builds were slow. We just threw it away and moved to mess one. And we managed to improve build time by a factor of 35. On my system, the old build system used to take more than five minutes to compile KSH. These days, it takes like less than 10 seconds. There's a significant improvement. So now you are dropped like 80% of the code and you are left with 20%. What do you do with it? You start refactoring it. What about this code? What is it doing? It's doing string comparison. It's trying to compare paths on the slash directory. It's doing this. Assigning variable fd based on this path. This code is so much easier to read. Apparently, what you see on the previous slide, this is a micro optimization. And this is like 1980s type of micro optimization. This is probably the fastest way to do string comparison. But it's also very hard to maintain this type of code. So we just refactored it. So this code is so much easier to maintain. And so much easier to read. How about this if condition? What does it seem to do? So the first if condition is checking if the attribute is set, if those two attributes are set. And the second condition is checking if those attributes are not set. Anyone wants to take a guess what is it doing? It's contrasting. It checks if either of those attributes are set but not both. This is a comment. This should never happen. Guaranteed by design and got sacrifice. It was actually happening. There was a bug in this code. We had to fix the bug. In Fedora 30, GCC was the base to version 9.0. And it broke KSH-bell in upstream. And the file that I was debugging had this command at the top. You may think it should be simpler, but you shall be confused anyway. So it's full of these tricky commands and tricky codes. And we need more developers. If you like to work on crazy problems, try to refactor this code. It's full of crazy problems to be solved. And our source code is hosted on GitHub. The name of our repository is AST. It's not KSH due to historical reasons. This is a code from book Practical Reusable Unix Software. And this book talks about the AST code base that we took over. And this book was in mid-90s, 1994, 1995. So this is the code. It was at first a small effort with about 10 people. But it has grown gradually over time to 25 people. Most are researchers who create prototype software. So there are two key takeaways from this code. The first one is there were 25 people working on this code in mid-90s. And these days it's just two. So now you realize why we dropped all this code. And the second thing is those people were scientists. They created prototype software. How do you differentiate between a scientist and a programmer? A computer scientist is someone who looks on one side of the street while crossing one-way street. A programmer always looks on both sides. So what is the reference I'm making? The reference is about CI and testing. This is our Travis dashboard. And so we test every commit on Fedora, OpenCuser, Ubuntu, Debian, 32.2.2, macOS, and all the internal scripts are run through shellcheck. And we also test on FreeBSD. Unfortunately, Travis doesn't support FreeBSD. So we had to build our custom CI to test on FreeBSD. This is our test coverage. Our test coverage is currently hitting around 74%. It's not a great number, but it used to be worse. Our test coverage was less than, I would say, 50% when we started. So we have significantly improved the test. And apart from our unit test in Upstream, we also run some third-party tests. We run tests from KSHDB in Upstream and also a set of tests that are derived out of US shell tests. And we also execute all Red Hat internal tests on every Upstream commit. Unfortunately, results of these tests are not public. We take care of them internally. And we need more tests to us. This code is full of crazy bugs. So if you like to test software, just test it, you will find many, many such bugs. This is our CoVarity defect rate graph. We started running CoVarity in Upstream in early 2018. And our CoVarity defect rate used to be above 2. And these days, it fluctuates between 0 to 0.5. That's a significant reduction. And this was the situation after dropping 80% of the code. Before we dropped the code, the defect rate was around 7. So we fixed many defects just by dropping the code. And last but not least, we made it simple to try. You don't have to compile it yourself. We provide packages for Upstream, for development builds. We provide packages for Fedora and CentOS through Copper. If you don't use Fedora or CentOS, if you use Debian, choose our Rubin 2, you can subscribe to our OBS repos. Both Copper and OBS repos, OBS package builds are updated on every Upstream commit. And we need more packages. And you don't have to be a crazy package, because we have simplified the build process. You need just these two commands to build KSH nowadays. And we made a beta release last month. If you don't sit in your favorite distro, please rebase to the latest version. The guidelines for packages can be found on this link. And since we are talking about package maintenance, I have to thank our downstream package maintenance. These are the people who have already rebased KSH to the latest version in their respective distros. So that's Boyan from Debian, Sai from FreeBSD, Ali from Arch Linux, Mike from Gen2, Ryan from MacPort, Stig from Magia. We will now reach our end users unless people start rebasing KSH in their distros. So thanks to these people, we are able to reach our target audience. And this is my last slide. And I will end my talk by asking the most obvious question. The most obvious question is, we deal with very old code base. So this question comes to everyone's mind. The question is, is it too late to make a change? If you answer this question, yes, then you are right. If you answer this question, no, then you are right, too. It's your perception of the future that shapes the future. In my previous slides, I showed you a 20-year-old bug. This is the fix for the bug. We fixed it after 20 years. And we started working on this code base like 20 years ago. And people told us that you won't be able to deal with it. There are too many crazy problems there. The code base is too old. You will never make a release out of it. We deal with this code base every day. No one realizes the problem more than us. And we made a bit early last month. We will make a stable release, too. We realize all the problems that we are facing today. But despite all these problems, this is what we still believe in. Change is good, better late than never. Thank you for listening. I'm curious if, hi. Are there any plans to be packaging this for Fedora proper without copper in the main repose? I have the latest beta version in Fedora. If you use Fedora, you get the latest release that we made last month. Nice. You cited at the beginning one of your major motivations for Cornshell's performance. And I was curious with regard to the testing that you're creating, whether or not you're creating performance testing to make sure that you're not having performance regressions. We are not doing it yet, but that's a good idea. We should do it. Hey, great talk. As somebody who used KSH for quite a long time, back in the days, PD KSH and then my BSD Cornshell, I think one of the things which strikes me is that we have really high fragmentation of different distributions of Cornshell. So I'm kind of curious as to what your thoughts are on how to make this better. Because I think one of the things that it really leads to is, for example, things that Bash has and ZSH have like ShellCheck, we have pretty poor support in Cornshell because we kind of have this huge fragmentation and it's really hard for people to standardize on one thing that how Cornshell should work, what are the different nuances. So I'm curious how you think we can get people to use one Cornshell or at least kind of generally as an industry standardize on one Cornshell. So what's the question about, is it about like there are different variants of Cornshell and how do we move people into using this shell? Mostly I feel if you think about things like the BSDs, their fragmentation has hurt them a lot in some ways. Like one of the reasons Linux is so dominant is, as well as its benefits, BSD is so fragmented. And I feel one of the reasons people don't use Cornshell is because it's so fragmented and it's really hard to find the tools that you want for the particular Cornshell you're using and so on and so forth. So I guess my question is not so much what are you going to do to make people move, but how can we make this one as attractive as possible? How can we bring things like ShellCheck? How can we bring other things which make this particular Cornshell attractive to the project? I would say if you look at the other variants of KSH, they are still slightly, I mean the AT&T Cornshell, it still has a better language specification. It has still more features than other Shells. Other Shells are like, they don't have exactly, they are not at that level, they are at this level. So still if you look at the feature set in other variants, it's still lower. They don't have all those features that are provided by AT&T Cornshell. Sure, but that doesn't mean people are going to use it. Yeah, so the one thing was the performance advantage. I think other Shells don't have that. But I'm talking about other kinds of Cornshell. I don't think even they have it. If you try the same example under that same KSH or PDKSH, it won't run as fast. Try it. So I guess that's it. Thank you for listening.