 Right, so let's give a warm welcome to our next speaker here, Suthu, who's going to be talking about squeezing water from the stone corn shell in 2019. Good morning. Welcome to this talk. Let me start with a quick question. How many people in this room are C programmers? Raise your hand. C programmers. Do you program in C? So let's start with a sequence. What does this code do? It prints something. What does it print? Unix. It prints Unix on a CD out. If you don't understand this code, it's fine. It was the winner of IOCCC competition in 1987. And when it won the competition, it was turned as the best liner, one liner ever received. It was written by David Corn. Trying to maintain corn shell is like trying to debug this type of code. Very smartly written code, very hand optimized code, but it's very hard to maintain. So what this talk is about? It's about AT&T corn shell. I won't talk about MKSH or PDKSH, and it will be a very brief status update from new upstream maintenance. That's me and Curtis, and it will be very shallow. I won't take a deep dive in the details. My name is Siteshwar. I work for Red Hat. I maintain bash and corn shell at Red Hat. I used to be an upstream maintenance official, and I also contributed to some other projects like selfishness. And currently I'm upstream in KSH. So what is corn shell? It's a very old shell. It used to ship with Unix, and it's very old. The first version I came out somewhere in the early 80s, and its initial implementation was based on original born shell. So the first question that comes to our mind is, if it's so old, why try to keep it alive? So let's have another quiz. This is a valid bash script, and there's a bug in this script. What's the bug? It should be trivial if you're programmed in bash. The bug lies inside this assignment. Bash will fork the last command in a separate process, and this variable will be lost. Why I'm showing you this code? Because if you run the exactly same script in KSH, it will work as expected. And it turns out that even though corn shell is slightly older than other shells, it still has a better language specification. It has support for discipline function namespaces, compound data types, nms. It has better project compliance, and it is still the fastest project shell. Why I say this? Let's look at an example. This is a simple script. It runs a for loop like 10,000 times, and it runs a sub shell inside the for loop. Bash takes like 2.9 seconds. G shell 2.4 seconds. Corn shell takes 78 milliseconds. It's so fast. But no one uses it. Why no one uses it? A few months ago, I was writing tests for built-ins, and I came across this bug, and I reported this bug in upstream. And when I reported it, someone responded on this bug report saying that exact same bug was reported 20 years ago to previous maintainers. They never fixed it. Why I'm showing you this bug? Because this is not the only one. There are many such bugs in the code. And I came in the scene somewhere in mid-2017, and this was the situation of KSH upstream back then. Previous developers have left. Codebase goes all the way back to the 80s. It has old tooling, no revision control history, no commands in the source code, tricky code, backtest coverage, lots of old bugs. Everything seems wrong about it. How do you deal with it? How do you maintain? Don't maintain it. And the answer is very simple. Don't change anything. Because the code is tricky, and your test coverage is bad. Every time you attempt to make a bug fix, you will introduce new bugs. So how do you deal with it? Don't change anything. And do only critical bug fixes. And if you care about stability, this approach sort of works in practice. Because if you don't change too much code, there's a very less chance there will be regressions. But this approach comes at a cost. It kills your software slowly. So how do you bring upstream to a healthy state? Let's choose some goal and start working towards the goals. Our top priority is that we want to create a modern but backward compatible version of Conchill. We want to make it easier to contribute. We want to improve its tooling. We want better post-compliance. Conchill is largely post-compliance, but still there are some gaps in its compliance. We want to fill those gaps, and we want better test coverage. In short, we want to bring this code base to 21st century. How do you deal with it? Now let's working towards our goals. Ken Thompson once said that one of my most productive days was throwing a thousand lines of code. We took this advice very seriously. We threw a more than 500,000 lines of code. We dropped almost 80% of the code. What did we drop? This is some statistics. The last version of Conchill that came out from Bell Labs had more than 660,000 lines of code. Currently in upstream, we have 137,000 lines of code. What did we drop? It turns out that Conchill was not a single package. It was a sub-package of a bigger package called AST. AST had all these things. KSH was dependent on these functionalities. It has support for non-position operating systems. It has re-implementation of lots of position functions. It had entire subsystems like VMLoc. It's a memory subsystem. It had its own AST-specific local subsystem. We threw it all of it. This was constituting almost 80% of the code and we just threw it away. We just took out the KSH sub-package out of this code. We also threw in the old build system. The old build system was a combination of a build system called Nmake. A feature detection system called IfFeatureExists. The build system was complicated. No one knew how to debug it. Builds were slow. We threw it away and we switched to Meso. We managed to improve our build time by more than a factor of 30. The old build system used to take more than 5 minutes to build KSH. Right now it builds in 10 seconds. That's a decent improvement. You are left with 20% of the code. What do you do with it? Start refactoring it. For example, look at this code. Can anyone make some sense out of it? It seems to be doing something strange, but it's doing something very simple. It's doing this. It's checking for paths under .slashdev directory. We refactored it. It's checking for paths under .slashdev. Go to .slashdev.ft. Otherwise, if the path is strd in strd or strdr and takes some action based on it, what you see in the previous slide is apparently it's a micro-optimization. It was useful maybe somewhere in the 80s or maybe early 90s, but we can just rely on compiler optimizations now. We care much more about readability. This code is much easier to maintain. This is another tricky code. What about this if condition? What does it do? The first condition is checking for an attribute, whether an attribute is set, whether either nvl... It's checking if either nvl just attribute is set, but not both. The second condition is doing something strange. It will actually fail if both the attributes are set. This is my favorite comment in the source code. This should never happen, guaranteed by design and gotsacrifice. It was actually happening. There was a bug in this code. We had to fix it. Don't rely on gotsacrifices. Always test your code. And we need more developers. If you like to work on crazy problems, try to refactor this code. Our source code is hosted on GitHub. And the name of repository is AST. It's not KSH due to historical regions. We appreciate any patches. We care about stability, so we need better CI. And this is our Travis dashboard. We test every commit on Fedora, OpenSuser, Ubuntu, Debian, 32 boot Ubuntu, Mac OS, and all the internal scripts are run through shellcheck. We also test on FreeBSD. Travis does not support FreeBSD, so we had to set up our own custom CI. Right now, it only supports FreeBSD, but I hope we'll be able to add more operating systems there. Maybe OpenBSD, NetBSD, and some variant of Solaris in future. How does our test coverage look like? Our test coverage is currently hitting 65%. It's not a great number. I'm not proud of it, but it used to be worse. And in last one year, we had added hundreds of new tests. And this number is slowly going up, and it will get better in future. Also, on some third-party tests, there's a debugger called KSHDB, which had some tests, which is written in KSH, and there's a shell collage. It had some set of POSIX tests. We run all these tests in upstream. And Red Hat has hundreds of internal tests for shells, and we execute all of these tests on every upstream build. Currently, these tests are not public. We have some plans to make them public, and there's nothing concrete yet, but maybe in future. And we need more testers. If you want to help us in testing, please help us in testing in upstream build. We appreciate any testing, whether it's manual or if you want to help us in writing unit tests. We have put a strong focus on defects identified by static analysis tools. So we have fixed warnings identified by Clang Analyzer, and we also run covariate in upstream. This is the covariate defect rate graph. It's slowly going down. Our defect rate used to be about two, around a year ago, and it's currently around 0.6. And last but not least, we made it simple to try. We are doing builds for REL and Fedora in copper. If you use REL and Fedora, you can subscribe to our repositories. If you don't use REL and Fedora, if you use open-source or Ubuntu, you can subscribe to our OBS repositories. Both copper and OBS repositories are updated on every committee in upstream. And we need more testers. There are hundreds of Linux distros and there are different BSDs. We can't deal with every one of them. So we need help from packages. You don't have to be a crazy package. It's very easy to build KSH. You have to run these two commands and the build should be ready within 10 seconds. This is my last slide, and I'm going to ask a very relevant question. And the question is, is it too late to make a change? And the answer to this question is really a matter of perspective. It's really up to if you believe in making a change or not. In my previous slides, I showed you a 20-year-old bug. This is the fix for the bug. We fixed it after 20 years. This shows that due to current maintenance, we don't care if a bug is two days old or two decades old. We just fix it. And we realize that we have taken over a very big problem. It seems like an impossible problem. Like 30 years old code, no git history, bad test coverage, tricky code, no commands in the source code. It seems impossible that we'll be able to deal with all these problems. And it seems impossible that we'll ever make a release. And it seems impossible that people will start using KSH again. But in reality, when things start happening, impossible slowly becomes possible. Thank you for listening.