 So, good afternoon everyone. Good afternoon, it's half past one. Welcome back to the ones who had the show in the morning. Welcome to the ones that missed the show about physics and quantum and a number of strange things that we had in the morning. Everything that will happen in the afternoon is on that website. And my name is Nicolas. What we have in the afternoon is, well, we have Paul. 50 minutes of Paul McKinney. So, be prepared for that. Then we have a brief lightning talk and then parallel programming. So, we have an introval and then we have another three sessions about multi-core FPGAs, gearman, and discovering inner and parallelism and sequential programs. And immediately after that we have a panel and burst of a feather that I strongly encourage to stay. And I mean the panel will be all of us. So, ideally you have people sitting on the front and then you just pass the microphone. But the idea is to have the same level of communication that we had in the morning. So, that's me. That's all here. So, you just do this here. Thank you. And for someone who hasn't checked who is this gentleman on my right, he is distinguished engineer of IBM. He also has the role of CTO Linux, maintains the RCU implementation within the Linux kernel. And a lot of other things that you can read there. And the other thing is that his hobbies include what passes for running at his age along with the usual housewife and kids habit. I love the sort of bios that you write for others in third person. So, essentially the most important thing is how people is cynical about himself. And I think that Paul has a masters on that. And I probably will speak on behalf of Paul also that feel free to ask meaningful questions. Even meaningless questions, if they're humorous enough. I say meaningful questions. Okay, I guess I have an idea. And that's my sweet revenge every time that people tell that to me. Don't do statements, ask questions or yours. Thank you very much. Thank you. So, I'm actually really, I'm actually really happy to see the quantum computing in the previous talk. Can anybody guess why it's actually feel so good for me? Infinite number of cores, I suppose. Actually, the thing is. I'm not missing anything. Yeah, that's a good point. It's actually a kindness to the viewers, don't worry about it. The thing is that if we actually do get our heads around this quantum computing stuff, the stuff I've been doing people have been pointing about will seem dead simple, all right? So, I'm looking forward to that. Anyway, so is parallel programming hard? And often that gives you a knee-jerk answer. But if so, why? Why is it hard? I'm going to talk a little bit about my very early experiences with parallelism. I was involved with parallelism before I was involved with computers. Talk about, yeah, well, we'll get there. I've been doing parallelism for a long time. Why is the excitement now? And some lessons from the past. And again, we'll get where we get and we'll stop when we stop. But again, please ask questions and I want to make sure I'm talking about stuff that's meaningful to you guys. My first experience with large-scale parallelism wasn't a huge number, but it was there. So I'm a teenager in my parents' house, and the doorbell rings like a fool I answer it. And across the threshold come these five things about this tall, all dressed identically, looking identically, just running around the house. And my sister and I were, well, that's impressive. My sister and I were teenagers, and so the house was not at all child-proof. And all of them were running as fast as they could towards something they shouldn't be messing with. And so I, you know, but I can't figure out which one to chase because they're all going somewhere. And about that time, their brother and their sister and their two parents come across the threshold laughing like hyenas watching me as I told them like a fool to myself. It was the Anderson quints. They were making their way down from their home in Washington down to California. And for some reason, they had this very strange aversion to hotels, or maybe it was vice versa, I'm not sure. But they were stopping off at friends and friends and friends and we were in the latter category. And so they apparently got quite a bit of enjoyment out of their first introduction to most of the families they stayed with that they certainly did out of me. The thing was is that there were more two-year-olds and there were even with the brother and sister together with the parents. And I didn't notice the brother and the sister were good kids, but they weren't doing a huge amount of child care. And the parents seemed to be able to spawn off threads of consciousness as needed to deal with whatever situation was arising and with five toddlers, situations were arising quite frequently. So when people tell me that parallel processing is against human nature or against nature, something like that, I remember that experience. And it's not like that's the only thing. And we saw this, you could think back further in time. So we got this guy here who apparently lacks the ability to think concurrently. And look, Erg, I only one thing at a time, just wait till I'm done. Is he done? Yep, he's done. Now, if only the poor guy can then don't think concurrently, he might still be alive in the last panel. So I think that one could easily believe that there might be strong evolutionary pressure towards human beings that could deal with concurrency. And it's not just a matter of the occasional people that have quintuplets or ancient history like this, although I'm not sure how ancient this is sometimes, but some days, but whatever. The thing is, it does come naturally to human beings. And there's any number of other situations where it happens, any number of sporting events. American football, I suppose. People look down at football players. And I've been there myself. I can't give them a hard time for doing that. But so the guy in the field, especially the quarterback, is keeping track of 21 other players, plus the referees, maybe coaches as well. Now, I will admit that I've only seen one football player go on to be a parallel programmer. On the other hand, that guy was a very extremely talented parallel programmer. His name was Jack Slingwine. He's the guy that invented RCU with me. So it could well be that we've been taking the wrong approach to training parallel programmers. Maybe we should be looking at soccer players or something. Who knows? Teachers have to deal with the classroom of students. More than five, usually. Construction, there's a lot of workers doing things. And if they aren't keeping track of each other, people get hurt. I have some friends who have gotten hurt because they fail to keep track of each other. But the injuries are fairly rare. Drivers, lots of cars are keeping track of. And when I'm on a bicycle, I hope they're keeping track of bicycles as well. But sometimes they are. Sometimes they aren't. Air traffic control is another case where there's a lot of concurrency happening. And of course, emergency services, it could be any number of people, depending on what the type of emergency is. So the fact is that there's been a lot of concurrency and tolerance for it throughout human history. And we saw this cartoon earlier this morning. Same sort of thing. The plants, I'm afraid, are not only nonlinearizable, they are moving concurrently. So it's not just the human race, it's the universe. Thing is, just because concurrency comes naturally doesn't mean that concurrent programming comes naturally. And we'll talk a little bit more about that later. But there's some key difference there. Concurrency just reacting to life as it comes along in all those different stages and all those different directions versus writing something that's going to run concurrently and deal with ulcers and things. Those are two very different things. So this is what I've done with concurrency with computers, I should say. I left off some of my earlier sporting and other things with concurrency. Some distributed simulation in the late 80s. I worked with Sequent doing a parallel UNIX kernel in the 90s, a little bit with AIX, and most recently, of course, with Linux. So I mean, this concurrency is around even in computers for a long, long time. Why is everybody getting excited about it in the last three or four years? What's changed? Moore's law stopped. I heard there was another thing before that I didn't hear quite. Commodity implementations. Commodity implementations, Moore lost both of those. Very much so. To an extent. This is the party line. This is kind of a weird graph. Each spot is a Intel CPU. And I had to switch from MIPS to megahertz at one point just because they stopped using drystone MIPS. And if you go back further, they use multiple clocks per instruction. So no matter what you do, you have a funny graph. But as you can see, up 2003, something bad happened. All of a sudden, the clock frequency didn't go up any faster. That's the party line. Another part is simple economics. I mentioned earlier this morning about curing a stack of 5 CPU boards across the parking lot. That's what one of them looked like. Or that's actually an earlier model of one of them. It's got a pair of 3, 8, 6 processors on there. It's got the little gold things beside the processors are floating point accelerators. And a stack of five of those was three times the price of my house. As a result, there were just a very tiny number of people that were able to do parallel programming. I worked for a company that produced these things. And there's a couple other people in this room that did as well for that same company. And so that company had to pay us to play with their hardware because otherwise they didn't have any software, and people couldn't make use of their hardware. There were a few universities that got parallel processors to either these ones or other ones. And so some of their students, maybe some of the researchers, could play with them. And there were some companies that bought them and used them that might have a test system setting aside that maybe you could get time on when it wasn't otherwise being used. But these things were horribly expensive. How expensive? In 1990, if you got a low cost multi-processor system, it might only cost you $500,000. 2006, I was working with a grad student who bought a dual core Mac. The only reason he bought the dual core Mac was so that he could do a presentation on parallel processing using a parallel processor to drive the presentation. So what's happening is that in 16 years, the price of a modest multi-processor dropped from multiples of the price of a house to a fraction of a used car. And the last five years has been dropping still. I mean, you can get for way less than 1K, probably if you're willing to get something fairly basic, you get a multi-processor for $100 or less. As a result, where before, just a few people had the privilege of working with these things, now anybody, I mean, anybody can go buy them and use them. You don't have to be, you know, these are, these are, you know, you can get a multi-processor for the price of taking a modest sized family out for a nice dinner. Okay, it's gotten down to that point. It's consumer electronics prices. And so what that means is the multi-processor suddenly can be used anywhere. In 1990, you had to have a really good reason to do parallel processing, because it was freaking expensive. Now, you could use it on a whim. And so suddenly, there's a lot more demand for parallel systems, because they're cheap and commodity. And therefore, there's a much larger demand for parallel software, because you gotta run something on these things. And therefore, there's a shortage of parallel programmers. This is not the first time this has happened to this industry, though. Sorry, can I ask you just, I read somewhere that a guy from Intel, Anwar Golumar, can't remember his name. He said in a statement about three years ago that only 1% of programmers in the world are able to do parallel programming. Just read it on the blog. I wonder if you agree, disagree. Anyone think something about that? I'm gonna get to that in a couple slides. I didn't pay this guy. That is an excellent question. I mean, that's exactly on point. That's the right question to be thinking of. We've been here before. Instead of the great multi-core software crisis, we had the great software crisis. In the 1960s, we wanted a computer at all. It doesn't cost you hundreds of thousands of dollars. Around 1970, you could get the little mini computers. They were very small. 4K in memory. Tape drives is the only math storage. Model 33 teletype. You might get it for 25K, size of a refrigerator. In the late 1970s, suddenly, you could get small hobby machines based on microprocessors for, again, less than a thousand dollars. I had the good fortune to enter the software market at this time. I graduated about 1981. And it was fortunate because if you could spell a computer, somebody wanted to hire you. Unfortunately, what happened is that everybody was making promises. We'll use a computer for this. How hard can it be? Usually promises they couldn't keep. So I spent about five years in the early 80s keeping people out of jail. This has actually worked out really well because people are usually willing to pay fairly well to avoid breaking their contracts or possibly having legal issues. But the problem really was, again, that there was an acute shortage of programmers. And the thing is that this problem somehow got solved. How did it get solved? What happened? As to you guys have great hair, you should remember this time. I mean, some of you young guys, I can give you a pass on this, I'm sorry? It's the hobbyists. The hobbyists? Okay, so what the hobbyists do that helped? The hobbyists were playing with and becoming specialists. We also didn't go after this. We didn't go after the huge problems were thought to be really important at the time of the crisis. They did things that people needed to get work done. If nobody would intervene. Right, there was people were doing modest problems that modest programs that weren't 40 million line things that got something useful done that their friends needed that they needed that meant that they could get some useful work out of the computer. Yeah, yes? That's how we raised the level of abstraction and also some ways to go full GRs. You've got object oriented frameworks, you've got toolkicks. So we raised the level of abstraction to make it easier for people to solve problems more rapidly to learn how to start solving problems. That's my answer. Okay, so the answer was raising the level of abstraction. I didn't get all of those, but four GLs, object orientation, and a few other ones. And I agree with half of that and we'll get to that in a moment, but there's a good thing there in the back or, okay. All right. And we've been through this. So you guys did really well. Okay. They became really available. A lot of people play with them. A lot of hobbyists learned how to program. Maybe did the thing as obvious or maybe got hired by companies that needed it and we had useful software. And the same thing is going to happen with low cost multi-core systems. People will play with them. They'll learn ways to do it. Either the ways of doing it in the past or they'll come up with neat new ways. Probably some of both. And the problem will be solved eventually. The problem is that we're impatient and we'd like the problem solved today. Preferably yesterday. So, you know, what can we do? Well, one thing to do is to kind of split up, categorize the things that people are trying to do just to deal with the great small-core crisis. So there was the good, the fad, and the ugly. All right. So the good. I've got a fairly stiff definition of good here. Although it's actually a lot of things that way exceeded this. Way exceeded this. Okay. This is actually fairly modest for what really happened in the best case. But you need an order of magnitude, improvement, and productivity. Or an order of magnitude increase in the number of people able to use the machine. And in many cases, multiple orders of magnitude in both at the same time. This happened a lot in the 80s. We have the fad, which is that there was a lot of excitement at the time, but, I mean, well, if you want to get Barbara Liskoff really mad at you, tell her that you really appreciate how much her work influenced C++. All right. Just, you know, I actually did that, suggest not doing it. Yes? I'm just trying to remind you, Bertrand Mayer has this line of a similar, similar, was such an improvement on all of its successes. Simulant? Simulant? Yeah, simulant. Yeah, the... That's the 80s here, too. Yeah, the 80s was actually in the 60s and 70s, was when it started. The 80s was in the 80s. But Gareth Stroustup tells a story about talking to, I think it was Christian Nagard. I can't remember the name of the other guy, but there were two guys behind similar. And Garner was wondering, well, how does it feel to come up with inheritance? And the guy said, well, I'm not going to try to emulate the Norwegian accent. I'm sorry, you lose a lot by not doing it, but on the other hand, you'd probably lose more if I tried it, okay? So he says, well, it was like that Greek guy, you know, left out of the bathtub, he's yelling Eureka and ran down the street, you know, naked. Except that, well, us, we don't run down the street naked in this country. And besides, it was winter, and that would have been a bad idea. So, but yeah, there were a lot of things. There were a lot of things at the time that were going to save the world, that were high-minded things that were going to make a really big difference that people probably don't even recognize anymore. The third category was the ugly. It was in use then, it's still in use now, and it's just too freaking ugly to die. And it's also TOO, frankly, ugly to die, as opposed to just TOO ugly to die. I'll fix that later. Things like C, C++, C++ actually came a little bit later. Fortran is finally, I think, sort of dying maybe, but it's hard to say. You know, cobalt is still in use a lot, although I don't think there's so much use new code in it. Anyway, I'm not going to try to predict, I have some opinions, but I guess the mission for you, if you choose to accept it, if you really want to be part of this, is to figure out what you can do that would show up in the top category, or potentially in the bottom one, and avoid the middle. Remember. What you can achieve. Or what you can achieve. Yeah, but I bet, I bet we can get some stuff from Holy Core in the top category. I bet it's the things that people aren't really thinking that much about, and I suspect that it'll probably be some guy in a garage that just does it because it seemed like a bizarre thing to do at the time. But the things do exist. They're out there. I just don't know what they are myself. So this is my opinion of what the great software crisis was. And you can come up with any number of people and get a good argument going on it, I'm sure. But for the biggest one was a spreadsheet. And that was a higher level of abstraction. I don't know if you've ever tried to do a paper and pencil spreadsheet, but that used to be what people really did. Either that or chalkboard, right? And they'd have arithmetic errors. They'd have to go back and fix them. And if you want to adjust something, you have to, you know, and it wasn't, Xerox wasn't that great at the time. It was more, mimeographs and things like that, and it was painful. A spreadsheet meant you could just put the stuff in there. You get the answer immediately. You want to change something, change it. You can save off as many copies you want with slightly different things, and it was just great. Of course, there's some spreadsheets that look like the success was its own undoing, but you can misuse any tool, and spreadsheets are any tool. Presentation manager and the word processor. I mean, the thing is, both of these things, I didn't think much of it at the time. I mean, it's kind of like, yeah, a spreadsheet, you know. I saw it, I didn't think anything of it. And I really didn't think anything of it was presentation manager, and the word processor, not much either, right? But those are the things that really made the difference. Those are the things that really made it so that the computer was something that my grandparents used. If you had told me in the mid-70s that my grandparents would use a computer, I would have laughed at you. I mean, that's the most flight thing I could imagine doing, all right? Yeah, because of those things, by the time the early 80s came around, they were using computers. Computer-aided engineering was another big one. What it meant, there's a guy that was in charge of the labs at SRI when it worked there. His son got into computer-aided engineering. This was in the mid-day late 80s. He wanted to get a summer job, and they had a company named Dressman, so it went and applied. And they wanted him to do it on the table with paper and pencil. And he goes, well, you know, that's not really going to work for me, and you do this on a computer. Except that his computer, that portable computer meant you could like, I don't know what, and you could put in a pickup truck or something at that time. And so his computer had to stay at home, and they went back and forth on that. They weren't comfortable at all having this guy who was just barely out of his teens working at home, doing stuff that they were going to depend on. So they went back and forth and eventually agreed that he could come in on Monday and show them what he'd done the previous week and they would just see how things went. And if that didn't work out, then they'd be back to the drafting board for him. So they gave him his assignments. They came back in the next Monday with all the work done. So basically they had taken what they felt was a summer's worth of work and handed it to him. And because of the computer and computer-aided engineering, which again isn't all that high-minded, he was able to get all that done and actually turned out in a weekend, two days. So that's a couple hours of magnitude, all right? And a lot of the guys in engineering who wouldn't have been able to use a computer in the 70s and had no problem in the 80s with that kind of software. I suppose we could have a lot of fun going through picking on the various paths. I'm not sure how worthwhile it is, but there were just a number of things that were gonna save the world that you probably can't even find on Wikipedia anymore. The ugly C-language, set-off, curl, visual basic, yeah. Nothing's wrong with it, it's ugly, it's good, we use it. I mean, it's just too ugly to die and it's useful. I mean, so there was actually a computer software crisis before, this is the first one I lived through when I was around to know what was going on. There was one that happened about 20 or 30 years earlier. Okay, and it started off because computers were available at all. And I don't know if I never had been able to track down whether this story is true or not, but it deserves to be true even if it's not, all right? So, a number of organizations, including the US military, were having a heck of a time because how are you gonna get an experienced programmer when you've just got the first computers and there haven't been any before, right? It's not a matter of them being cheap and expensive, it's a matter of them existing at all. Well, the military being what they were, they got a bunch of their research psychologists on the job. And after a bunch of testing and questionnaires and all that, they found there were two classes of people who could be reliably trained to program, only two classes. Anybody care to guess those two classes? Go for it. Engineers and mathematicians. What was that again? Engineers. Engineers and mathematicians. You're close on one and missed violently on the other one, but that's not bad, that's actually pretty good. Any other guesses? Psychologists. Psychologists. I'm sorry, that would have been classic, but it didn't work out that way. I like that answer though. Football players. Accountants? Not a chance, sorry? Mental patients. Mental patients. Well, I guess that describes all of us, I suppose, but yeah. Kids. Kids. You know, the army wasn't, I don't think that was part of their test. It would have been interesting if they'd done it. I don't think they thought of that at the time. Any other guesses? Football players. Like I said, I only know one football player who went on to be a parallel programmer. I'm sure there are others, that's just the one I want to know. Okay, so engineer was close. Math mechanics was one of the classes. Guys that like fixed engines and stuff like that. Work on, you know, troop carriers or whatever. Any last minute guesses for the second group? Cooks. Cooks. That's an interesting one, I wouldn't have thought of that. They would more align with chemistry. One of the things that would ask women for the war effort. Do you prefer knitting or sewing on the one hand or cooking on the other hand? The ones that like knitting and sewing got sent off to do a bill of electronic equipment. The ones that like cooking got sent off to do chemistry. Very good, exactly. Exactly, musicians. And you know, I think that all the religious wars we have in computers today are between those two groups, mechanics and musicians. But those were the two groups, mechanics and musicians that the US military was able to reliably train to produce software in the 50s. Possibly. But that's, I wasn't there at the time. You might be right, I don't know. But I'm telling you the story as I heard it. There is a lot of mathematics and music. I think the difference is that musicians are into creating something, which you have to do programming. Mathematicians are more into analysis, which is okay if the program already exists, is kind of my experience. Whether that has anything to do with reality or not is an excellent question I don't know the answer to. So again, we're gonna have the same three sorts of things. And a lot of things, I think at the good side, the thing about those things on the good side is they're very close to the application level. The things that turned out to be fad were mostly low level things. I mean, they were high level. You gotta keep in mind that at this time C was a high level language, okay? To be a low level language, you had to be assembly. And that's kind of changed now C's a low level language. It's a lot higher level now than it was then, but it's, I don't know. Anyway, I suspect that a lot of the things that are really going to make it so that ordinary people can use multicore computers are gonna be things kind of like that. In fact, I think there was a camera which, pure article was, but they had an interview of the guys that parallelized Adobe Photoshop, okay? And you can do the same thing presumably again. But at that point, the people using it don't know or care that it's running in parallel. Just like the guys using a spreadsheet don't know or care what algorithm it's using to propagate the value, use as long as it does so. So I think that kind of thing is going to be the big deal. So I said earlier that concurrency is natural for human beings. Concurrent programming might not be. So what's hard about programming? Why is programming hard? A lot of people can't do it very well. And you can argue that even the people of those who are best at it can't do it very well. But there's a lot of people really can't do it very well, yeah? Because when you think about some things for themselves. Okay. Maybe you need to try to answer the everything you're used to doing that. Yeah, so if you're gonna do a computer program, you better plan for all the contingencies. Otherwise you got a bug. Whereas if you're doing something yourself or you're telling somebody else to do it, you can, assuming they understand what the heck you're supposed to do, you should be able to get something very general instructions or some general kind of a wish, not a plan, and get it done. That's certainly one of them. Yeah? The previous one about those programs was actually a large amount of short-term memory. Large amount of short-term memory. Okay. A whole lot of information for a short period of time. Okay. Yeah? Breaking the problem down in this piece of paper. And that, yeah, that ties in with the planning. In fact, my first hint that my oldest daughter might be good at programming, she actually is doing entomology. Real bugs are supposed to design bugs. But in third grade, they had her do a, tell me how to make a peanut butter sandwich, write down the steps for me, peanut butter sandwich. And typical kid will say, I'll put a piece of bread down, put the peanut butter on the bread, put the jelly on the bread and then put another piece of bread on top. Which point you've got pieces of bread with a charge of jelly and peanut butter in between them? She did 38 steps. Okay. So there's a lot to that, yeah? Yeah, so doing programming often requires a much more detailed view of the problem. A lot deeper analysis of what's needed in a lot more cases. Okay, so these are all hitting on, what I did to try to get this out was I talked to people who had tried programming, were able to do it, but didn't like it. And, you know, talked to them about through the reasons why they didn't like it and came up with these three. And we've mostly been talking about one of them. The last one. Which is that people expect to be successful despite fragmentary and infallible plans. In other words, not enough detail, not having a human being to interpret the higher level stuff and several other aspects of that that would come up, which are quite an insane disciple. Yeah, I think that's the most important one. The other two are that people expect that there's something intelligent that is gonna have common sense. And, well, there are some things that's, some embedded things that seem to be getting there to some extent, but usually that isn't happening with a computer. In fact, most of the time they portray them as tools because people don't expect tools to have common sense as opposed to another person. They also expect intelligent beings to understand their intent. And we may get there at some point with artificial intelligence, but they've been promising that for what 50 years now, so I'm not holding a breath. And so the thing is, though, and so those are the big three that I've thought of and there were a lot of variations as far as I can tell on the last one, most of this has not much to do with parallelism, with one exception. That's actually the one you guys talked about. If you're doing things in parallel, it puts even more stress on your planning. All right? Because things like deadlocks, if you think of them, are failures to plan. I mean, they're annoying and you'd like not to have to plan to that degree, but if you have a deadlock, it's because you didn't plan for the case of two people trying to acquire the locks in the opposite direction. So more planning is required and therefore there's some additional difficulty with parallel programming, but it's as near as I can tell, fairly small compared to getting to programming in the first place. Well, I'm gonna go through this in too much detail. This just talks about the reasons, I think you understand them because of the things we went through. Eliza is actually a early counter example, wagon bombs, I got the name right, little sort of therapy session program that was very simple yet passed the Turing test with a lot of people. But in most cases, yeah, you don't try. Anyway, the solution, by the way, on the planning is to let the computer do the planning. Inside the Linux kernel is a fair amount of that already. Things like lock depth are very helpful. I mean, there are pain in the rear sometimes because they make you plan when you don't think you have to, but they help. And various of the performance analysis tools help as well in being able to look at what's going on and work things out. There's also some of the theory improving systems, Promo and things like that that can help analyze a plan. But I think that in a lot of cases, increasing the ability of the tools will help out as we go forward. So okay, this is how we've talked about how you can get in trouble, how can you succeed? And there's a lot of ways I'm not gonna go through all of them or any even a large fraction of them. What's happened in the two organizations I've been involved in that have done parallel programming has been an apprenticeship approach, both at Sequent and also in the Linux community. You surround people that come in that don't know about parallel programming with people to do. You provide them quick feedback on what they're doing that's wrong and what they should do instead. And really quickly at Sequent within a couple months we have people that were writing competent parallel code. It was just a matter of you do it this way, kind of a thing. One of the things that we have now that we didn't have then is a huge number of open source projects that have parallel code. And that means that people can learn from those without having to be in a place where there's a code base in the 90s. Almost all the parallel code bases were proprietary. If you wanna learn about parallel code by looking at code base you had to work for the company to produce it. And you learn about that one code base only. And today we've got a ton of them out there and it should be possible to learn fairly quickly. If you are looking to take an existing sequential program and parallelize it I would suggest picking the open source project that for the most closely matches what your program is doing. The techniques are most likely to match what you're doing better. There's been a lot about embarrassing parallelism. How about it's trivial? I was on a phone call with a gentleman who will go nameless. And I was, he was having a problem with parallelism. I suggested just taking and using the shell script ampersand thing. Just run the program a bunch of times with different inputs. And he got really upset with me. That's not parallelism, it's parallel to me. Well, people do that. The colossal parallel program is in trouble. Well, yeah, people get in trouble anyway. I couldn't convince him. He was just, he's very angry upset. I don't know. The last one's probably the most important and that's taking validation seriously. As we saw, parallel programming has more cases where you have to plan more carefully and therefore your validation is gonna have to be more stringent. And that may mean you have to take your serial validation and get it up to speed. Get it to be more vicious. Find a lot of bugs that are probably hiding because people avoid them. And get to the point where you have a chance of having a validation system is gonna be equal to the task of dealing with parallel systems. One thing that Sequin got lucky with by accident, our first piece of hardware before my time, this was in 1983, I joined in 1990, turned out to support 30 CPUs. So they started doing for 30 CPUs from the get go. And as a result, they designed for that and they happened to have a guy named Bob Beck who was one of the few people who was able to just sort of into a parallel program and get it right. I don't count myself in that. And as a result, they were able to do the job once and have it running at 30 CPUs. The experience in most other places has been, okay, we got it running on one CPU, let's do two. Okay, three, four, and you end up oftentimes having to rewrite it. Now, yeah. As a fellow traveler, I will say the most critical decision was deciding at the beginning, not to put a big hurdle lock in. Instead, it was a hardware that we will lock into every single routine, every single thing and do the parallelism once. Yeah, so the, yeah, Steve actually joined Sequin before I did, it was two years before or something like that. And yeah, the, I'm not sure if the documents are public but the documents on it, they went through that and talked about that as a possibility and rejected it because they needed to get to 30 processors and they knew they couldn't make 30 processors if they did that and so they didn't. Oh, so yeah, that's, in any case, if you decide what your target is up front, you can get there in one step instead of having to go multiple times. Actually, what the Linux community has done is worked out fairly well because we're able to throw a large number of people at it and just recover from that sort of thing. You need to have a solid core of experienced engineers, somebody who's done parallel programming or have access to it. You really don't want people who don't know anything about it trying to mentor people who don't know anything about it. They better have access to parallel hardware. That's not a big deal today but it used to be and they need to understand what it can do and what works well and what doesn't with it. One thing they need to have access to all the source code. The reason is that there's a lot of problems you run over with parallelism that are distributed throughout the source base. So you have a deadlock, for example, that goes through a whole pile of components and comes back around to the start. If you don't have access to those all source, you're gonna have a tough time debugging that. This is one of the reasons why the Linux community doesn't like binary modules much. It's also the reason why Microsoft is willing to do Windows Vista to themselves. They kicked almost all their device drivers out to user land in order to avoid this problem. The other thing you need to do is make sure the team you have can deliver decent software and we'll get to that in a little bit if we have time but I'm just gonna go through some patterns that have worked out well in the past. This is a classic one. You have a database or something like that, some other piece of software. You write a single threaded code that talks to it and all the parallelism's handled off in somebody else's stuff. Works really well and it's been used a lot and you shouldn't be ashamed to use it. You can have, go ahead. Oh. Yeah, well, if there's somebody capable of taking the buck and they're willing to do it, that's not a problem. No, it's, sometimes it's fun and sometimes useful to just do something the hard way for doing it but if your job, if your idea is just to get the job done, passing the buck where if there's somebody willing to take it is a good thing. This doesn't have to be a separate program. Samba, I believe, uses something called TDB which is an in-memory database and they use that as kind of their center of parallelism with memory mapping if I remember correctly. Another one is, this is the one the guy got upset at me on the phone, just taken, whacked into pieces and, you know, do each piece separately and combine the results to the end. Really simple, but it works. And if I hit the right button, you then go to the next slide. And that produces sort of like this, it has a little more structure to it but it's the same basic idea but with some infrastructure to do this, spreading out in the gathering together for you. This is the one I think people forget and it's really important. Parallelism is at its heart a performance optimization. If your application runs fast enough on a single processor, why not just let it be single-threaded? I mean, if it's fast enough, it's fast enough. So I'm not gonna go through this in detail but just different ways you can purchase things. It looks at traps and pitfalls and yes, the sequential code has traps and pitfalls too but what ones are specific to parallelism? So let's start with the first one here, single-threaded code and so on. The thing is that there's a lot of things that are cheap and easy for single-threaded code. For example, single-threaded patterns. You can have global counters or monitors, sort of like the big kernel lock that Steve mentioned earlier and the global transaction IDs. These things are just cheap as dirt for certain because you can put them in your API, pass them up and down, everybody's happy but you try to do that in parallel and suddenly you got this one bottleneck, this global thing that's causing you trouble. Ordering guarantees, sometimes you can't avoid them but if you can, you might want to because the less coordination you have between different pieces, the less communication you have, the better your performance and scalability. Global locks again and I talked about strongly non-community this morning, it's still a problem this afternoon. The thing is that these work great for sequential code, they're often very expensive for parallel code and if you want to get to tens of processors you might have problems with it. What this means is that you should give up necessarily. If you need your single-threaded CPU to go faster, your single-threaded application to go faster, go ahead and make the change, just understand these are big animal changes. You are likely to have to do some serious rework on your application, so rip and replace. You might not have to, you may be able to just replicate the thing and split things up but if it's not simple, it's gonna be tough. So if you've got a sequential project right now you probably got people working on it. They might not know anything about parallel software although as time goes by it's becoming more and more in the culture so it'll be less and less of a problem getting people who can do that. You might be able to get an experience parallel programmer and then you need readily available hardware, access to software and so on but what do you do if you can't get an experience parallel programmer? Yeah. Give up there. Give up there? Again, if it's fast enough sequentially that's great or it can be somebody else's problem and thus have your people do the part of the sequential. Classes are becoming available. You can actually get decent classes in some places. And the problem is I can't really be the judge. I didn't learn this stuff in a class room. I learned it from an logic analyzer hooked up to a symmetry, all right? But there seem to be people getting something out of the classes. I can't tell you which ones are which but I asked people who've gone through them recently. And again, there's just a lot of open source projects out there that do parallel and you can go and inspect those and learn from them. Watch out for license and compatibility of course. This is the most important lesson. If you have only one of something you're gonna have all your CPUs trying to get through this door and there's gonna be a big line and big pile up, it's gonna be ugly. On the other hand, if you split things up you just partition things well. You won't have much queuing, things will go fast and life will be relatively good. Anyway, we're getting towards the end of this. Let me, yeah, if you have one, this is worth talking about. One of the things that happens if you have a long-lived project especially if it's for a given purpose at the application level, you get people that are working on the thing and they move on to something else. You know, what for people who are maintaining it. It's kind of like a building. Initially, when you build a building you got architects, you got construction engineers, you got all sorts of people putting things together and eventually you got janitors taking care of it. So if you've had a software project that's been existent for a long time and the guys that wrote it have gone away, you need to ask yourself whether your current people are really up to making a big animal change like introducing parallelism. If they aren't, you may need to make some adjustments. So yeah, we went through this in the morning. It's just important in the afternoon. How far should you take parallelism? Well, it depends. The further up the stack you are, the more you care about productivity again. The further down the stack you are, the more about performance and generality. So in the Linux kernel, we can afford to put a huge amount of time and effort into fairly small performance gains because they benefit everybody. If you're an application with only a few users you have to be much more careful about what you invest in and you want a lot more higher level tools to be able to get it done quickly. Anyway, I think that's a good place to end this. If there are more questions, we can take them. Is the floor feeling that he answered the question that he presented to himself? Is parallel programming hard? And if so, why? Yeah, what is the answer? So one of the things that a lot of parallel people seem to kind of brush under the carpet completely is that a lot of problems aren't parallel. And that is something that people have a hard time just facing and admitting and it seems to be a general problem when you then tell them, hey, I'm sorry, my problem is chasing pointers. You can't throw 16 CPUs at it. And a lot of people, you can do it by doing a lot of speculation and throwing energy at it, yes. A lot of people seem to have trouble admitting that sometimes you are just going to be sequential and have dependencies. I'll make a deal with you, Linus. If you ever see a patch from me that attempts to paralyze p-thread mutics lock on a single lock, it's time for me to stop. No, some things are more parallel than others. Sometimes you can think about them differently recast and there's a dithering. There was this dithering algorithm a while back that involved kind of taking things together in strange ways and it felt that you couldn't paralyze it because there wasn't any way to run down the thing, either way, that got rid of the dependencies. Some kid in college in the, oh, it was the 2000s sometime realized that you could just look at the thing diagonally and the dependencies went away and became parallel. Now, it turns out that a lot of other dithering algorithms don't have that property. They're stuck, but again, parallelism is a performance optimization. Like any other optimization, there's some places it applies and some places it doesn't. Yeah, getting a woman, having a child in less than nine months is hard, even if you have nine women, that's a silly joke, but the point is- Well, the thing is it depends on whether you want a human child or not. If you were only in the first, other kinds of mammals, it could be quite a bit shorter, but. I never learned, I can't either with this guy. So a gentleman and then the other question, please. What I've often found when trying to parallelize systems though is that the inter-processed communication overheads more than often exceed the benefits to be gained through the parallelism, because IPC is still incredibly expensive. Memory access is incredibly expensive. And- So what's really happening to try and address some of those issues? So one of the things that's a really good thing is the quote from this morning on from Dr. Patterson, where he was realizing, he was the guy that had the 13 dwarfs, was it? Or he had these little computational kernels he wanted to parallelize, and he's recently realized that what you need to do is take as big a chunk as you can and parallelize that, because a lot of times, if you take a big chunk and cut it out, you'll have more computation for a given amount of communication. That doesn't always work that way. I mean, nothing works all the time for everything, but that's one thing to try or one thing that may work. Just about development practices, you highlighted mentoring, pairing, experience within the experience program as everybody having access to the code and being able to refactor it. That's two out of a list of extreme programming, isn't it? Any comments on programming practice? Do the strategies that you've seen working for parallelism work for development in general in your experience? It's been so long since I've done sequential programming commercially that I'd have to be hard pressed to know. I think there's a lot to be said for that. I was thinking in terms, with pair programming, what you have is you have people of more or less, often equal capabilities, maybe different viewpoints, working together on a problem, and that's a useful thing, I think, regardless. What I'm thinking was a little bit different where you have somebody who's new and doesn't understand things being surrounded by people who will give them quick feedback on what they need to do. Not on that it's broke, but on, okay, do it this way and it'll be better. And that often people really quickly react well to that. Sorry, I just want to say thank you for your time. I hope that you will be at five at the panel and then we will continue taking questions later, but then we can continue with the program and give the opportunity to everyone else. So, join me, thank you very much, Paul, for having me. Thank you.