 Okay, we can at least start talking. We are here talking about Drupal and QA, the testing process, Drupal limitations and satisfying customer expectations. I hope everybody's in the right place. And also I hope I'm in the right place, you know, in the right room delivering this. What we're gonna do is talk about just quality. We'll be talking a little at a higher level about what quality is rather than, you know, focusing on actual specific testing methodologies, testing techniques, which are changing heavily from Drupal 7 to Drupal 8. In Drupal 8, you can do a lot more testing that's already within Drupal 8 instead of doing... What a developer here told me is basically Behat, Selenium and Gherkin, and cross your fingers. So we'll do a lot, you can do a lot more with Drupal 8 rather than the cross your fingers and hope for the best. But we're gonna talk at a little higher level about what quality is. First of all, who am I? My name is, well, it's pronounced like it's spelled, Anastasios Dascalopoulos, email dosco at exov.fi. This has been my nickname since I was like four or five. So, dosco is what I go by. An, dosco, and here we are at DrupalCon Dublin 2016. It's been great for me. I hope you've enjoyed it also. Well, let's talk about quality, software quality. And also, I like to turn these things into conversation. So if you have something to say, just pipe up, and I have to admit, these lights make a difficult to see everybody, so please just shout out if you have anything to say, if you have anything to contribute, if you completely disagree, that's actually great with me too. But let's talk about quality. First of all, what do we mean by software quality? A lot of people who develop don't even really think about quality. The whole point is to simply just get something out the door, get something to the end users, and go on to the next thing. But quality is something you have to think about throughout the entire process, and it's not something handed over to testers at the end. We'll be going through this with a whole idea of continuous testing as part of continuous integration. And we'll be talking about all this. So again, to answer the question, what do we mean by software quality? Software that functions according to the requirements of developers and to the needs of the intended users. I'm talking about this a little more. This definition we give, it sounds like an easy path to satisfy both creators and customers, but as we know, things go very wrong very often. What do we do? Do we contribute this to developer carelessness, poor management? Do things just go wrong in software development? Yes, they do go wrong. I found a bug, cool. Or can software just be described as lemony, like that one bad car that comes off the assembly line? Or do things get developed wrong? Is there anything we can do to make sure that software functions in the way that both we, meaning those involved in the software development process, and the end users who actually employ the software and who pay for the software can use. But first, I wanted to talk a little bit about the word bug. What do we mean by bug? What is a bug? Okay, interesting, this isn't changing. That's cool, I found a bug. Now to us, bugs are good things. Okay, cool. Finally, okay, so the word bug first appeared in a letter from Thomas Edison in 1878. Thomas Edison, the inventor of the light bulb, phonograph, motion pictures, numerous other things. He used bug in the same way that we basically do. Okay, he says, it has been just so in all of my inventions. The first step is an intuition, then comes with the burst, then difficulties arrive, this thing gives out, and it is then that bugs, as such little faults and difficulties are called, show themselves, and months of intense watching, study, and labor are requisite before commercial success or failure is certainly reached. Okay, so he used in 1878 the word bug in the same way we do. There is another story, which apparently came later that the word bug came from experiments in the late 40s in Washington, D.C., back when computers used to be like really big room-sized mechanical devices. Apparently at one point a moth, literally a bug, got stuck in a mechanical relay. And that basically crashed, well, we use the word crash also as a mechanical term that comes from these early computers when the equipment, the physical gears would literally crash. But that moth that got stuck actually stopped the entire computer. And interestingly enough, if you go to the Smithsonian Institute in Washington, D.C., you can actually see that moth preserved under tape. It actually kept it still, so you can actually see it. But so we can see for almost 150 years, people who build things, meaning new, as of yet unused, technological things, have had to worry about the little faults and difficulties as Edison refers to them. And it's interesting to see that such a successful inventor, as we would call him today an entrepreneur, such as Edison, realized that removing the bugs, as he called them, the little faults, in other words, is a long and difficult process that often take months to eradicate, to get rid of. Before we move on to this, I found out something a little bit, knowing where we are about the etymology of the word bug. Apparently, bug actually goes back to Celtic, an early Celtic root, meaning a threat, or something like an unknown threat. And I find it interesting that this, what would you call it, an adivistic, preternatural, basically supernatural, unknown threat usage, is basically what haunts us today in software development. So the word bug just meant like unknown, kind of spooky threat. This obviously, this apparently gives us the word like bogey. Even the word like bogeyman comes from this whole same idea of just an unknown threat, just a threat out there. And speaking of threats, has this happened to anybody's phone yet? Has anyone, you've seen word of this, you've seen this on the news, right? This is the Galaxy S7 in action. I'm very, I can have to tell you, I don't know if this happened on your flight, but before I came here on thin air, the stewardess came on the intercom system and said, if anybody has a Galaxy S7, please come see them as soon as possible before the flight takes off. Now the way I see it is if your product isn't allowed on an airline flight, then there's something wrong with your QA process. Actually, how this got past testing, I really don't know. I'm just curious if somebody just didn't notice this happening to the telephone. Was it not in the requirements? I kind of know testers who probably wouldn't say anything because the phone catching on fire wasn't in the requirements. So let's just pass it and go on. It really just boggles my mind. I know in the US, every single one of these have had to be recalled. And I know they test for this because when I was at Nokia, I had friends of mine who were physical testers. They were hardware device testers who would do things like drop telephones from certain heights to see what would happen. They would spray them with water, throw them in a container of water for a number of minutes, and they would also set them on fire too. They would try to set them on fire, but it wasn't just destruction or vandalism. They were actually engineering to see exactly where the failure is and where the point of failure is. Apparently the point of failure comes somewhere in the battery, apparently a battery lining breaks surprisingly easy and the phone catches on fire. I have to tell you when I was testing Windows 5, this was about 10 years ago using an HTC device. I noticed something similar to this. When you were live streaming, you physically couldn't hold the HTC device for more than 20 minutes before it got too hot to hold. Literally would just get too hot. I did want to see if it would start smoking or catching on fire, but it didn't, it just got too hot. So these things happen and they happen even more. I got this news actually last night. This is another Galaxy device. This is another Samsung. This is the Samsung Note 7, a different device. Okay, just like with the Galaxy S7, 7 doesn't seem to be a lucky number for the Samsung company. So if you have a Note 7, just use a different phone or let it run down or just keep being careful about what happens when you plug in. Or you can just set it on fire and see what happens. That's what I do. But there are other bugs out there. We're just gonna talk a few minutes about some more because these things, I'm showing big examples of the problems that can happen. These are all bugs that have been released. Very disastrous bugs that have actually gone out in the wild. Let's find Melbourne, Australia. According to Bing Maps, where is Melbourne, Australia? Yeah, it's somewhere off the coast of Japan. This was released a few months ago. Two things surprise me about this. Well, three things, counting the location of Melbourne, Australia, according to Bing Maps. This was apparently blamed on Wikipedia. This was caused by an error in coordinates by Wikipedia. Surprises me that Microsoft uses Wikipedia APIs and that they can get things so wrong. Other examples. I first got into this business through aerospace, through aviation. So a lot of these things are gonna be aviation and aerospace related. The 1962 Mariner 1 only lasted about one minute because of a mathematical error. And the way probably coding and software at the time couldn't really represent certain mathematical things. This was the over bar in physics. After five minutes, the Mariner 1 had to be manually, well, just blown up. Something a little more recent. You may have read about this, but the 1998 Mars Climate Orbiter, this is the problem with imperial versus metric measurements. And Lockheed Martin, the software on the ground calculated the Mars insertion thruster firing. They're basically the little rockets that take the capsule from space into Mars atmosphere. They were calculated to fire in the English pound seconds. But the NASA software on the orbiter itself expected the calculation to be in metric system called Newtons. The expected result, the orbiter broke apart in the Martian atmosphere. So it's probably lying in pieces somewhere in Mars. Or maybe it missed Mars entirely and it's just kind of floating in space somewhere. It cost $327.6 million because somebody didn't basically look and see what are the calculations expected in. A few more things. The Mars spirit in 2004, a little more recently. The DOS file system caused a flash memory to fill up and it kept the rover in an endless reboot cycle. This was fixed a few weeks later but without a huge cost of time and money. Again, it ran into the millions. And also the Mars global surveyor values written to the wrong memory addresses forced the surveyor to begin unnecessary mechanical operations that caused basically the whole thing to go into safe mode and it hasn't been contacted since. So basically when people start walking on Mars they can go visit the global surveyor, I guess, manually reboot the whole thing. And it should start working again. Things a little closer to home. I don't know if anyone's ever heard of these. The 2007 Excel problem. If you multiply 850 times 77.1, the end result was always 100,000 and similar calculations like this. Problem error and function that converts binary numbers to strings for displaying. The calculation was correct deep within the computer but the problem was in the display. Displayed it as 100,000. And some little tricks that still exist in software. You can try these yourself if you want free hotel rooms, free stuff. You didn't hear it here but you can try it. Sometimes this was fixed in a very large, probably familiar sounding hotels, something, you know, booking error. We'll just call it like that. And error in the URL display left the fee parameter in the URL field, thereby making the hotel payment up to the users to simply change the fee parameter to 0.00 and getting a free hotel room. You can do this with other things. Yeah, this one is difficult for me to see. The 2008 Google Maps error or a LinkedIn National Cemetery outside of Washington DC in Virginia has the icon that displays it as a restaurant. Horrible. In 1997, this still happens in some e-commerce. Make sure when you're developing your e-commerce sites that this doesn't happen. Customers place a negative value in the quantity field which allow the customers to get free credits at checkout. Okay, make sure that you can't put negatives in your field. That sometimes still happens. Other things that this was in the news a lot, I remember. The 2012 Apple Maps, if you've had a few drinks or something else it pays to go to get a real mind trip. There's a huge problem with scaling where roads kind of disappear, bridges kind of dip into rivers and then come out the other side. Numerous examples of this here. There's a road that kind of goes in and out. It's kind of like a dolly painting but this was the landscape in the 2012 Apple Maps. They can still be found. I swear. And a few more things. Having done this a lot with airplanes, Avionics, I don't mean to make you paranoid but Avionics software is also very buggy but there are a lot of backups, there are a lot of manual backups which you don't have on a computer. This is unfortunately one that didn't work in 2015, the Airbus A400, a military Airbus airplane that's supposedly, well they're trying to get it to replace the USC-130 for military cargo and transport. And the A400 in Madrid, Spain crashed soon after takeoff because of a software bug. Apparently in one of the engines, they're a little, the Avionics has like a little text file. The text file has a bunch of numbers and calculations that provide torque to the engines to get the engines to spin really, really fast and strong on takeoff. And therefore the airplane goes up instead of down. That's the real key. And one more thing, the software error, I found this a few months ago, caused a release of prisoners early, an application that calculates prisoners sentences based on good or bad behavior. For 13 years calculated, there were release dates two months early on average. There were some very violent convicts who were released years early. Of course you're sentenced. This happened, I believe, let me say in Washington state. Of course you're sentenced to a certain amount of time but apparently based on good behavior you can, or bad behavior, you're gonna be let out early or kept later. Everybody got out early because of this software problem. And a little closer to home for Drupal 7. I spotted this a few months ago on simply a C run stop on Vragrant. I'm seeing if this can be reduced, I mean reproduced on Drupal 8. But notice when the last run of this was done, according to this. Yeah, so this is Drupal 7. I should report this, I guess, but I just like reproducing it and seeing what happens. Everyone keeps saying how new Drupal is but according to this, last run 46 years and seven months. So Drupal has these kinds of things. I like looking for these not because I'm being sarcastic or trying to just find problems but there's a certain creativity involved in finding these kinds of things. So we'll see if this happens on. I'll have to put the same command on Drupal 8 and see if we get the same result. To give you an idea of the cost, these aren't just silly little jokes, silly little matters that can be easily fixed. These cost a lot of money. This figure came last month from Better Software Magazine. It's like a trade journal for software testing. The cost of bugs to public companies from 2012 to 2014, the last numbers, was $2.3 billion with a B that is. A shareholder value lost on the day of software bugs announcements. Okay, so somebody actually calculated on the day a big bug is announced in the press, how far does the stock value of a company go down? If you add it up, it goes to $2.3 billion from 2012 to 2014. One thing I'm saying here, software bugs are increasingly becoming more frequent and consequently more costly. I think bugs have obviously always existed from the day that the moth got stuck in the mechanical computer. All the way to what we see now. Bugs are becoming increasingly, maybe not more frequent, but the reporting of them is more frequent. We know more about them. And we know more about them. And so, as the article says, as software becomes more and more ubiquitous, you know what the internet of things and all that, software development methods on their own are proving less and less reliable. Maybe a little extreme statement, but something to kind of get your point that more and more bugs are being released and more and more bugs are being reported. And so there's a bigger and bigger threat to what is going on in the software that we're releasing. And this is something that happens to Drupal also. Drupal has, shall we call it, limitations. There's nothing wrong with it. It's a good system, as we all know, but it does have limitations and what we're gonna get to after some more talk about what quality is. We're gonna talk about how we can work within these Drupal limitations to satisfy customer requirements. And what I've done over the past few months to get things to pass, to make the customer happy, despite Drupal's limitations. Well, let's talk about quality. What do we mean by quality? Quality you can get now. The point of what I'm gonna try to say is the whole idea of continuous testing. There's a whole idea of continuous integration, continuous development. And especially with Drupal 8 now, you can do continuous testing too, which is not set apart from continuous development. It's whole part of the integration process. Integration, meaning like development and testing are happening all the time. They're happening in parallel. It's not like you develop and then you test, you're developing and testing all at the same time. So what do we need? Well-defined business requirements. We'll talk about how, or even if business requirements help. Specific business related defects are prioritized for test cases and fixing. And especially repeated regression test automation sets to verify bug fixes and also to detect other bugs. You're not just going through and testing what has passed. You're always looking for bugs in test automation. And you're always maintaining the scripts. You're always editing the scripts. You're basically changing everything. You're changing the scripts in test automation to test the changes in the software itself. I heard once and I have to agree with, in test automation, the actual script creation isn't that big of a deal. A really good tester, a test automation tester. We've had some good speeches here so far. A good automation tester is good at maintaining the scripts. Not just creating them but maintaining. The whole point of test automation is not to get the green light. It's to find bugs. A good script finds bugs. A good script doesn't pass. A good script finds bugs. It's that mentality that I'd like to get everyone to change. What is a successful test pass? To me, a successful test is something that finds bugs. I've gotten in trouble over the years with this because development managers will say did the test runs go well? And I would respond yes. And they say, good, everything works then. My answer was no, things don't work. That's a successful test. I found a lot of bugs. That's a successful test round to me. So we have to make sure we're on the right page. You know, if there weren't any bugs found, I would think I'm doing something wrong. So yes, I've gotten my share of grief because I do find bugs, but it's my job to kind of piss people off. You know, there are bugs, you know. Things don't work. People often say, okay, DOSCO, the software's ready, now try and break it. The thing that I always have to say is I don't have to try and break it. Because generally, it's already broken. So we have that attitude. Don't get me wrong, I like to break things. You know, I'm very bad at ordering things through the web because I end up crashing the e-commerce form, you know. That's just what I do. Maybe I break things and to me that's good. You know, I've always enjoyed breaking things and I love this job. I get to break things. Let's talk about some ideas about what is software quality and things to keep in mind as you're developing. Software quality, the elusive target, this article has five perspectives regarding software quality. And again, let's talk about this. If you agree or disagree, let's talk about this. The first perspective, manufacturing. Software quality is the capability of a software product to conform to requirements. Okay, that sounds good. Just the capability of software products to conform to requirements. To me, this statement is kind of a hedge because to obtain quality, all the software has to do is conform to requirements. As we know, anyone who's actually out of work in developing software so much can go wrong and there are so many repeated changes to an extent that it's difficult to just conform to requirements. In the end, yes, we do have to conform to requirements, whatever those may turn out to be, but simply to just conform to requirements is no guarantee that the software does work. I have had development managers and I've worked with development managers that will say in the end, yes, we've satisfied the requirements, we're done, it's released, next project. But again, this may not be what the end user wants. Just because it conforms to requirements doesn't mean it's not still full of bugs. You can't just conform to requirements and release unless you change those requirements. And going on, this is an interesting perspective. It's what they call the transcendental view of quality. Quality is something towards which we strive as an ideal but may never implement completely. This is close to paraphrasing what the U.S. Supreme Court Justice David Potter said in 1966 regarding obscenity. An obscenity case, apparently in 1966, went all the way to the Supreme Court. Had to do with the film by the French director Louis Moll, I believe it was. Movie theater manager was even jailed because he was convicted of showing porn and his case went all the way up to the Supreme Court. And how this case was settled was the justice David Potter said regarding this, it's same regarding quality. The fact is you know it when you see it. How do you know you have quality? You know it when you see it. Maybe some things just can't be defined. It's like, how do you define a good movie? Or even other things, how do you define a good pizza? You just know it. So even with good software in this view, you just know it when you see it. You can't pigeonhole it. There's no way to define the essence of good software. Going on with these and the context of use. I like this one. It's like it says, a very concrete down-to-earth definition based on the simple concept of are the customers happy? Does production software fulfill customer needs? Are the end users that is the paying customers happy with the software they're using? Are you getting complaints? Are sales numbers going up? If you're not complaining, if you're not getting any complaints, if things are selling, well, yeah, you have good software. That's one thing about software testing. Basically, we know we've done a good job pretty much when nothing happens. If there aren't any crashes reported, if there aren't any complaints, then we've done our job. Often the credit, we don't really mind this, often the credit goes to development because the developers, of course, have done a good job in developing good, easy to find software. But it's also our job to find all these bugs. So basically, we've done a good job when nothing happens, when just good things happen. Are there a few complaints? Some people will always complain is the software selling. That's the context of use. And we'll be getting back to this whole idea. This is the view called the product. Product can be, quality can be appreciated by measuring the inherent characteristics of the product. In other words, the quality of the product simply measured by what it says it can do. If the product performs the task, then yes, it's a high quality product. I would agree with this, of course, but there's another way of looking at things. It may be a high quality product, but is it what the end user really wants? We can look at high quality products in other fields. For example, let's just say the Ferrari Italia, a high quality product, very fast, very reliable. But what if you need to haul a load of firewood to the camping site? Then the Ferrari Italia is not gonna do very much good. Yes, it can be a high quality product, but you ultimately have to satisfy the user requirements. And the last thing here is the value based perspective. This perspective recognizes that the different perspectives of quality may have different importance or value to various stakeholders. Okay, we may find, we may develop a great piece of software, but if it's not what a customer can actually use, then the value of the software ultimately is very low. And yes, very buggy. An example of this that I remember from my own experience is that there was, let's say, I shouldn't use names, but a very large Nordic shopping, not just say a mall, not to use any, not to give anything away, but several subcontractors were hired to do the retail software at this large Nordic department store, and the software was basically released, but it was so difficult to use like the store workers, the actual end users, the actual store workers weren't able to use it. It was just too complicated for the workers to actually use. The company that developed it basically said, well, it's not our problem if people don't know what they're doing. But the whole point here is, yes, it is your problem. You have to have software that people can use. That's a good thing that Drupal does because it does provide software that people can use, but you have to keep in mind that let people use it, don't make it too complicated. But moving on quickly to actual Drupal problems, I'm just going to skip quickly here to some more concept of quality to keep in mind. This is what I would like everyone to keep in mind. Quality is a customer determination, not an engineer's determination, not a marketing determination, nor really a management. It's based on the customer's actual experience with the product or service measured against his or her requirements, stated or unstated. And always representing a moving target in a competitive market. So your software has to be what people can actually use. Another author who writes about this very quickly, quality has multiple meanings. Quality consists of those product features which meet the need of customers and thereby provide products out of satisfaction, which we've been saying. Also for us, quality consists of the freedom from deficiencies. Nevertheless, in this definition, such as it is, convenient to standardize on a short definition, the word quality is fitness of use. Can you use this definition? Quality is fitness of use. Can you use something? As we've seen from the Galaxy S7 and Note 7, you can't even use it. There's some products out there you can't even use. And customers need to use everything well. But going on a little more, let's skip around. Actually, let's just get to Drupal. I want to get to this point in time. As a tester, as a keyway guy, I've always found myself to be the voice of the customer, the voice of the end user. Yes, the voice of management, the voice of the requirements writers, the voice of the stakeholders. But ultimately, I try to be the voice of the customer who's ultimately using it. To me, ultimately, in the end, we develop software for the person at home. Software development is all about satisfying the needs of the end user. So we'll look at some examples in a few minutes to see what Drupal does and does not do to satisfy end user requirements. To this end, like I say, we have to find out what the customer wants in order to fill one part of the software quality, duality known as verification and validation. Just very quickly, validation is, do you have the right system? Verification, does the system work right? Often that doesn't happen. Going on, like I say, verification system right versus right system. Validation, we have to know what the end users really want. Often this gets lost during the development process when numerous changes have to happen. Numerous things have to be developed. We have to start over again. We forget about the end user. So never forget about what the end user actually wants. And also, we have to remember, does the end user actually get a working product? Moving on, in terms of just DevOps, this is where we come into play. This whole world of quality assurance and software testing has been changing a lot. It's changed a lot when I started. When I started, I was at a Nokia office, I guess in south central Finland, in a big room with like 40 other people, all with like word documents in hand, all basically doing the same steps at the same time. That is old history now. This was in like 2003. Things are changing. There basically just isn't time to do that kind of testing anymore. There isn't the money to do that kind of testing anymore. So in DevOps, we define it as a method in which we can more confidently evaluate our products continuously to understand the risks associated with it at any point in time. This definition just calls it a risk. That's a funny thing about the software testing industry. In Kiwi, very often the word bug, issue, the media likes to use the term glitch. Actually for legal purposes, we're not really using the word defect anymore because defect means you're releasing defective software. And guess what? You're becoming liable to legal problems, legal hassles. So we use those words interchangeably. And I've been a big fan of QA independence. It's a bad idea to have somebody be a developer and a tester. You really can't just change hats. You can't first be the developer and the tester. You can't do now I'm the developer, now I'm the tester. You need a little bit of independence. You need to be able to step back because often two people are too close to their own software. A lot of developers don't want to find bugs in their own software. They don't want to assign bugs in their own software to other developers. The way I see it, bugs are a good thing. In fact, in Japan, in the whole Toyota process that's been written about for a few years now, workers who find a problem on the assembly line are financially rewarded. That was a problem with the U.S. auto industry where often a worker who found a problem was blamed for somehow causing the problem. We get that too in software development. Like I said earlier, often we're not very popular but in a good development environment we're needed, we're necessary. We become very popular because we point out the problems. And getting to the thing, oh yeah, this is one problem I've always had. All bugs have to be fixed. Really, there's no such thing as a minor bug. All bugs have a deleterious effect on the end user. There's a big problem known as incremental degradation. It's defined as the result of tiny progressive reductions in the quality of a product or a service. In other words, if you allow one minor bug to be released, then you'll allow another minor bug and then another minor bug and then another minor bug. And often what happens there, the severity of those minor bugs become bigger and bigger and bigger and bigger until you have like a huge crash, a blow-up. Actually, this idea started from the 1986 Challenger space shuttle disaster when basically there were numerous bugs in the Challenger's system. But the engineers thought, well, everything still works fine even though we have all these bugs. What's one little problem extra? And as you see, all those little problems will add up to one big major problem. I've read that, for example, Hewlett Packard, 40% of all customer complaints come from what we call minor bugs. So minor bugs are things that have to be fixed just as much as major bugs. As developers, I want you to get that mindset in. If it's easy to fix, then just fix it. Everybody's happy. There are two types of bugs, two types of software bugs. One is the software bug specifically. Bugs are a result from a coding error or some other machine produced error. And then there are the user expectation bugs. Some little things like design error, text or field overlaps, misalignment, they're both bad. They both are threats to customer requirements, customer satisfaction. What we're going to do is we're going to see a few slides from Drupal 7. And then we're going to debate on are these bugs or not. But they show the limits of Drupal as what we found in our things. There are some NDAs that keep me from actually showing you certain bugs, but I'll describe them as much as possible. But we're going to see are these bugs or not? And also, like I say here, there's no clear-cut definition of a bug. What is a bug? I don't know, nobody knows. There's no real definition of a bug that everybody's really agreed on. The best one I found is any unexpected or undesired software behavior. Agree or disagree? Well, let's look at a few things before we wrap up. Here, this is a lift-up that's supposed to start a YouTube film. All you have to do is push the play button. Can we find the play button? Play button? Especially if you're a user who's just scrolling down. Play button. Apparently, it's kind of in here maybe somewhere. Also, there's text here. That's not an error, that's the finished language. So don't worry about that. This was resolved when this was actually going to be released, but I objected to this because nobody can actually see that this is supposed to play an animation. So I reported this and fortunately everything was darkened a little bit and later released to customer satisfaction. Here's another problem. I think this is a problem. Does anyone have a problem with this lift-up? There's something that bothered me about this picture. Again, I didn't want this to be released because to me, this is a buggy lift-up. And the bug I found is... It's something in the Drupal that's a picture that has a link here. In a main page. Yes. That was supposed to be a link. There was supposed to be a link here. The problem I had is look at the sailor's head. I don't like the headless sailor here. Luckily, this was later resolved. Actually, one big Drupal limitation that unfortunately I couldn't show you is we had this one customer that had a lot of thumbnail images. The thumbnail images were supposed to show like a person's head. But the thumbnails just showed like parts of people's heads. That just looked really weird. We resolved that problem by basically just getting rid of the thumbnail images. So these are the problems here. Unfortunately, I'm running out of time. This was a weird... This was actually a customer complaint. The customer said the button is not green enough. So what did we do? This is an example of a client dissatisfaction. So it's kind of up to us to figure out what does a not green enough button mean? Yes, we had a green button. I passed the case because there was a green button. The button was green. But the customer wasn't satisfied. There was something wrong. And so the client satisfaction occurred because apparently they were expecting a greener button. And the solution was maybe a dark greener button was added with more easily red text that replaced the earlier button. And apparently that was okay. So these things will quickly be more easily done in Drupal 8. So we have a given development where everything's based on the tests first. And also, same thing with unit testing. PHP unit is very well integrated into Drupal 8. So you can use it to do not just unit testing but also with functional tests as we've had other presentations here. And just in summary, I just want to let you know that I found that quality is an elusive and difficult to define aspect of software development, but one that we ultimately need to find. This is what's part of the bug hunt. This is part of the hunt. We have to hunt for quality. You have to actually seek it out. It doesn't just happen. And one that ultimately must satisfy end user expectations and needs. And the last thing is the job of quality assurance to work besides but independently of development to both validate and verify the software to make sure that Drupal develops the right system so the customers will have a system that works right. And with that, go out, develop, start with Drupal 8, start with all the great things that'll let you test your applications with Drupal 8. And good luck and have fun with it. Okay, I'm done. Thanks. Sure, if anybody has any questions, hardly anybody, no one ever does, but if you have any questions, I'm happy to. Yes. I think you're working, because moving to more outside ways of actually working due to those very same reasons that you're outlining there. So you have user stories, which instead of being requirements, they have a user perspective, and their role is to actually make sure that that user perspective and have the acceptance idea. I mean, wouldn't that be a solution instead of relying on the testing engineers to actually get that perspective in? Yeah, yeah, it's part of the, it's the part of the test engineers to be a part of that system also. Yeah, so... Essentially, you make them better so to say, messy. Yes, yeah. And the key way within that system works within that. To be the voice of the end user, yes, and also to make sure that everything's working correctly, as we said. There's a little bit of independence required. Oh yeah, yeah. Well, that's what it does. More often than not, that's the PMs. Exactly, yes. You don't think they're better. Yeah. Again, this is a project management. This is a point that I made quickly that I really had to go through that a developer, when I asked developers, why aren't you doing more unit testing? Why don't you do this? And one developer said, well, we're not being paid to do unit tests. It's not budgeted. It's not budgeted in time. It's not budgeted in money. So if we would do unit testing, if we were paid to do unit testing, the problem is, especially with Drupal 8, I'm not just being such a big Drupal 8 evangelist, but it is easier to just develop and test. You can develop the tests in parallel with what everything else you're doing. So yeah, that is ultimately a project management responsibility, but it's the job of the QA to get the project manager to do that. A little bit of politics are needed and a little bit of the whole idea is that if you spend a little bit of time doing unit tests, like budgeting for unit tests, you don't have to use that time to do some kind of post-release bug fix release. When I first started, unfortunately that was part of just the usual development cycle, we release and then we do a bug fix release. I thought that was nuts. You don't just say, well, we're just going to give time to do a post-fix release. That's often needed, but to me you have to try and reduce that or eliminate it. One way to do that is to test as early as possible. Make QA part of the development process and make the unit testing part of the development process also. So you're not developing and then testing, you're kind of developing and testing all at the same time. That's my point. What else? Like I said, go out and have fun. Check out Drupal 8 and see what it can do for quality assurance.