 Welcome to this last part of the testing and quality assurance module and so far we have mainly been talking about the development or in I call it implementation testing. So the kind of testing that happens during development usually done by developers, unit system and integration testing and then in the book by Somerville there's also the talk about something that is called release testing. So this is really the process where we check whether the system is ready for release, it's fulfilling the requirements so it has quite some focus on validation testing. What we often have are terms like a factory acceptance test or generally an acceptance test and that's the process of letting the user the client or other stakeholders test the system and figure out whether they can accept it or not and the factory here simply means that it's actually in the real environment so there's quite a difference whether you would invite the client over they use the system or it's in the real environment. For example think about a system that is used in the hospital in healthcare if you invite a nurse over to test your system or the nurse uses it in the hospital there's a completely different level of stress or other things going on so this can heavily influence for example the user experience or the usability of the system. So this is something that is typically under the umbrella of acceptance testing and very often the system here is tested as a black box, black box testing so the system is running you see the outside you see the user interface you see the command line interface whatever you have but you're not looking at the internals you're not looking at code statements you're not looking at interaction between components it's really just a box basically so that's for example compare looking at a car from the outside where everything is fine whether it starts versus opening it up and actually looking at the individual components whether they look okay so that's what we usually do in the acceptance test setting. So that's release testing now to wrap this up this has somewhat this of course comes from the waterfall the V model the idea that we develop we implemented somehow and then it's being tested and then it's released and then we have an acceptance test the system is accepted the client runs it now this has changed dramatically with agile development and also more recently with something that we call DevOps or development operations we'll cover this more later but one of the big issues with the old waterfall development is that the customer would basically come in in the beginning to tell you what they would like to have what they need and they would come at the very end to check whether it's okay and in and more often than not it was not okay because well things changed requirements changed you forgot about something you understood something else under the way and so on so this was usually an issue which agile development addressed by saying we would like to have acceptance testing at every sprint so what happens for example in scrum is that after each two to four weeks in the process and the official process the customer representative should be doing an acceptance test so after two to four weeks the customer representative says whatever you implemented in a sprint is acceptable or not there is an actual demo an actual acceptance test it might not be a factory might not be in the actual environment but at least it's with a real customer or at least a customer representative so this is something that of course has changed so I would say the border between development and release testing is getting more and more blurry which is also why I in this module quite often talk a bit about both and similarly what is getting more and more common here is to look at the operation side because if you look at this this is all about the development we develop something we test it we check we release it and then it's operated then it's running everything is good but nowadays of course you know from companies like companies that are famous for this maybe there are a lot of companies do this but like Netflix Facebook and of course all the other big ones Google or Microsoft they actually run a lot of tests also during operations so it's not just release and you're done but you actually check what is happening during operations and just to give an example Netflix published something that is known as the simian army which is a different kind of monkeys basically and that's a system that does messes with the running with the productive Netflix system and sees what happens in order to find defects in order to find out whether the system is running as expected for example in the most extreme case they have a component that randomly takes parts of Netflix and just crashes them it takes down a component it switches it off and it sees how does the system react and of course the aim is to see is our resilience essentially is it good enough does the system react by for example distributing a load differently and the users don't really realize it or if it's maybe an extreme crash several components go down at the same time then well it goes it gets a bit slower for the users but it's not extremely noticeable so they do these things at the runtime and of course they need to collect a lot of data a lot of input from users but they use that essentially to see are we having any undesired properties in our system are there any bugs we don't know of but it has nothing to do with development release it's during the operation so more and more we actually also have something that I would call maybe operation testing which is not covered in the Somerville book but it's more and more common there are also things which we'll cover later that are called for example a B testing that you roll out different functionality to different users and they're also somewhat related to that okay so overall this concludes the module on testing on quality assurance remember we talked both about inspection the static analysis the static processing of different implementation artifacts or design artifacts and we talked about testing where you actually execute something and remember it's not not one thing fixes everything but you actually need a combination you need reviews you need inspections you need tests and you need tests at very different levels during development before the release during operation but also at different levels of your code at unit level at integration level and at system level thanks for listening and see you in the next module