 Well, good morning everybody. My name is Sanabel M. This is my first time here in a Drupal come And it's a pleasure to me to be here the title of my presentation is a smart test Accelerating the detection of faults in Drupal smart test is a new testing module to improve the process of testing in Drupal Before starting with the presentation of this work. Let me introduce myself. I'm my team I'm a novel and Sanchez and I'm a computer engineer I work as a teacher and as a researcher at the University of Seville here in Spain and In particular, I'm a member of the research group of applied software engineering I works with I work with Sergio Segura and Antonio with Cortez both of them are my PhD advisors Gabriele Dalgo that's work as technical support Well, this is the path I will follow during the presentation during the talk first I will explain explain the introduction and the motivation to make this work then I will describe the Our approach and the evaluation Then I will show you smartness and a demo of smartness and finally I will summarize the conclusions Introduction Well, why am I here in a Drupal come I'm a PhD students and my this is my topic of this is Deautomated testing on the ability intensive systems What is that of variability intensive systems of Variability software refers to the ability of a software system to be extended Customized to be configured to a change to be used in a particular context. So Software is posting a high degree of variability are referred as variability software systems So for example operating systems like Linus applications like eclipse or cloud applications like Amazon have been reported as Variability intensive systems. So any software that has a high number of plugins or components that can be configured Leading to different final configurations can be defined can be considered variability intensive systems Okay, in order to model the variability of this kind of system this type of variability intensive systems In my research contests are really popular the future models future model helps to to model the variability And to to know to represent all the components in a system and the relationships among the components So a future model represents all the possible configurations of the variability in terms of systems in terms of features and relationships among them For example, this is a official model as more efficient model Representing a mobile form systems is just an example All the components in a mobile phone in this example are represented by features that are represented by rectangles all this one are rectangles So this is the core support for the mobile phones GPS support the screen media and so on the Relationships among the features are represented among edges The higher alcoca relations is among the features in among the components in a mobile phone in these examples Are divided into mandatory relations that are represented by a blood circle so Mandatory features like cores are features that always have to be Presented as have to be included in all the configurations of a mobile phone So core support have to be included in all the configuration of a final mobile phone. That's means Another kind of hierarchical relationships are the optional Relations so for example, the GPS is an optional feature For mobile phones. That's mean that can be optionally included in the final configuration of a mobile phone There are another kind of relationships of hierarchical relationships like alternative or all Okay, in addition to the hierarchical relationship feature models can also contains constrict constraints That are constraints that are usually presented in the form of requires constraints and excludes so for example in this Feature model if our mobile phone presents a camera has a camera the high-resolute and the screen our Screen has to be a high-resolution because we have far requires between a camera and a high-resolution Fisher Another kind of Constraints are the exclude constraints. So GPS if GPS is presented in our final configuration in our mobile Our screens can't be a basic screens must be a color or a high-resolution screen because we have a discrete constraint This is an example of a final configuration of a mobile phone system Our mobile phone has core support screened with high-resolutions Media with camera So this is a feature model in my research context are really important because helps to To model the variability of variability in terms of systems and also they are really important because they can be Automatically analyzed in order to extract important information from them motivation Testing variability in terms of systems is extremely challenging due to the potentially huge number of the available configurations and their test so for example Debian we see is a well Linus a well-known Linus distributions has been has been reported as a variability in terms of system Has more than 37,000 of packages. So These packages can be configured in order to lead into millions billions. Excuse me off configuration. So Test each configuration is infeasible in many cases and time consuming To alleviate this problem many Contribution has been proposed So for example, this case selection techniques is one of the most used and one of the most important technique in this area This test case selection techniques reduce the test space by selecting a portion of the configurations to be tested However, the number of configurations resulting from this kind of Test techniques it could be a still high Unexpensive to run if you think imagine When you are and this is especially costly during regression testing When you have to repeatedly execute the test when any relevant changes made to the to the code Another kind of testing techniques For variability in terms of system is test case prioritization Technique that the schedule test cases for execution in an order that attempts to increase the effectiveness and meeting some performance goal Many goals can be defined So for example a test that may wish to order their test cases in order to increase the rate of fall detection for example Given a goal you can define different prioritization criteria differing order in criteria So for example, if we if we want to increase the rate of fall detection We can order our test cases in order to base on previous execution based on the fall detectives in previous execution or And based on the components that are propensity to faults Another important problem testing variability in terms of systems is how to evaluate these testing techniques last year the number of contributions to to alleviate the process of testing is Growing rapidly, but it's hard to find real software system real variability in terms of systems with available code with good description with good documentation with Fault reports and so on in order to Evaluate the testing techniques as results many authors in the academy Context and does that contest use artificial software and artificial features excuse me faults Random information in order to simulate the evaluation of these testing techniques our approach I'm trying to to to show you our approach to to resolve the whole problems Okay, the first step that we follow was a Look into the open source community in order to find real variability in terms of systems to Evaluate this kind of testing techniques. So we study and analyze tools like Drupal John Blah World Press Eclipse Presta shop ERPs like open Bravo and magento. We look for tools with available code with Backtrack insects to the backtracking system with bus related with the module that goes this path We also look for tools with with good documentations Was really hard to find this kind of Characters in these tools. So it's interesting. That's among all the tools considered in our study Only one of them meets all the specifications. Do you guess which one? Yes, Drupal Drupal has available code Drupal have the tail for reports a Severity and the screech and full descriptions Discretion about the severity of the false a status of false Asset to the backtracking system Also, most of modules in Drupal has automating test cases Drupal have an extensive documentation and all these Informations together with a high number of members in the community of Drupal more than 630,000 users and developers made that we chose Drupal as our case study Also, Drupal has more than 30,000 modules that make into a variability in terms of system Okay, with all this information we perform on a study To evaluate our proposal for testing techniques So the static is this one Variability testing in the wild the Drupal case study This work was published as an article in the journal software and system modeling that ranks Ninths in the ranking of engineering software Journals in the world. So if you are interested in reading this article, this is the reference Okay, so now I will try to summarize this word that inspired us to to to develop the smartest module The first step was Analyze the info files of each module in Drupal in order to know the dependencies of each Of the module with the rest of the modules in Drupal Also, we analyze we study the Drupal documentation and we use the module treats in order to To have a graphical representation of all the modules in Drupal and the Relationships among them Well, this is our first contribution contribution in our study We model a complete Drupal feature model, but we only consider 48 modules in our study This is the complete feature models. Okay, and this is just an example Okay, in order to show you an example of a part of our future model Okay Regarding according to them the Drupal documentation when you install and enable a new module in Drupal in the Drupal system you are adding a new feature So based on that we model each module in Drupal as a feature in our Drupal feature model Okay, so the first step was We select this we selected all the core modules that in module in Drupal I have to be always enable have to be always in a bus in the system So you you know nodes comments feel users then we Chose among the most install modules in Drupal I'm on the core modules optional core modules in Drupal. I'm additional module So for example, we selected core modules in Drupal that can be optionally included in Enable in Drupal like path and we included Additional modules like views or Google Analytics Okay All the core modules in Drupal that all we have to be Enable in Drupal has been model have been model like Mandatory features mandatory relations in a future model And all the optional Modules have been model like Optional features that can be optionally including the configurations. Okay Dependencies we analyze all the info files Dependencies of each module in Drupal and we Found 21 no redundants dependencies. I said no redundants because we found Redundant dependencies in Drupal like for example Forum has a requires has a dependencies with feel module But feel module is a model that always have to be enabled in Drupal. So this was considered Consider as a Redundant dependency so we eliminate all the redundant dependencies and we obtain As a result 21 no redundants dependencies. So our Drupal feature model in our study has 48 modules 21 dependencies and more than 2,000 millions of configurations That's just with a part of Drupal. We have a variability intensive system Okay Once we have the Drupal feature model we extract information from From Drupal in order to to try to guide the testing in Drupal So the first information that we extracted from from each module in Drupal was the the module side in terms of lines of code lines of code without counting blank lines and comments This information can help us to know the complexity of a module and maybe The propensity to force of this module is just an hypothesis Then we collected all the number all the chains in a module. This is based this fat is based in several Reported that set in Academic and industry reported that said that when you made a change in a code is likely to introduce force So based on that we collected the number of commits from the give repository of Drupal In a period of two years From May 2012 to April 2014 Okay, we also obtained information about the number of tests including all the modules in Drupal in our study all the module considering our study and we obtained 352 test cases and 24,000 and 152 assertions We also analyze the cyclomatic complexity of each module in Drupal and we use the PHP lock Tool in order to count in order to to calculate the cyclomatic complexity this information is really important because We know that cyclomatic complexity as Related with the propensity to force of modules Also, we extracted information about about the Drupal installation of each module the number of installation of each module We collected this information in order to to know the popularity the relevance of each module in Drupal and also we Collected information about the number of contributors and that's are working on each module And then the last information that we extracted from Drupal for the backtracking system was and the most important one Was the number of faults we collected faults in in two consecutive version of Drupal 7.22 and 7.23 Versions of Drupal and we also collected the fault during a period of two years from May 2012 to April 2014 Okay, we first collected all the faults found in each module in Drupal Then we eliminate those fault that were not affected by the Drupal community like duplicated faults like Non-reproducible faults and faults at working as design Then we analyze all the faults in order to find the fault that were caused by the integration by the interaction of two modules of or of different modules We obtained 160 integration faults in TOTA And From then we we found 132 Faults caused by the interaction of two modules 25 of them caused by the interaction of three of three modules And three of them by were triggered by the interaction of three modules interestingly The modules with the higher number of integration faults were C-tools and views Okay With all this information We perform a correlation study in order to In order to know the relation between all this information Started from Drupal and the propensity to faults of the module related with this information So the first correlation that we perform was this one we We studied the relation between the modules a module site and the number of faults in a module The result of this correlation was a strong positive correlation. So those modules with The higher the site of the module the higher the number of faults found on it. So this is a Really interesting information that we found in our study correlation number two We study Their relation between the changes found in a module and the propensity to faults of this of that module and The result of this study was a strong positive correlation So the higher the number of changes changes found in a module the higher The faults found on it. So that's really interesting to in order to guide the testing the third correlations We study the relation between the number of faults found in a module in a versions of Drupal and The number of all found in that module in the following version of Drupal and was really interesting because the Result was a strong positive correlation. That's mean that if you Have a module with a higher number of faults in a version of Drupal is likely to have this model is likely to have Higher number of faults in the following version of Drupal Correlation number four We study the correlation of the psychrometric complexity of a module and the propensity to faults of this module We obtain a positive correlation. That's mean the higher the the psychrometric complexity of a module the higher the propensity to faults of that module Correlation number five. This is interesting because We perform a correlation between the the relation in Between a module with a higher number of contributors and the propensity to faults of that module We hypothesize that those modules with a higher number of contributors Will have more faults in order to base on the fact that Coordination problems because of the higher number of contributors, but the correlations was negated That's mean that not only reject our hypothesis a bad Confirm the opposite. So those modules with higher with higher number of contributors Has less faults. So that's really interesting Correlation number six. We also Study the number of test cases in a in a module and the number of faults found on it We hypothesize that those modules with a higher number of test cases will have less faults Based on that they are exactly tested, but the correlation was poor So we cannot conclude anything in that case Another important information is that we found out The 91% of optional modules has faults in our case study I'm only the 41% of core mandatory modules has faults. That's mean too okay, so Basis in our Correlation study why we try to define Prioritization criteria in order to prioritize in order to order our test cases Looking for the propensity to faults of our Module in order to detect faults as soon as possible So we define four different Prioritization criteria base it on the correlation that correlated well with the propensity to faults So we define the scythe driving criteria for driving criteria change driving criteria and cyclomatic Criterion For example scythe driving criterion order our test case basis on the side of each module So those test cases related with with the modules exposing a higher number of Lamps of code will be executed first Based on our correlation study. We will get Ascelerating the detection of faults so The same happened with the rest of the prioritization Criterion that we define Then we evaluate this this proposal We try to to acts to to give an answer for this question Could we accelerate the detection of faults in Drupal using previous information? Well, we use the Drupal cases study and we Apply test case selection and we get an acceleration Detecting faults of 86.2 percent. That's a really good result. This method was This result was calculated with a metric is a standard method for for measure the acceleration of faults call APFD and then we apply Fault driving criterion To the selection to the result of the test case election and we obtain an acceleration Detecting fault of ninety four point two percent. So this is a really good result too so based on this good results we We ask us why not? Apply our proposal our study to a real tool to to the Drupal to Drupal system first we try to To analyze how Drupal test Is modules and is this excuse me and we try to To analyze simple test is the core module for testing in Drupal in order to know how work Simple test and in order to know how we can helps with the process of testing. I will show you the process that I Followed in order to improve simple test First I will show you simple test simple test has Show you a list of tests and group of tests In order to you can select the test that you want to run and then you click run and the tests are executed This is simple test. So this is the main view of simple test. So we try to improve the Tester experience in Drupal and we Develop a smartness in order to to improve Asma simple test. So this is smart test. It's just a first approach This is the main view of smart test okay, and Smartness and the main view of a smartness is a customizable dashboard With information with real information in real time So this is information about my system So the first we did that you can see in my in my dashboard You can't configure your your own dashboard is system information based on previous execution So you have information about the code coverage of your system of all the module installed in your system in your Drupal system You also have information about the test that failed in the last Executions and the the number of exceptions found in the last Execution of your test and the number of tests that pass and also you have information about the modules with less coverage Presenting in descending order and information about the modules with more faults and Also, we have information about the time that took the last is execution Also the second widget that I have in my dashboard is cloud of tech so you you can See worse on the side of the worst of the tax and represent the lines of code And the color represent the psychromatic complexity So you can see that views simple test see tool systems has a higher number of lines of scope and psychromatic complexity Yo, so we recommend you that you start to testing this kind of module because our propensity to force at least in my system Okay The third widgets We show you a top five of modules in the certain ordering the sending order of lines of code Also, we can show you a top five modules in the sending order of psychromatic complexity And this widget is really important to speak up show you The number of tests that fail and the number of exceptions found in the last test executions in the sending order to so in my case image has 24 tests that fail in the lax executions for the Exception, so we need to to to focus in this module in order to resolve my problems. Okay, and the last one We presented we are last test executions graphics with a top five modules in order to the modules that pass most Okay, so this is a customizable dashboard and you can add new widgets This is my dashboard that I configure but you can add more widgets. So this is the view in order to Add the widgets. So I will try to Create a new widget. So for example, you you can select top five modules of top ten modules or select the module that you want This is all the modules I have Enable in my system So I will select tough them modules For example coverage You can select the graphic ties line column and you can select the widget type graph it or cloud and the name code coverage through outcome Okay, and then generate and this is my new widgets I show you can see a widget with 10 top modules In the standard order based on the code coverage. So this is totally customizable You can remove the widget that you want And also you can save your dashboard you can create a new dashboard and save all the Configurations that you want in just dashboard and then load The configuration that you want. This is a totally a configuration customizable dashboard Okay, let's move on to this case prioritization This is a similar view of simple test but It's different because we include prioritization criterion We include criterion based on lines of code Cyclomatic complexity of each more module for fail test test with exceptions code coverage and Geat changes of each module So you can select the prioritization criterion of your test case Of your test suite that you want. So for example, I will select cyclomatic complexity and please look at Here Because the order Will change so Click safe configurations and the order of the test It changed because we order our test cases in order to the cyclomatic complexity in order to find those tests That's has more propensity to faults. So if I change the prioritization criteria We can Get another different order. Okay, so we show you information about the Cyclomatic complexity of each module of each test and also we show you information about the time that to the last Execution I think that this is important in order to know how many time How much time type the test so we also Show you a stop button in order to Include the time that you want for running your test Okay in hours or minutes. So I will show you an example of the executions of The test because we improve simple test with simple test you have to wait Until all the the test are finished have finished in order to to get results We improve this this view in order to get resolved when the first the first test had been finished This helps to the test there to to to start to the boogie and to get feedback continuous So you have resolved the first step the third test has been finished And this is all the information You can see information about this test class name status Message and you can also follow the trace Okay, so this Help us to to to start the boogie and to reduce effort to so the the tests are being Had been executing in the order that I selected in basis on the cyclomatic complexity Because we count of ten faults early at basis on on those on this Proposal okay, so if you click stop process Finish the execution and you can You can save all the information here You has all the information about the test that has been executed until now, okay? Also, we propose to to use several configuration of our test of our smart test module we include a view in order to to fill out the form To extract git change it from the git repository of Drupal So also you can configure the cron in order to automatically mining the repository with the git commits Okay, so this is just an example of Smartest No, no, it's not the end. This is just some screenshots Okay, and finally the conclusions What's improvement do we add to simple test? Okay customizable dashboard with runtime extracted data to to git the the testing test prioritization to the test faults faster Automated testing with feedback in real time and time up option to automatically stop the test execution Things that we are working on Add new widget to the dashboard Includes multi prioritization criteria based on genetic algorithms to the test faults faster We are working to develop this kind of algorithms Study the integration of a smartest with all the system like we had PHP units Get smart is accepted in Drupal now is in sandbox Any suggestion? Okay? Thank you very much