 Okay, so my name is Roy Nouriel, I'm the director of product management at Perfecto, this is my second time in India in Bangalore, and I'm extremely excited to come here again, I landed at 3 AM, you will see part of the experience that I had on my way from the airport to the hotel, but I'm excited to be here, I think that India and Bangalore are one of the places that most of the development and testing work is done, and for me it's a great opportunity to engage, to learn and to see how we can make things better in the future. This session is not about Perfecto, but for those who don't know Perfecto we provide continuous quality lab that support mobile and digital, and again that session is not going to talk about Perfecto, and if you have any question feel free to come later on and ask. 3, do you want to present yourself? And here is from Bangalore, sorry for that. Okay, so let's start. How many of you have iPhones over here if you can raise your hand? So it means that you are more tired than I am because I'm sure that all the others also have smartphones, right? If you don't have a QR scanner, now it's a good time to install it, you may use it later on during the session, okay? So we'll get to it later. Today we are going to talk about pyramids and how the approach for testing changed as we see it, and the main idea is to share with you a new concept for testing for mobile and digital applications. The pyramids have been here and not those ones, I mean those ones as an approach for mobile test, I mean for test automation in general for the last I think 10 to 15 years. Who is familiar with those concept? I mean, and it was mentioned earlier today, the pyramid, the unit, integration test. So there are many flavors to those pyramids and in fact they make a lot of sense when you try to understand what is your automation strategy, okay? And let's talk about what's the main concept behind all of those pyramids that we saw. So basically when we come and decide what should we automate and how we should build our automation suite, there are many things that we need to take into account. And part of that is what coverage we want to achieve. Another one is how fragile is the automation suite that we are building and the lower that we go in the technology stack, we have more stable tests. So if we start with the unit test, it's basically it's a white box testing, right? That usually the developers develop and they test mainly the functions, okay? As I see it, it's a huge safety net with big holes in it, but you get immediate feedback. So you get two things, you get a good coverage and you get a fast feedback and it's important. And that's why that concept recommended to have the majority of your tests as unit tests. So that's the first layer. The second one is basically an API, which is more a black box testing, which is you test the application, the components of the applications, but on the interface layer, okay? There were some discussions during this conference about how to build your test and that's an important layer. Again, you go one level up. It takes more to develop those automation assets. They provide a more accurate feedback on the quality of the application because they test it from the outside in and not from the inside out. And they give you a very good coverage and usually they are stable. By the way, usually traditionally those two layers were developed by developers, right? Who develop over here are unit tests? Okay, are you a developer? QA? Okay, so that it happened occasionally, but usually that's those tests are developers kind of tests, right? API is somewhere in between. And the more we go up, we have more QA traditional tests. The UI layer is basically, you know, we're here in Selenium conference. So Selenium and other tools that automate the UI layer, which basically give you a good understanding of the quality of your application and will touch, you know, what change when we're looking at mobile and web application. Does this make sense? Are you, all of you are familiar with this pyramid? Okay, any questions before we move forward? Okay, so what changed? Let's try to see what changed. Oh yeah, see that? Let's try orange. Oh dear, yeah, sorry about that. Very working. Well, could be application issue. But when you store that badly, it's on my desktop. We could try using a mouse to drag the badly to the trash. And after you've done that, you might want to launch the badly from the desktop. We already tried that a few times, I mean, all it did was mess up windows. Well, it might be worth waiting a couple of weeks. They've got the 90s blackweeds coming in then. Could you give me a date? Certainly. I'll help you with that. Yes, yes, I've also got a problem, to be honest. With my Apple. Oh dear, well, that is an old Apple, isn't it? When did you buy that? Last week. Last week? No. Now, could I have turned you out, or since then? What's the problem with it? Well, I've tried to put my dongle in here. I know much about these things. But my wife's seen a few dongles at her time. She says a little bit on the small side. I can do about that. I'll tell you what, let me try a boost in it. And yeah, I'll start to help you make well. Funnily enough, yes, my grandson's birthday soon. Okay. Now, it's already got Apple and a battery. I mean, have you got anything else? He might have left that. Well, we're doing a special overall needs. I'm in our company. I don't plan on them. The kids seem to not get an Xbox. There are a few things that changed. And let's talk about them. The pyramid was designed for 3D application. Okay. We're basically, the main logic of the application was in the backend and the client side was a thin client. Okay. The browser or any client was a thin client. So it made a perfect sense to test on the backend. Okay. So then unit test and system kind of test, API test made perfect sense. But you guys are here in Selenium conference and it means that you test web applications. Right. Web or mobile application. And as part of that, you can see, you know, I just took one diagram, but there are tons of diagrams. We all know that the majority of the transactions that are coming to websites are coming through mobile devices. Okay. So I met few customers last week and we talked about this test strategy. And basically what they told me, the majority of them that around 60 to 70% of the access to the websites is coming through mobile. From any kind of platform, it can be either tablets and, I mean, smartphones or any kind of platform. Okay. And that change the way that your applications behave, which mean that if I'm not testing that client, I basically test my application in isolated environment. Okay. That it means that I cannot predict the quality level of my application, which basically that's the main idea when we build our test strategy, right? To know how the application will behave at the end of the day. More than that, those devices are much more complex than desktops, right? They have tons of sensors that change the behavior of the application. So if I opened yesterday my browser to engage with one of the application, it immediately asked me if I give the application access to my location, right? Or any other sensor that I have here. How many of you test your application on different locations? How many of you test your application when you rotate the device? Okay. That's an important thing to validate because when we talk about web application, when you rotate the device, it immediately behaves different. And there are many cases that the application will either will crash or disappear. It immediately changed the location that the user was engaging with your application. We are much more sensitive in our era to quality because quality is visible for everyone, right? If we're talking about mobile application, so we have the app stores, and if we're talking about web application, social media is killing everyone, right? Immediately, if you release a crappy application, it's immediately in the news. And the sensor adds huge complexity here. So let's take a few examples, and I promised you that I would share with you a moment, and at Perfecto we called them the digital moment that they had yesterday. So I had a delay in my flight. I landed in Bangalore, exhausted after two days from my trip from Boston. I tried to order Uber, okay? And I was surrounded with people that were able to book their Uber from their phone, but I couldn't. That's my screen. You see, it happened at 2 AM, and it didn't work, which basically means that someone at Uber didn't validate that if I come with a phone from the US, there is a problem with roaming, and maybe something else, I don't know. But that was the experience. So I couldn't order Uber, and I went and I took a regular taxi, okay? That's immediately lost of business. And those devices equal to your companies, huge number of money. Did it change the perception of IT towards those applications equals money? That's where the money is going to form. Branches to software development. And one of our hobbies at Perfecto, we like to collect those embarrassing moments, and I can share with you tons of them, some of them related to the network environment, some of them relate to other applications that are installed on your device, as I said, rotation, location. I don't know where it's... I have a funny one here that your location is not valid. Please change your location, okay? Which doesn't make sense, right? So let's talk how we approach this change. So basically what's stand behind that, and that's a concept that we've built internally and we validated with our customers, and I would be very happy to get your point of view to understand if that makes sense to you. So what we're saying is that from automation perspective, it makes perfect sense to keep this logic of the pyramid, which means still we believe that the unit layer is significant important and we still need to invest here the same integration and the UI layer. But what we are saying, you already made an investment on the UI layer. Make sure that you're testing your application in a way that it will replicate your end user environment. And basically you're doing end user experience validation, right? That's what you need to do. And that's what changed. And that's why we called it the hourglass. So what we're saying is that you should take the UI layer and validate that on multiple environments. So if we're talking about web, web introduce huge complexity of environments because right now you have huge number of browsers that are getting updated very frequent, right? And in addition, we talked about mobile that you need to validate on multiple devices. So devices, we have the different operation systems, the different carriers, CPU. You know, we didn't even touch wearables and how they impact the way that you need to test. Okay, but that's one they mentioned. So if you have 10 tests, we'll touch in a second how you need which devices you need to select and you should multiple and execute those tests and multiple them on the right and relevant environment, okay? And that's a work that requires a joint effort from the QA to the guys also from the operation side that can tell you which environments your end user are engaging with. You need to know the landscape of your market and which new devices and which new browsers are going to come to the market soon because, for example, we're talking today, Apple released now all of their iOS 10 versions that are going to hit the market immediately because the moment they release it, the adoption happens in 48 hours, right? We know that and you need to be ready and part of that is also to test the beta versions that they release and you need to know the landscape. The second dimension is basically the environment, okay? So to understand what different networks you need to test, what different orientation you need to test and you can see over here the different parameters that you need to take into account and that's how you build your test strategy. And when we are talking about agile development process, you don't have enough time for that and that's where automation is crucial, right? Who today take into account those parameters when you build your test strategy? Do you want to share? Good. Basically what it means that if you're not doing it, your test cannot predict how your end user will experience your application, okay? I shared with you a few examples and we can share more. And the data is another parameter that I'm sure that you're taking into account, right? That's not new. Your application changed that perspective. Does this make sense to you or you think it's a nice idea, but not more than that? Makes sense? So I want to share with you, as I said, we're not talking about perfecto, but that's a free tool, okay? That you can use today. I will share with you in a second how you do it. That you can use today to understand which devices you should test. If we have it here already. Okay, so that's basically a tool that we share that give you the ability to build, understand which devices you need to take into account. We have few other tools that you can play with that also give you different browser that you should look at. So what you can see over here, you can define which locations are relevant for your customers. Okay, and that's important. Usually, by the way, you can get this information from the operation side. They can tell you where the transactions are coming from. It also helps you to understand what is the risk that it introduced. You can change what is the primary environment, what operating system, and if you have your own data, which devices are more common for you, you can play with it. So I need your help here with your computer because can you scroll down a little bit? So those are the devices that basically that we recommend you, and you can see over here the deviation between the different environments which help you to understand which devices are more important for you based on the inputs that you provided. Can you scroll a little bit down? So I will give you later on the link, but basically it also give you the landscape, which mean which devices are coming soon that you need to take into account. Let's move forward. I don't want to waste too much time on that. So I told you that you should have your QR code. That's basically a link if you want to play with this tool. By the way, the slides will be available later on, so you can also get it later, but that's a link to get this tool. One last thing before I hand it over to Sree. I've mentioned earlier that Agile introduced a huge challenge how you embed those activities earlier. There were a few discussions in this conference that talked about taking the assets and embed them into the continuous quality, continuous integration. That's a high-level suggestion that if you want, I can take it over and deep dive. That's a conversation that you guys need to do with your developers and to take your assets because if you're here, it means that you develop with Selenium, which makes perfect sense for the developers. Usually it's the same platform, the same technology stack that they're using to take that and embed that into their continuous integration server and provide fast feedback. When you do that, take into account both the environment that you're covering as well as the environment that you're covering. That should be part of your strategy and point of view when you talk about coverage. One exercise that I like to do I come to our customers and I ask them what is your automation coverage? Then they tell me 60%. Then we start to drill down. When you say 60%, what do you mean? After we have this conversation, usually it ends up with 20% coverage. It's a matter of decision, but they should be aware of what is the coverage that they have. At the end of the day, the user is the most important thing. That's what you guys are here. You're responsible for the quality and the engagement of the user through the different platforms. That's really in a high level before I hand it to Sri to give you some technical insight on how you can execute that in parallel and integrate that into the CI. Any questions before I hand it over to Sri? Perfect, good. So the explanation from drawing on shifting our focus or enhancing the focus from the traditional triangle of the pyramid was our class. So it is not only important for us to validate on the UI, but now giving the challenge of the number of channels through which a user can access your application is made. So this has to be taken into consideration. So the left part where we target on devices, right? And when we talk about the devices or on the missions, so we also talk about the OS, which are something, the OS level, maybe the browser which they are using, right? So on the left-hand side and on the right-hand side, the other important part, so as Mini says, it's all about experience. The experience is not only with respect for mobile in general. If the experience is bad, then you'll never go to that hotel or to that shop, right? So how do I capture the user's experience? So there are various factors which will affect the user experience. When we think a tester is an end user, then you can figure it out. What is affecting my mobile application? So the right side of our class was trying to capture the end user preferences or the conditions of your end user. So the demo part which showcases a pretty straightforward one, so this example just focuses on pre-learning the desktop browser, so which we are aware of like Firefox or Chrome, Internet Explorer. So we might have which browser we have, which browser versions we have. So actually we have Selenium which can take care much on this one. And the usage of so this is one of the insurance websites which is part of the decode to a space. So we have Selenium script which is running on this device on the local machine and on the Firefox. So what's important now when we talk about the left part of this device is that we have covered more. So it is not only the local machines or the local browsers which we are focusing on, now we have enhanced the execution towards the devices as well. So as Roy was pointing out, many of the customers say 70 or even much more percentage for many of the websites are coming from the mobile browsers. So it becomes important for us, so not only to run things on the local machines and the local browsers which are on the desktop but also the while browser. So in this example it happens, so we call it as the VAS to LOSHA sometimes, right? But nevertheless, so the focus is on the local execution as well as so we have to ensure that since work on the devices as well. So luckily we have the new concept of responsive web design which is being widely accepted and implemented where it becomes so where it gives us the flexibility to rely on one and execute on many. So the same case where we are running on the desktop browser as whereas the device browser. So it is important for us to cover the various platforms through which our business is carried out. So in this case so we have used the perfect thing where we are able to run the same script on the device browser as well as desktop browser. So one thing when it comes to the test automation the good part is we are able to save time and even the left part of the hourglass where we have N number of devices, N number of browser so how do we achieve these things? So not only automation is one of the pillar but to support this pillar would be the component or good cement which is called as parallel execution. So until and unless we do automation and along execute them badly so we cannot achieve the left part of the hourglass. So in this case we have used the selenium and along the combination of test engine so we are able to do more things so thank you on that. By the way, how many of you are testing your, do you have a responsive web application? Yeah, so that that started to become a huge huge challenge right? You need to test different screen resolution and the validation is a is a huge thing right? So that's in a nutshell what I want to share with you guys today we're here so we can you can come over, we can talk about how you can implement those kind of things and help you with that the concept as well as to talk about responsive web and some of the challenges that we see over there and would be great to talk with you, understand what are the challenges that you guys face right? Different resolutions and how you test them, the differences between real devices and emulators especially when you test responsive web applications that's a QR to the slides so you can get it from there. If you want you can come over and I will send you a copy of the slides. Thank you very much and if you have a question now it's time for questions. . So it's a good question and honestly I don't know the answer okay and I can tell you what we see okay what we definitely see that organization are moving and refactoring the website towards responsive web. They understand that the website is no longer good enough for multiple platforms okay and we see huge projects that migrate existing websites to responsive websites on the other side it doesn't mean that they let go to their native application or hybrid application the distinguish between native or hybrid is more around the engagement that you want to give to your end user on the cost of developing to different applications right and it's really depend what kind of application etc but usually they have both. The web application and native or hybrid application for different purposes. I can tell you that what I've seen that organization originally planned to develop a native application for tablets and they close those projects and move to responsive web application. So for smartphone usually the notion is that we would like to have native applications and for any other resolution we want to cover that with responsive web. From this report I see the belief of not other way of using hybrid. So I'm not sure that I understood the question. I followed you but I didn't see the question mark at the end. So basically that was what I tried to achieve with this session. That when you're doing it only on browser basically you're missing your users also for web. And the example that I gave some of them are for native or hybrid application but some of them are for web. As I said at the beginning most of the transactions today for web coming through mobile devices and then you need to make sure what happened if your application asked for location orientation. So that was part of the concept. Okay. Any more question? So I will ask you a question, a request. One of my commitments that I came to present here that in my sessions I'm not going to talk about perfecto. And if you want I'll be very happy to have a conversation with you about why we're different than the others but let's do it separately, okay? So you should know. You know there is a say that I asked not since I wanted to know. I wanted that everyone else will know but that's... Any more questions? So you guys are ready for lunch. So thank you very much and I hope that you enjoyed the conference and we'll have you to talk. Thank you very much.