 So this is a fairly decent size buff room, but I'm excited that you were all able to squeeze in. So it seems that the unofficial theme of this Drupalocon is confess your sins as demonstrated by Dris. So I would like to tell you in that spirit about a self-reflection I had about six or seven months ago, and I was thinking about what I was doing, and I'd like to share with you that I've realized that I'm not so good at what I do. And if I want to level up the honesty level one more notch, I would even say that I suck at what I do. And obviously it makes me feel a little uncomfortable sharing with you. I feel that I alienate myself from everybody. So again in the spirit of Drupal of being all-inclusive I would say that we suck at what we do. There. That's perfect. And I say so because I think that our definition of what we're doing is missing one key element. If I ask any random person over here what are you doing, you'll probably say I'm a web developer. What are you doing? Web developer. What are you doing? I'm a web developer. Nobody says I'm a web maintainer. I maintain my websites. Because we're constantly thinking about developing new stuff and new stuff and new stuff, we hardly think about what we currently have. And to demonstrate what I'm saying I will give a very basic example the fact that we are breaking our design once every 10 days. So maybe you do it once every 14 days or maybe once every 7 days. But the fact is that it happens quite often. And again I'm not blaming anyone. We know how it is. A junior developer is coming. We ask him please move the logo 10 pixels to the right on the home page. We deploy it to the live site and two days later we realize in some unknown page the footer is completely broken. And when we're coming to a Drupal con when we are reading Twitter, when we are reading blog posts we keep hearing about big pipe and GraphQL and headless Drupal and advanced Git and whatnot but we hardly hear about how people are actually working on maintaining their sites. And I have another harsh statement that we don't care about our live site. So indeed it's a harsh accusation so I'd like to be a bit more accurate about it is our developers don't care about it. And it's not that they don't really care. They care about it. But they're actually afraid of the live site. Here I will dictate the tweet for you. Developers are afraid of the live site. Why is that? You have a team of five people. They're the senior developer. He or she are doing all the deployments. They have the credentials. None of the developers want even to get the credentials because they simply don't want to break the site. They are afraid of it. How do we see it? Well, developers care about their local computer. They care about the live site. And we can see it through the different amount of work that they go in, you know, improving their tools. So for example, this is a snippet of a Travis configuration. You don't need to read whatever is inside. You just need to look at what you have over here. This is some developer probably sat down for a good few days and worked and make sure that they are able to mock everything, the code, the database, Apache Solar, whatnot. We're seeing crazy stuff. I mean, I've even seen that there was so much testing going in Travis that developers had to split the amount of tests because it reached the 50 minutes maximum. So this is really excessive testing. And I'm not saying this is a bad thing. This is like best practices at its best. It's a good thing. How good it is? It's an Obama holding a beer with the thumbs up doing an impressed face good. It's that good. But what happens after the client has paid us a million dollars to make their site and we really, we've done everything, all the best practices, all the tests. Travis is working for hours and hours and hours. And then we launch the site. And what do we do? Usually what we do, we connect it to Pingdom at best. Now Pingdom is a terrific service, right? It's basically sending an HTTP request once a minute asking, are you up? Are you up? Are you up? It's okay. How okay? Adjusting Bieber doing adjusting Bieber face okay. It's that okay. But the problem with Pingdom, it has no insight. It knows just if your site, it knows only about the really, the catastrophe, if the site is up or down, if something really wrong with the application or something really wrong with the server. But it doesn't know anything about the key functionality. It doesn't know if you can do a user login, if you can add to cart, if you can see the recent news item. In fact, about a year or two ago, the Hezbollah hacked Drupal or Gael. And according to Pingdom, everything was fine. But no, it was not fine. It was completely hacked. So how do you know your site is working? You have to monitor it. But before we dive into the technical stuff and all the different tools that we're using, here's the really quick two-liner value proposition, the one that I can always tell people and see if they care about it or don't care about it. I'm assuming you have a site and the site is crucial for you or for your client. And you would like to know that things are working in it. Key functionality is working. Like I said, the user login and to add to cart and whatnot. And if something, if that key functionality won't be working, you'll be losing some money and you'll be losing their prestige. That's the value proposition. And now from that point onwards, we can see different tools that we are using in order to try and give an answer to that problem. So today I'd like to go over different tools that we're using. One of them is something that we've developing in Gizra for the past six or seven years. It's called chuv.io. Chuv means again in Hebrew. It's an open source project. We call it four developers. By developer we started it as an internal tool to try and tackle the problems that we identified in Gizra and like 24 hours later we said an internal tool equals a public tool so it's just out there. What we are trying to do with this tool is basically try to solve the big problem using simple solutions. So let's understand the problems. Basically we're talking about two main subjects. One is visual regression. The second is live monitoring. So what is visual regression? So with its most basic thing, visual regression testing is talking about having some tool that I will be able to give it the URL and then that tool will go mimicking a human going through the browser, take a screenshot of a site at a certain known time that we know that this is working fine and then it is getting a baseline image. This image is now our baseline and from that point onward we can rerun the script over and over again. It will go to the site again, take another screenshot and compare it with the baseline image. If everything is fine, everything is fine. If something is wrong, meaning we have a regression, it will create a regression image, it will also create a diff image so we as humans can easily see the problem. Over here you can see a quick video that I took of Platform S edge. All the errors that you see are deliberate. I've created them. They didn't really have a bug. This is only for demonstration reasons. You can see that there is a baseline image. There is some regression. We are able to identify it and if it's hard to see we can click on the show diff button and see the differences. So there are many, many different tools, open source tools that allow you, open source or close source tools that allow you to create those baseline images and do this diff. Basically there are two categories. One category is sites, different services, sites that allow you through their user interface to add the URL and it will take the screenshot for you and will do all the job. The second category is using code, different libraries. What you're seeing over here, I'm using WebDriver CSS. It's an open source solution. It's maintained by people from Sauce Labs and since one month ago I'm also one of the maintainers of it. What you can see over here, the code, again you don't need to understand too much of it. Basically we're declaring it should show the homepage and then I pass the URL and the .WebDriver CSS. That's the command that does the screenshot. The reason that I've decided I want to use something that is code is because I'm a developer and my team are developers and we want all our tests, all our code to be version controlled. I want it to be part of my Git workflow. I want to be able to do a pull request. I want to be able to do a code review on it. Another very important factor and it might sound weird because we created it, there's no vendor locking meaning all our data is sitting part of Git. We know absolutely no data that is sitting somewhere else. A very important factor of WebDriver CSS and again WebDriver CSS is a completely independent tool. It has nothing to do with Shove. We just created some nice integration between WebDriver CSS and Shove. One of the good things about WebDriver CSS is that it can run all the tests and not just Phantom.js meaning not only against a headless browser but against real browsers. That's pretty important because in the end we are designing and implementing the design for real browsers to be consumed by real people. You can use it against your real browsers in the office or have your own Selenium cloud that nobody has because it's super complicated to set up or you can pay 100 bucks a month or so to browser stack or source lab. These are third-party services which basically allow you to provision lots of different browsers on different platforms. What you can see over here, the gizra.com site is actually built on Jekyll, a static site generator. So whenever we're doing a Git commit we are running tests on the design of the site meaning we're asking browser stack hey, please provision an Internet Explorer 11 on the Windows 7 and check our design just before it's being pushed. And having that predictable, repeatable and locked environments that we're getting through browser stack or source lab mean we can constantly check the baseline and know that there are not differences due to suddenly something in the browser has changed. We're always checking against the same environments. And what's fun about writing those tests in WebDriver.css is you can actually write it once but target different platforms and different browsers. So again, not really important the code that's going on here, but basically I've written the test one. It just knows it needs to go to a certain URL and take a screenshot of either the whole page or certain section on it. And I can run it for the first time using a Chrome browser. It's completely arbitrary. I decided the Chrome signifies Chrome 43 and later on I can run it using IE 11. Now one of the problem and our problem as Web developers is it is so hard to really target all those different platforms, all those different browsers and all those different viewports. So again, a nifty feature in WebDriver.css I've written all my tests. It's completely agnostic to which size of the viewport I can set the screen width and tell it, listen, go into that URL now four times, one for a desktop size, one for a tablet and one for mobile. And we can actually take it even one step further and use the fact that Chrome can emulate different devices, iPhone and Android and so on and actually emulate, not just check the mobile viewport but really emulate a real device because maybe the tablet is working fine, maybe the mobile is working fine but when you really emulate it as an iPhone you might see regression. But there is a very important aspect, I think it's even more important than just saying I have features, viewports and browsers and that's the morale of the developers. I mean, we all love to assign the junior developer that just came in and tell them, listen, in the end of the office really far, far away and nobody wants to go, there is a computer, it runs Windows and it has IE11 on it, please test it. So, I mean, even if you're like a heartless person like me, it's like headless Drupal but less cool, even if you're a heartless person and you say, okay, I don't care about their feelings, I mean, just sit all day, you know, nine to five and check it constantly, check IE11 and IE10 and whatnot. I mean, in the end, they are just humans. They cannot do the job of a machine and this is something that the machine can definitely do. So obviously visual regression it sounds nice if we would have a static page but our sites unfortunately for visual regression are not static. How do we deal with dynamic content? I mean, over here you can see this video which I keep flashing and I have this flip card widget and I have the carousel that the caption is changing and every time the text is completely changing, the length of the text and the text itself. So how do we deal with that? Exclude, remove, hide. This is not just the slogan of some racist group on the run but this is a really command of web drivers CSS which allow us to deal with dynamic content. I had to take it out. So basically when we are approaching such a dynamic page, we need to decide what are the different strategies of how to deal with different elements. So when I say exclude, I actually mean I want to have a black rectangle over the element when I say, that was exclude, when I say hide, I actually want to hide the element but still keep the space that it is taking when I say remove, I want to completely remove it. So again, you shouldn't care too much about the code but I'm able to tackle it through code, through CSS or ExpassSelect or I'm able to say this part I want you to exclude, this part I want you to remove and so on and then we are reaching this part and if I would do a baseline image it would look something like that. That's the output that I would have because all the dynamic content, I don't want it there because I don't want to have every time I show this image I have this annoying dev come to me and say but hey Amitai, then you don't have 100% test coverage to which I reply hey, fuck you, up until now you had 0% test coverage so it would be fine if you just have 40% it's actually the first time I show this image but I would answer that exact same answer. It's okay if you have 10%, if you have 40%, if you have 60% test coverage it's much better than having a 0% test coverage and actually that annoying dev that was asking that annoying question kind of lives inside of me because I was telling it myself as well. I mean it's okay if my visual regression isn't doing 100% test coverage it cannot but there's some functionality that I want to check I have the text over here I want to check that some text is there on the page I would love to check there's at least one paragraph there's no more than four paragraphs over there and then we move into what we call the live monitor. So before we dive into the live monitor the visual regression part that I've shown you it can run against multiple environments you can run it again the live site it has it merits or you can run it against the dev sites finding the design issues just before they are being put into the live site or you can configure your Travis like we've seen earlier on every git commit every git push to check to check the design so the idea of the live monitor is actually to complement the visual regression testing with functional test and to test key functionality so it's not just the user login it's not just to add to cart these are things that we control so maybe we have tons of tests for that but we know that you know in the end all the tests that we do on the dev sites there we're just trying to mark the live site but it's not really the live site because the live site might be using third party services right I might be using Facebook comment I might be using Stripe.js for the cart I have no control over the code but I want to know when something goes wrong right I want to know that if I have Stripe.js cart and suddenly it's not working I want to do something about it I want to be proactive about it so maybe I can send it to my developers and they can fix something and they realize it's some configuration that got broken but maybe I don't have control about it but I can do something really important I can put even a message saying you know we have a service a service interruption we are aware of that and the value of it is if you have an e-commerce site and some person is coming to buy your watch for $20 and they see that something is not working there you just lost probably more than $20 because if you don't have any message and the site suddenly looks broken and well they won't have no trust in your site so you probably lost a client and you lost some of your prestige in the community and it's really hard to put a label a price tag sorry on the prestige so here's an example of a site we monitor we do not maintain it we just monitor it for a third party service and they had a problem this is a site that has private content and public content what happens when we have private content in sites sometimes they become public why just because that's reality and who finds it usually the client but not the client-client not the client that pays us the end user the one that actually got into the site as anonymous user by habit went into the URL expecting to see an access denied and realized all the content is open for everybody so they send a kind but firm email hey something is really wrong and the developers are rushing and fixing everything and what happens next we push a fix to do the live site and what usually happens is that the developer will check it just before it goes to sleep and the day after and then after two days it will start to lose interest because they have other things to do so that bug was fixed but what assures us that it will not repeat nothing maybe you know each one with their own God so they're praying a lot but nothing that's how it is and I mean if you feel that oh my God I know it happens to me everybody else are doing it differently no everybody are doing it like that I mean all the big companies and all the small companies are doing that so over here what you can see this is a be had test again if you don't know be had it's not very important we have some tests saying given I'm an anonymous user when I try to visit a certain URL then I should not have access to that page not that complicated right pretty simple even but very valuable so the idea of shoe is shoe doesn't care it's agnostic to which framework you will use to write your tests your live monitor test it could be be had or Casper JS or cucumber whatever basically it we're taking a very similar approach like Travis when you log in into shoe you log in or you register and log in through your github so we have your github credential for your public repository or if you allow us also for your private repositories and we show of all your different repository and just like Travis when you click a repository from that point onwards shoe will start listening to the repository and every every few minutes every hour you set the interval it will get clone your repository into an isolated docker container and run your tests for you in a way it's a glorified Jenkins right it's doing this task for you over and over again you could probably set up some Jenkins instance but what we are trying to do is make it easier for everybody and in a way it's not just Jenkins because when we are running when we are running the test periodically if we are hitting an error we're not immediately sending an incident we're not immediately sending an email because you know there could be network issues or whatever something could go wrong so we're running the test if it failed we're immediately running it on a different server on a different IP just to mitigate any any risk that something in our server is not working even if it failed for the second time we immediately run a third and only when it only when it failed for the third time we're sending an email and the idea of the test that we are doing on the life site is almost a complete opposite of how we are running tests on development on Travis meaning we're not running now a test suit of 50 minutes not at all the idea is to check really quickly the key functionality so your tests are running and you know running about one minute or so can I log in can I add to cart can I see the recent news item and when I say they add to cart I don't mean that you really need to follow the entire flow including going to PayPal and somehow paying and somehow getting a refund again watch out from that purist sound in your head that tells you well I didn't do nothing up until now but let's try to jump to do everything it would be probably enough to click on the add to cart and to see that the product was added to the cart we've already caught a bunch of errors that you know we didn't know about it before we did some live monitoring so just like Travis we have this dot shoe of YML file and we're trying to facilitate and make it as easy as possible to write the test so you can add a Selenium add on so you could basically test using real JavaScript you could actually see what's going on here we have encrypted keys so even if it's a public repository you could add you could add secret credential but this is the really nitty-gritty but let's talk about what's really the value of what or not what my goal is doing shoe so part of it being a millionaire somehow it's a nice goal but I don't believe it will it will reach or not I won't reach it immediately but my passion right now is try to make it not just a best practice because I get to show any person told them about visual regression and live monitor and they were like it's not important everybody's saying yeah it sounds right right I see a lot of things over here but if I would stop my presentation over here it would have stopped in the head nodding I feel so that's why I want to dedicate the next few sentences to what my goal is my goal is not just to declare it this is a best practice there are so many different best practices and you know in the end we have a project we have we are rushing to do it as quickly as possible we need to choose our different best practices so my goal over here is to make it a community approved best practice something just the same way that B-hat was 3 years ago when people hardly use it and now it's the defect of the way of writing your projects so I'm writing blog post oh sorry this one so a part of the blog post a part of the tweets the next thing that we've done in Gizra was create a yeoman generator so basically you can write yo shuv it will ask you for your URL and then it will scaffold all the files that you need to run your first visual regression testing to run your first live monitor testing and a dot shuv yml in shuv.io there is a bunch of different tutorials that can guide you through the visual regression testing because I know it can sound overwhelming this is for most people probably completely new and also the concept of doing the live monitoring but for the last slide or I'll say that the take away of this presentation beyond that we say that there is a visual regression testing and beyond what we're saying the live monitor and beyond the fact that we should address the fact that there are trivial things that we are all doing really really bad and they can somehow be solved is if you decide to go to take that path don't try to jump from 0% to 100% ok take a balanced approach I mean if you heard my the gizra way presentation I was saying that best practices need to be practiced even in gizra we haven't jumped and immediately started doing all our projects adding visual regression testing to all our projects but six or seven months later now 100% of our projects are with visual regression and it's enough to have one single baseline of your home page it's enough to have one single silly test that will just go to your live site and check that the user can click on user login and the form is presented and that's kind of all of it thank you questions you already know that I don't care about that awkward silence alright I can do a 15 minutes like that if they could oh my god actually doing contribution for Drupal 8 I don't know how to deal with that I should take a balanced approach but yeah you could do it I think there is an issue an issue on Drupal.org talking about different ways of running the test and this is I mean it's possible and again I think that part where I say there is no vendor login is super important I mean Drupal doesn't want any vendor with me or anybody right so the code the code is will belong to Drupal now the only question is if there is a regression for example where is the regression being pushed which is just to make our life as humans easier to identify where the problem is so yeah it can definitely use you any more questions alright thank you everybody