 Good afternoon all. Hi. Welcome to our presentation today. I'm Luke McNeese. I'm the founder of cloudtroopers.com. We're here for the business showcase. I know we've got a pretty mixed audience here today, a mix of geeks. And we're here today to try and show you some of the things we've learnt over the last couple of years with some rather large US customers. And there's also probably decision makers out there, business folk who are here trying to evaluate Drupal. Does it work in the enterprise? How does open source work in a big company where there are strict rules and regulations? So hopefully we can address a lot of people. Hang on, we've lost the screen again. Apple, huh? Let's just go to the backup plan. Sorry, we had the Apple TV but we'll fix that along the way I'm sure. What would a trade show be without audio visual issues? It's just coming. It's thinking. We're back. Is there a way we can lower the projector a bit? The top's hanging off the top. Anyway, so the agenda. Quick overview of who we are. Showcase of how we've made Drupal work in a really massively scalable way. And then for the business folks, an enterprise checklist. You're out there evaluating other systems integrators. These are some of the questions you should be asking of what they can deliver. And finally, one of the tools we've built over the last couple of years is called the black box timeline. When you're in the enterprise and your codes running on hundreds of servers, it's often very difficult to debug in a production environment because you haven't got access to the production servers. So we've written a nice little tool which allows you to aggregate disparate data from various different servers and events and glue them together in a nice timeline. And that's what Mikey and the crew over there will be showing us a bit later on. A bit about us. We're a full project lifecycle development company, specializing in open source. We're not necessarily only open source. We also have Oracle and a few other proprietary skill sets out there. But primarily focusing on Drupal, Symphony, Vagento, systems operations teams. We've got great program and project management. We've also got a very, very special division in our company called the Continuous Integration Team, specializing in website testing, mobile testing and internationalization testing. I'll drag on a bit about that a bit later on just to show you some of the stuff we're doing. We're 50 people. The main office is in Cluj, Romania. There's about 10 of us in the UK and we've got a couple of bodies in the United States as well for our US customers. We have... There it is. We've only been for two years but this isn't our customer list. This is the companies we've all worked for over the last 15, 20 years. There's some mostly US companies you'll be seeing there, but a couple of European things here and there. A lot of travel background, a lot of banking background. We've got some in-house black box trading systems, high-frequency trading systems, right through to Universal Studios website in Orlando where we built the Harry Potter e-commerce website when they launched Harry Potter World there. We've got a wide view of the spectrum out there. One last thing. I think we might also become an IVF clinic in the last two years out of 50 people. We've had six biological deploys and we've got another three on the way. But Georgiana's up there at the moment and you'll see how that's progressing nicely. Today I wanted to mainly showcase Allegiant Airlines. It's a low-cost carrier based in Las Vegas in the United States. It's about 100 aircraft. It's very similar to Ryanair and EasyJet in terms of what they offer. You pay for everything. You buy a cheap ticket, then you've got to carry on bag and check-in bags and seat assignments and all that good stuff. They decided about two and a half years ago that their old technology just didn't give them what they needed. They needed flexibility, they needed scalability, so they embarked on a three-plus year transformation project across their entire enterprise. They're one of the few airlines in the world that have their own reservation engine, so they've gone out and written a huge Java-based reservation engine. We've written on the top of that Drupal and Symphony and a few other technologies to present to the world. They're an unusual company in the sense that most airlines sell their tickets through CRSs or GDS systems like Saber or Worldspan. They sell everything through their website, all the call centre which is also the website, all the travel agent portal which is also the same website. We maintain one source base across all three portals and all sales go through that. They're not a small airline. They do over a billion dollars in sales every year and in the last 12 months our Drupal Symphony system has sold over a billion dollars. I don't know how many other people out there for a competition. I don't know the word it's competition, but if anyone else has done over a billion dollars of sales on a Drupal site in the last 12 months, I'd love to know. It's a good competition. So when we started this, we wanted scalability. That was the biggest problem. The last website would fall over with a couple of thousand people on it. So what we did, we came up with a Drupal system for all the metadata for the templating for the workflow, all the good stuff that we love Drupal for. But what we decided to do was make Drupal a minor player in the actual live production environment. So we have Varnish. If it wasn't for Varnish, I wouldn't be standing here. I'd be out of a job. So when you come and visit the home page, which is number one, that page will come from Varnish. In there's a search box. You can submit a search. That search goes to Drupal, who has to register the search, but at that point Drupal's game is over. It submits it into Symphony. That's number three. It also gives a page back to the customer. So the customer is sitting there now with a page, with a spinning dial waiting for results. So Symphony then goes and does a query against the reservation engine at the bottom. That's the Java system. That takes a little bit of time. It takes its results and it puts it into the search results cache. That's a key value database at the moment. We're using memcached, but ultimately we will be moving to Mongo, assuming we can get through all the stress testing to prove that it works and it's looking great at the moment. But the key value database is not really that fussed about. But it has been the little engine that could this system. Handling more than 10,000 customers at a time quite easily. 100 searches a second. Very high end stuff with extremely fast response times. One of the fastest travel sites out there. So I think on that front it has been a great success. So what we did, because obviously you've got a production environment. You want to be up 24-7 all the time. You don't want to be down for code releases. And you also want to be able to do A, B testing. So we went with a three siloed approach. So one silo could handle the entire site load. We've generally have three in play at any given moment, except during code deployers, where we'll take one silo down to do code pushes. We can also use the silo three as a B test if we want to. Obviously we've got a lot of analytics running in the background to compare whether or not A or B is working or not. But it works in a very elegant way. So a code deploy is as simple as draining sessions from a silo. Takes about 25 minutes to drain all the sessions. We shut the silo down. We deploy the code through rpms and all the good stuff that makes it automated. And then we bring that silo back up and then we'll go to silo two and repeat that process. All up it's about a three hour process at the moment. We've got some things under foot where we should be able to bring that down to 10 to 15 minutes. But we've got to get a few, like I already mentioned Mongo, we need to get that approved before we can actually speed that up the entire process up to 10 minutes. Now I've made a point of any mention code at the moment. I'll address content deployment in just a second because they are two separate things. Of course in the enterprise environments you're going to have to get used to a huge process to get anywhere. Now, you know, Drupal when I started was a, you know, one machine system. You could push low life code. You could go and install modules, you know, all the good stuff that got me over the line. Like, hey, this is an amazing module. I can do what I want. But when you go into the enterprise, the network ops team kind of hate that. You know, they won't let you anywhere near the production environment. You know, at the bottom, I haven't even put development on this. It's a way down the bottom here. You've got your developers working wherever else. You're pushing into GitHub. You're building RPMs for content and code deployment. You go into integration. You get to play with all the other systems working at the same time. And eventually you come to QA where you all go in at the same time with two, three-week process of testing. Then that gets green-lighted. Then you go to staging for a day. And then ultimately you roll out to production. That's a three to four-week process which used to take 10 minutes but now it takes three to four weeks. But it buys you reliability. You know, with an operation that only sells things through a website, it can't afford to be down. Now, part of the problem with Drupal, and I'm sorry, I will mention a few bad things, configuration management. At the moment, it's all embedded in the same database. Now, I heard Jai's mentioning in the initial keynote that in Drupal 8 they're going to be addressing configuration management. And that will be a fantastic day when you separate content and configuration to separate entities. Because each of our environments in staging, integration, production, you'll be able to just change instantly your configuration files without having to write fancy drush scripts to go and deploy stuff. So what we've done, we have the Drupal master. That is where all the content is maintained. That's where all the code releases eventually make their way to it. But when we're in doing content deploys, we're pushing from the master Drupal out to the various silos. And I said content here, not code. During content deploys, we can push live content updates to any of the silos any time of day without bringing the whole site down. We've got a unique cash warming process using the cash warmer module along with a few other magical things. Because if you go into production with a cold Drupal cache, you will collapse in about 10 to 15 seconds with 5,000 people on site. So we've managed to engineer our way around that. And so 24-7 our business can deploy content without any hiccups to their operation. We're at the point now where actually the net ops team doesn't even get involved. The content owner can push content at 11pm from her couch sitting at home over the VPN. This is a little bit of a schematic of how we do it. The tricky thing with Drupal is the user database, when you can use sharding, but there's one user database. So you've got three production silos you're sharing users across all three of them. It means you can lose a silo and a customer can be bounced to another silo and they don't lose anything. Sharding will take you so far. There are some guys here with some amazing technology that I'm dying to look at which might actually make it even more scalable than I thought. But just keep in mind that the user's database is shared across all three of our silos at any given moment. We also have a no sequel cluster in there for search results, sessions in any number of other things. That gives us scalability and most importantly reliability because relational databases just wouldn't get near that without some magical sharding technology that I haven't seen yet. Now for the business folk, this is kind of what you're looking at for the enterprise. You're going to have to have a scalable and robust architect. I just took you through that. You're going to have to have scripted deploys. It's absolutely vital that you do not change anything code related live on a production database. It will screw with your caches and once you screw with your caches, it's game over. You can have a warm varnish cache, but a cold Drupal cache and you're to absolutely toast. You're going to have to deliver package software as in RPMs, Debian's, MSI's or whatever it is, you have to deliver that. You're going to have to have automated QA. We have scripts that I'll address later on that start from the very beginning in development and run all the way through to regression testing and real-time monitoring. Same scripts everywhere. You're going to have to have systems monitoring. There's any number of tools out there that do good systems monitoring. Our customer uses Nagios. It's pretty good, but there's a lot of tools out there. Just make sure you've got one. Penetration testing. We have a team of guys in Cluj that have come in periodically every few months to bang the crap out of our Drupals and see whether or not we can break it up to this stage. They haven't managed to break it other than a few little minor security patches here and there that we've actually pushed back into the community for things that we fixed. And obviously security patching. You just don't launch a site and just let it sit there. You're going to have to continuously have someone monitoring all the security patches and making sure they get out to production as quickly as possible. Right. So on software delivery, your customers net operations team will have a flavor of choice. Our customer is a big believer in Red Hat and they use Satellite and for the controlling all of their software and they use RPMs for delivery. If you're starting on a big long term project, you've got to start thinking about this on day one. If you start addressing that two to three months down the track, the cat's already out of the bag. As I've already mentioned, Drupal 7's configuration management is tricky and I'm being polite. It's one of the biggest bug bears of any system that I've used over the last 10 years. But nonetheless, you can get around it and hopefully Drupal 8 will address some of those problems. And day one, go down and start hanging out with the net ops team. If you aren't friends with them, you're going to have a sad, sad journey on the overall project. Just having that friendly face that you can go and have a chat with and just run things past them before you go out there. Because a lot of these big companies, they're scared of open source, they've never seen it before and they will freak out when you turn up with something and they don't understand what it is. I know this is kind of stupid but you'd be surprised how many projects go south because people don't involve the people that actually have to run it. Scripted deploys. So we use Drush, I don't know if there's another tool but it's amazing. When you go into the enterprise, you're going to no longer have GUIs, you're going to have no longer command line access and you're no longer going to be able to access the database. Which can be rather difficult when your website's having major problems and you're not allowed to have a look at it. Again, it's the NetOps team who is sitting there trying to give you feedback. We negotiated early on a host account that we could log into on the machines with read-only access and they'd only give us the password if there was a disaster. That's happened once in two years. So keep that in your back pocket, ask your NetOps team for a read-only account so you can get in in the thick of things to figure out what's going on. And obviously, I'll get to the job server in a second but your drush scripts, it's great that you can run them in deploys and from RPMs but also make sure you plug them into your job server so you can run things ad hoc whenever you need to. Now continuous integration. I've worked, geez I don't know, 15, 20 years now, 15 years on the web. Customers rarely are willing to pay for it. They don't see the value. It's the ace in the hole on this project. It has been one of the greatest things I've experienced in all my years. It's not a nice to have, it's a must have. Like a week after a coder started coding, meets with the CI guys, they start punching out an automated test. That test in development will then be used in QA. Once it's in QA, it'll be used to smoke test in staging and production. We'll also use it for live monitoring once it goes to production. All our jobs pump data back out to Nagios which then graphs it into nice little graphs. How fast can a robot walk through the booking path? How long does it take to pull back X number of pages? Any number of little tests. Basically for every use case, you generally have an automated script. So that means when you come to load testing, you can use these scripts and statistically look at 40% of people come down the booking path, 20% of people go through online check-in, 10% of people are looking on various marketing splash pages about Las Vegas or Orlando or whatever. And maybe 5% of people will be logged in editing their account. So you can take these continuous integration scripts, we use Selenium and plug it into JMeter and you can do amazing real-life testing. It's been a very, very handy tool. We use Jenkins as a jobsover. It sits in the enterprise and from that machine we can run everything. We can do code deploys. We can go and run drush scripts out into various production environments. From time to time we'll have caching issues like varnish might get out of sync. So we can remotely just log in to varnish from a single point of contact and clear the cache. Obviously that's security restricted. So we have to ask our nice NetOp teams to do that. But it has made us so much, it has increased our ability to respond in disaster situations. It also means if you build this up correctly your business analysts and the like can go out and just press a button, fire up a a Drupal instance in your development cloud. Look at the requirements for a given process, run through it all, test it and close down that all just using this Jenkins server to control everything. And obviously tied in with your monitoring tool of choice which is in our world Nagios. Another thing that I thought might be interesting before we jump into the demonstration of our remote debugging tool. I took a report of last month's hours. So roughly 33% a third was development. Now I don't know 10 years ago I think that probably would have been 80 or 90%. But if you're going to deliver quality week after week it's not only just development we've got a 25 to 28% of our time is on QA both manual and continuous integration. Obviously we've got project management we've got requirements gathering and systems ops. They help in the building of the RPMs obviously and helping our developers to take a piece of code package it up and be able to push it into a package that the customer that can easily install. And obviously training and handing over I'm a bit surprised it's only 2% I thought it might be a bit more. We didn't have a big release last month is all I can say it's usually about 5%. But as a business owner you're probably expecting to only pay for the 33%. There's a lot more in there that that happens to make sure that you deliver quality. Okay, we're now going to demonstrate our enterprise debugging tool. So if you've got 120 machines out there Drupal symphonies Memcache you name it. How do you in our context we have a booking path custom comes along they start a search they choose their flights they get a hotel they get upsells for rental cars and anything any any number of things along the way and that could be serviced by any of 120 machines. So how do you go through 120 log files and figure out what the hell's gone on. So that's what we built for this because it was an airline it was called the black box. It sits at the back we use at the moment syslog any machine can pass an event out on syslog it all gets aggregated at the back end into a single place where we can then have a visual representation of someone's experience in the site and by experience I mean not so much what happens on the website we use google analytics for that but what happens on the server side. So you might start a search but that might have six or eight different interactions with back end systems. So we've now built a tool that allows you to easily do that. And are you guys ready. All right I'd like to introduce to Mikey he's he's our CTO and co-founder at Cloud Troopers and Georgiana and Niku and Vlad. You can plug in or do you want the Apple TV? So what happened to us it's that as Luke mentioned we had quite a few issues with the code initially. We've got quite a few systems that need to interact. We've got the front layer which is a Drupal website. It has a java script pretty much large java script application that runs inside Drupal and then that one needs to talk to an intermediate layer which is symphony and then one goes to talk to a java system and the java system probably will talk to somebody else and I guess everybody soon or later will end up in that kind of situation. We have all kind of systems that need to actually work together and we discovered that it's absolutely impossible to do debugging. So if you go to I have a customer that actually claims that he couldn't reach a particular page or the booking didn't go through or something went wrong there was a probably a half a day sometimes even a day of investigation so our developers were actually more involved in tracking and finding the bugs instead of actually just doing their job. So the the main issue you've got with with this system is that every single framework every single system I've seen actually has a different type of logging. Everybody logs in files or databases or somewhere. And the other thing that's really really difficult to track is how to identify a single session end to end. The user comes in this is a page and then he just browse for him it's just the browser. So we're thinking like how we can actually put all this together and give an overview of the developer for the developer to to see everything in one page. Initially it looked like it's impossible but then we start thinking more and more and during our development we discovered a few other things that helped us to to build this thing. So if we can put everything in one page we'll take we estimated it will take probably three hours to determine the bug but after we start using this kind of system actually was taking 15 minutes or even 10 minutes to identify specifically because some of the the stacks where we're working with we're not under our control as I said the Java or any other system I know the payment system which we don't control and one of the problem is that there isn't the generic way of logging so we're looking at creating something that's it's fit for everybody and that's one big problem the second problem we got is that everybody is concerned when you do logging about speed so if you do a log and that takes I don't know 100 milliseconds you'll quit doing that because if every single page has 10 things that you need to log we'll be actually punishing on on everything you do so it needs to be to do testing is really fast right so we actually went to re-evaluate the project and we said that we need speed and we need a simple way for everybody to be able to connect to this logger to actually collect the data okay so what we did for for speed we we realized that the longest thing it's send a message and wait for the the system to store the data and then come back and says okay I've did it for the logging purposes it's not necessarily a must to actually get an answer back it's not something that you you expect that will not fail maybe it will fail maybe not you know it's it's kind of like no SQL nobody guarantees you that no SQL will store all the data consistently and everything it's it's there so you need to protect yourself against that but we're looking into that world so an API that makes sense to us and a restful API is probably the simplest thing to do I mean just a very plain basic interface where you just send messages and the system leaves you alone you don't need to be concerned or be bothered by like by what happens in the background this is a very basic the simplest thing you can do you're gonna have a time that you're gonna send to the to the logger you're gonna have a mask and the big trick in this in this system is to actually have a consistent glue system so imagine you have a Drupal and Drupal has a session but that session can be actually sent to the JavaScript stack where you say look this is the session ID use this one when you do the logging and the JavaScript talks to another service which is a symphony server and you say look I'm making a request and by the way the session I'm actually using is this one and that system goes to the next one so the only thing you need to propagate against against every single system you've got it's a consistent way of retrieving this data so you need to decide who's the master in our case it's actually Drupal Drupal actually makes a hash and then it's actually sent down the whole stack to the last level which is straight into in Java and then in the message because we're thinking you can't actually use a static way of static it's a schema type way of storing the data we actually accept anything in the message you can put there anything it's a simple JSON it's just a one line message or it could be a very complicated structure I'll show you that in a moment the the trick in here is you're gonna get a four or nine if you did something wrong basically you didn't put the minimum basic stuff into the message you sent so you need to give the time you need to give them you need to give them glues an ID and a message even if it's empty but the biggest trick of all is the 204 I don't if anybody in here ever used the 204 header there's anybody that used it no anything anybody right 204 it's one of the biggest trick of HTTP 204 says I got your call I'm not gonna answer to your call go away this is a very important thing and people need to start looking more to the HTTP headers the big trick is you receive an incoming call and you say I've got it so the system that actually makes the call can go away immediately we're talking about milliseconds right usually if you try to log something it can take as I said 100 milliseconds to get back as like I've got it I've stored it everything's fine with this thing what you do the first thing in the in the the header of the file you just say go away I've got it leave me alone which means that you can actually not spend time imagine that you use curl curl it's available or probably everything on the platform right it's the simplest way for actually interacting with that API and curl will go away in milliseconds we're talking about anything between five and 20 milliseconds it depends of what your network capacity and the lag but it's really really fast we're using this in in in other ways in our website it's it's really cool as a system by the way as a just as an idea you can use 204 to multi-thread PHP think about you want to start multiple processes you can actually fire process if the processor tells you 204 you don't need to wait for that process to finish you can actually fire another one and another one and another one which is kind of like not really used in the PHP world we use it and we've seen some massive improvement in in what happens in in the background so I'll try to show you small demo hopefully the guides of internet will help us so what we got here this is what we call the dashboard I'll try to make it a little smaller so you can see it can anybody see it now this is the the basic interface we're using for the the timeline it's very funny we started with a timeline and we're actually considering this a project that's like a black box sitting in somewhere and you work with so that's why we call it black box it's practically a timeline right so what I'm going to try to show you now it's how this thing works I'll I'll try to push some data into the system so I can show you how it works and I will do the live demo as you can see it really fast we run the unit testing on the back create all the information it just showed on the screen immediately now this is the glue that I was telling you and you'll understand in a second why this glue it's very important because it will help you to help the the user that has a problem I guess everybody got the phone call from from somebody saying oh this customer can't actually get into our website or something doesn't work and like what's wrong oh we don't know it doesn't work right and you can't actually figure it out and you don't know anything about that guy but there's a way to solve that I'll show you in a minute now this is I'll I'll just pick one of the items in here this is the what we call the timeline now what's in here we have three bars the bottom one is actually our based by the way this is not original work this is a project that we adapted it's called timeline you can find it on the internet has been abandoned about three four years ago so we picked up the code and we start improving the code because we consider it's a very very good way of actually seeing stuff so at the bottom you can actually see a bar that you can move and this is at the our level right so the next one this one is actually minutes level and this is second level so let's assume that we want to see something at this time if you double click we'll center in there and you can see what happened these are different systems we have as you can see at the top we have a Drupal we have a symphony we have a resweb we call it a resweb it's a java stack behind and there is a javascript they all log into the same same space now and this system for example you have an interaction from a javascript this says somebody asked for a flight and then we return the flight and this part over here shows the fact that that's not an interaction it's not just a simple thing it's actually a longer request so somebody asked for data and then we send the data back after a period of time it can be used to actually measure anything that involves time right the other things you can actually see are actually dots the dots are used to signify single moments in time somebody clicked on something right or they did a choice in a javascript part of the system or they did an interaction now the idea is that this stacks over here that you see they're not actually hard-coded I'll show you how you can actually create but as long as you create a new type of system we'll just generate more and more colors in there and actually can follow six stacks 20 stacks we don't care it's just it's a nice way of interacting with multiple systems so let's assume that we want to see what symphony did so we can actually uncheck all the others and we filter everything on the fly you can see only the one of the stacks you can see two of them it depends very much on what you're you're looking for now this just shows you the overview and you can actually scroll as I said in every single one let's go to the final step this is a complete booking from selecting the flight and going to the point where you actually paid for the flight now in this one you have a user that actually submitted the button and you can actually see that we have different services that we call to do the booking you need to validate the price that he he's got in the cart is the right one and you need to check that if he's got an account with company and as you can see they're actually stacked against each other and you can actually see how they actually proceed and how long it took every single one really useful when you try to debug and see why a system is slow because you can actually you have a page I'm not trying to to replace systems like New Relic or X debug but this it's it's really useful when you you look at somebody session particular session and the beauty of this it's that you can click on one of this and then you have the JSON that you sent right and you can actually see everything that was sent in there with all the parameters all the servers everything that you has been sent and then you can see it in a text form or you can see it as a raw form so pretty much you can do whatever you need with this information and this is the other end of the story that was a request and this is actually a response of the system what the system responded so you can actually debug if something goes wrong you can actually go and see exactly what has been sent and received and so on I think it's a very cool way of actually debugging stuff and specifically because this data goes between different stacks you don't have another way of doing it really really easy and now I'll show you another thing which is the website that we built as I said finger crossed hopefully look for a flight pick a date so right now you're gonna notice something in the header it's a hash or an ID or a session or whatever you want to call it if I go to the black box timeline which we're looking earlier and we go to the home page I got the same hash in here all right look at it all right it's the same one if you have a customer actually has a problem and says the webpage doesn't work for me I said fine send me your URL they will send you the URL they'll include that hash all right I can copy the hash I can go here I just go here I said this is the hash I'm looking for generating the timeline it's actually giving me what that specific user did input outputs all right and it's the underneath system it's not the the browser I don't care necessarily about the browser if he's got an error something went wrong I can actually bug it now I'll go in here and let's say I'll pick a flight another one and I'll hit continue we'll take a second it's a last but not the very popular destination a lot of hotels so it takes a while to actually get all the pricing for every hotel just to imagine it's quite a long list with hotels and information by the way all the all the content you're seeing on the screen is Drupal it's generated and and it's actually managed by the Drupal this is why Drupal is very important for us now if I go to the timeline I do a refresh that thing we'll show it here and as you can see it's a very complex call what we did here it's quite interesting is the 204 that I was mentioning you earlier the guy choose dates he's flying all right at that point I can actually start calculating a lot of things for him he's not just the hotels because next page is hotels but I can do a pre-emptive strike because I can actually go and look for cars for him and attractions and all other stuff so as you can see we're actually multi-threaded the PHP in here we're just calling services and say look this guy will want probably a car rental for those particular dates and please do it for us in the background we have a different services they're all written in and in symphony they actually go and talk to the java stack and fetch the data and then they store all the data in a location the beauty now is that if I click continue this the next one is a car page it's instant right because it has been generated already for me and I say I don't want this one and I go to the next page which is another one and this is a big one it's almost as big as the hotel with a lot of items on this page but this has been pre-calculated already for me right while the guy was looking for the hotel the this data was actually parsing and created and put in memory so it's as I said it's multi-threading in PHP but if I go back our timeline I'm gonna see that say this was the the big call over here but I go here and I can see here I did something in the shopping cart right and what happened with that one and then you go to the next level and you can actually see every single item what happened in the timeline and oh another thing that I I forgot to mention to you if you look at the system right where you have this big call if I start filtering have a look at the the other two bottom bars right while you're filtering it's actually changing the layouts and everything so it as I said it's a very interesting way of actually seeing what's happening in time and and doing the debugging and this is pretty much what we worked on and we're interested if you guys are interested to use something like this and please come to our booth get the card and drop us an email because one of the plans for this application is to make it open source but we want the community to actually come to us and say yes we we will need something like this and but I wish to actually have another item in there so if we we forget enough feedback we'll make it an open source we'll put it on GitHub and we'll we'll ask people to to help and contribute and make it bigger and nicer thank you so we bought this Apple TV which was meant to make our lives really easy but it didn't work well we've almost finished our presentation here today thank you Mikey Georgiana Vlad Nicu who put all that together we've got a booth if you'd like to stop by have a chat please do we've got a seated massage lady there Renata who will give you a 10-minute massage for free if you've got an office includes Romania we've also spun off massage troopers taking some of the things from my dot-com days in America where we used to have free seated massages in the office we now offer that as a service to Cluj to other technology companies and overall I just want to leave you a few thoughts I mean in in the enterprise on big multi-year projects don't sweat the small stuff I know that seems kind of stupid but really don't and don't spend too much time over-analyzing if there's anything I can take away of the last few years you know we're living in 2013 we live in a world where there's no single right option but lots of wrong ones just don't choose a wrong one it's simple all right now any questions answers anything hi amazing job you've done guys congratulations I have two questions the first one is how do Drupal and symphony communicate with their integrators so and the second one would be what is the back end to use for the black box thank you okay first of all Drupal posts searches in symphony and at that point once you're in the booking path everything is contained within symphony we have a javascript MVC application sitting in a Drupal page but Drupal has nothing to do once you're in the actual booking path all the data you can see is consumed by symphony through static views out of Drupal but there's no actual real-time activity with Drupal sorry what was the second question what what is the back end used for black box well we used it initially just to help us debug production but it's growing into a couple of different things you know it helps our network operations team gets to the bottom things very quickly I think he's interested in the technology behind him what technology yeah exactly that's a geeky question let me answer that one he's the smart guy I'm the technical guy so what happened we actually built this in having four items the first one is an API which is agnostic it doesn't care about it's built actually in Silex which is a very light form of symphony and because symphony it's now based of everything I mean it's it's the base of everything right it's god so Silex is actually picking up the calls and Silex is actually delivering the 204 that was telling you before that this one goes to a persister we call it a persister and in there we at this point we have a MongoDB driver we're working on a couch based driver and we're also working on a sequel driver which will be probably most probably my sequel the biggest problem we got with my sequel as you can imagine is the fact that you're sending non-structured data so we're trying to figure out a way where you can actually run queries there was another part of the demo that I didn't had time necessarily to show it but the beauty of the the system is that the next part which is the reader API it sits between the database whatever the database is and the viewer the viewer is just a JavaScript application you've seen and that part actually can be configured to do specific views so for example if you want to see in our case we have the one item that we're using which is let's see the last hundred bookings or let's see the bookings in the last 10 minutes and the problem with that is because the data is stored in a in a particular way we can't do it very easily in sequel but we're on top of that we'll do it so pretty much we're agnostic on the the data in the database or the the structure we're storing the data we prefer no sequel because it's really easy and can actually scale I mean we've got good experience with couch base and Mongo and we use that but it could be anything so that's why we said we're gonna open it to the community and there will be kind of a small spec for everybody to drive and to to build another driver make sense take it, take it I made you drop again okay well we're almost out of time if there's no more questions I just want to say thank you and sir that just asked the question we've got a gift up here for you thank you very much thanks guys