 Welcome to building impossible applications with a Drupal and VJS. I'm Dane Pasqualele. I'm the CTO of Cold Front Lines and spot zero on Drupal.org. I'm Philip Tsikosima. I'm a Drupal developer since 2012, working at Statistics Canada. PDC-MA on Drupal.org. So, a bit of context. We'll talk about what this presentation is actually going to be. So, this presentation is going to be a case study on a project that we wrote for Statistics Canada. So, who are Statistics Canada? Yeah. Okay, Statistics Canada is Canada's federal statistical agency. It is heavily invested in Drupal since about 2012. We released the WECA distribution in 2012 and very alpha version. And we've been using it internally since then and a lot of sites in the federal government are using our distributions. And inside the federal government. Yeah, also. So, that's pretty cool. We are currently rewriting it in Drupal with a new version called WXD. So, Statistics Canada publishes a lot of Statistics and add a lot of datasets. So, it has interesting challenges of disseminating that type of information. And just so you know, the official abbreviation is STATCAN, not STATSCAN. So, you'll see STATCAN written everywhere. That's why. All right. So, give us context into the application that we wrote. We wrote an application for a program that Statistics Canada offers called RTRA, the Real-Time Remote Access Program. So, what is this? I think it has a very government sounding name. It's got an acronym. It's kind of confusing. The federal government checks it out. So, STATSCAN has very large data holdings. They collect a ton of data. The raw data that they collect from all the surveying and censuses they do can't be released publicly because you could glean public information off of it. If you sort of specify things enough the way you do with Facebook advertising where you sort of put enough restrictions on your data so that you're only targeting one person, you can eventually do that and sort of violate sort of privacy. At the same time, STATSCAN would like to expose this data to researchers. So, researchers can do fundamental useful research with all the data STATSCAN collects. There's lots of seats over here. You guys can come in. Is this the finishing of the briefing? No, no, we've started. We've started. Everybody's confused. Oh, no. Perfect. Yeah, all right. There you go. Oh, no problem. Don't worry about it. Oh, yeah, there you go. Oh, they're giving you a popcorn now? No, I'm not. No, I'm not. Let's continue into what RTRA is. So what RTRA is, it's a program that allows researchers to query all the data while mitigating the privacy issues by adjusting the precision. So the system will take a look at the result set that's returned before it's returned and will say, hey, you just got data that was about 10 people. So that means the precision you will get will be sort of in groups of 50 people instead of an exact count. Or when you're getting sort of statistics percentages, if you're taking a percentage of people and it decides, oh, you're taking a percentage of 10 people, well, the best precision I'll give you is into the closest 5% or something. So it mitigates privacy issues with how it handles the rounding of the data. And the way that this works to allow researchers access to the system, allowing them access to any of the data, is that researchers write their statistical programs in a language called SAS. They use RTRA to transfer that from wherever they are on the internet to induced back-hands secure network. They have a place where they can upload it and then some other things transfer it into the back-end which is essentially an air gap, but not really air gap network. The program is sort of scanned and validated automatically and then is run against the data sets. The system adjusts the precision of all the results so that there are no privacy issues and no methodology issues. And then it aggregates the results and transfers it back out to the researchers. So researchers can run statistical applications on data that they aren't allowed to see. All right. Some more background into this. As I said, the word SAS a lot and it doesn't mean software as a service in this case. SAS is a proprietary programming, a statistical programming language and software package for running statistical programs. The development on it started in the early 60s and it's had consistent proprietary release of the proprietary software package since the 70s. It's only since 2014-2015 that they've released any kind of free version and it's free just for universities and as long as you don't do any commercial works with it. And it very much acts like a language that was heavily developed in the 70s. We've put a context into what statistical programming is. There are a few statistical programming languages. Is anyone ever used one? I list R, SPFS, Math Lab. Oh, great. Good. You guys are experts. For everyone else, they're programming languages that are designed so that they're built to make it easy to run on large sets of data so that you essentially will have operations that you run but instead of sort of in a normal programming language where you check a variable, that variable actually represents a table and that operation you're running, like if you run times five, you're actually running this column of this table times five and sort of applies the whole thing to the table. And if you run sort of statistical operations on it, they'll grab the row of the table and then run the whole thing, run it on a much larger data set than in sort of a traditional programming languages. And so the results of a statistical programming language program are generally some sort of aggregate or semi or statistical answer. So what's the mean of this? Do a bunch of preprocessing to a table, to every row on the table, and then calculate the mean of one of the columns. Or do grab a bunch of data, run some preprocessing on it, merge it together in a certain way and then get the average of this column. And these languages make it easy to express that. So, probably not only of the background of what we're writing a program for, what problem are we trying to solve with this application that we wrote? So, can you take this? Can you take this? Now you can do this. So as some of you might already be suspecting, adoption of the RTRA, sort of system is quite low. And that's because there's a lot of caveats to just being able to use it. Programs have to be written in SAS, and I'm sure most of you haven't even heard of SAS before now. To make things even more complex, RTRA only runs a specific subset of SAS. So there's certain banned functions. So you can't go manipulating or touching any of the data sets or sort of keeping data or making aggregates of data that don't use the vetted processes that use the correct methodology. All of that stuff gets scanned out in a function of your program that will be rejected. So even if you are lucky enough to know SAS, you need to understand the RTRA variant of SAS to be able to write these programs. Because this is a statistical language, and you start with a big data set, it's more specifically to even get your program to run and to tell the system what data set your program needs to run against. You have to give your program a magic name that has some standards to pick out which data set you're running, and then put some embedded information in there. So if you screw it up, your program will be rejected. And because you're uploading your program somewhere and you wait and check, you get your results back, the sort of development feedback loop is quite slow. You don't really know how long it could take. You could wait an hour or half an hour and then be like, oh, I didn't know what to do with your program because you named it silly. Or named the file silly. So that's a problem. Well, let's say that's all great. You need a bunch of documentation. You need the documentation for the surveys that you're writing a program to do statistics against. And that is because these surveys have very massive tables. Some of them, the tables that they generate have, what was their biggest one? 5,000 variables? They have 5,000 columns. So that's not how many rows of data they have. They have 5,000 data points they've collected. So if you want to select what data points you want to look at, you need to know what those columns are called. That means you need the documentation for that survey data set. You need to go through it and then you need to look at not only what the variables are called because they have sort of a machine name, but then you need what the English name is, you can understand what it is, and then read the descriptions and then most of those are multiple choice answers for anyone who's ever answered the census. So you need to go and find out what one or two or five actually was so you have your translation code for that. So if you're writing your RTA program and you do your statistical stuff, you need to have this set of documentation this set of documentation to figure out what the things you're actually the questions you want to answer are. And there's a few of them. There's like three or four different places you need. Two different URLs, different sites, the different documents you have to download. So it's a bit messy. On top of that, in some of the surveys, there are deleted or renamed variables depending on what process they have to have. Your surveys can be different. There's more documentation for how you deal with that. And so these are all the things you have to keep in mind to successfully write programs for this. So let's come up with a solution with the client want. Yes. I have a slide about the clients later which we're very fortunate. But yeah, the client wanted some kind of graphical interface where they could allow users to create SAS programs without knowing SAS. Because a lot of their researchers are, for example, university students. And their researchers, they want to get some data. They don't want to spend time learning SAS. And people have to pay to get access to these big data sets. So universities are paying for it and students are not using it as much as they could and they need to learn all these stuff. And we've been explaining this for a few minutes now. It's a lot of information just on board before you can actually do anything. So they wanted to simplify this process. So they wanted some kind of user interface to be able to instead of reading your documentation and finding all the surveys that exist and then all the variables in that survey, we just give a list of users here's all the surveys you can or data sets you can play with and all the variables you can play with. So have some kind of interface that aggregate all this documentation and allow the users to actually write the program. So that's basically what they wanted. So before this came to us, they've been pitching this idea around internally for a long time and this project sort of whisked out a lot as this is not possible. You can't write a system to do this. And I would tend to agree especially with some of the pie in the sky ideas that happen. And it sounds reasonable if you don't know anything about programming and you haven't really used a lot of languages programming language can attest to they are either extremely limited extremely difficult to actually use in a kind of professional sense. If you need to write a serious program in it they generally don't cut it or terrible. They frequently are various combinations of all those things that are then terrible or difficult to use and extremely limited. And this is just sort of a nature of GUI programming and it's why a GUI programming language hasn't kicked the butt of the text editor from which they've purported to do since their inception. There are actually a lot of existing GUI SAS programming tools but they sort of suffered from these and they're either mostly for the GUI SAS ones they're just too complex to expose to the target audience. It sort of would walk you through it but there isn't any kind of supporting information for what you need to put in all these text boxes. So the previous developers but other developers who this has been brought up to were saying this is not worth investment it is just sort of come with the service that we're offering. So Sorry Yes So before how to tackle something that seems impossible and it's actually both down to having great clients and we've had different clients at second some are great some are less great These were extremely fortunate that these people came to us with a problem and not a solution they were like please build this exact way with this exact parameter and this color and this you know they were like we have a problem we don't know what to solve can you please research and come back to us and they were very good to just give us a lot of leeway of trying to figure out and then it presented to them in a very narrative kind of workflow So yeah I'll put down this show progress and collect smiles because they were always smiling which was considering we went into this project not knowing sass their criticism was always constructive and it can't emphasize how much then being on super on board for this really really helped this project so after meeting after meeting with them we sort of figured out what we had to build we need to build an application that was a program so it had to be very responsive because it had to be fast to interact with so had to be client side we knew we needed a CMS because of the monstrous amounts of metadata we needed to store the user easily as we come to find out we had tens of thousands of variables to store we needed to be able to check programs against variables to do some automated checking to make sure that what people were writing we could pre validate it before they need to submit it to the program so we needed to be able to access that from the CMS we need to be able to generate valid SAS from whatever was generated in the GUI and most importantly because it's going to be a tool released by the Canadian government it has to be fully accessible and fully responsive so you've got to be able to write SAS programs from your phone or a screen reader and multilingual we forgot that one easy peasy so the first thing we did is come up with an architecture of how we were going to build this thing so here's sort of our application architecture we used Drupal as our main CMS and then programs people would write, variables, data sets and a SAS compiler would all exist in Drupal to store all that and then we would put our reactive front-end GUI attached to Drupal that would be in VJS and that's what people would actually interact with to write your SAS programs it was kind of built as a decoupled application at first but we realized that actually all you need is a user interface for this so we basically attached it as a so it's Drupal that's providing it except it's just a client-side app but it's hosted by Drupal so just a little bit of background on VJS how many people know what it is here or have used it good crowd I'll go very quickly on it so it's a framework for a reactive component so it allows you to create really quickly a dynamic interface on a client-side the three words from the official VJS pages approachable, versatile, performant I like it more than other framework like VJS I'm purely a Drupal developer for the last six, seven years 2012 and I learned this is my first project that I learned VJS port so I loved it why? because it was very dynamic and I was just thinking of the pain it would be to develop this in a Drupal way with a lot of like the Drupal Ajax way of doing things I was like no so I wanted to go with more decoupled approach for a while and that was a perfect project to try alright so step one learn SAS SAS is a language with a lot of video sequences and a lot of magic there are really two variable types numbers, strings and numbers and converting between the two of them is strange it doesn't do it automatically there's generally a whole lot of magic and a whole lot of really really old feelings and stuff and a good example of weird things in SAS loves the colon it is an operator it is an operator that is used for like 10 different things and there's actually a paper there's a white paper I read on it that is about 30 pages long that describes the different uses of the colon as an operator because you can use it as a comparator you can use it as if you use it as a comparator with other comparators to use that comparator as a comparator between strings so you can use it as sounds like but if you use less than in that you can be like contains this substring in a list of strings so it's awesome and then when you can use it functionally, you can use it macros the language is great the basic structure of SAS is actually very straightforward but there's a lot of really tricky specifics just in terms of like what are you allowed to put in sort of different the way you're allowed to create variables and what you're allowed to call them and what you're allowed to call datasets is slightly different and so there's a lot of tricks in it so UX so we came up with some UX guidelines for UX values because how do we build this how do we build the programming language in a visual way the first thing was we had to limit the scope and that helps because archery is also a subset of SAS I can't do everything SAS does so already we had a small scope but even then we could limit it to just some use case of it like the most useful use case or basic level and then just okay you can't because we're not replacing what they have right now the way they do it right now is the program SAS in a text editor and they submit the file and then it runs in background and come back here it worked or it didn't work we're not replacing that you can still do that it's just a new way of doing it so basically our competition is a text editor so if anything is taking too long to do or was too complex to do if you had to click 20 times to do something that would take 2 seconds in a text editor you just like forget it we're not doing it it's skipped if something we designed was more complex than typing it into a text editor we just considered it a failure that was not the right way to do it our competition is a text editor we had to be better than a text editor so we either meant redesigning or dropping it as a feature so the main thing that was really good by doing this is that what we're trying to find the usage pattern that are really tough to do we're just typing some text versus having a user interface for example selecting something knowing what the possible list of choices like you have a drop down you have the list of things grouped by groups so it's much easier to select something and try to find what the name is and then go type it so in certain patterns I guess it was very useful there's actually one pattern that you could develop later sure we'll get to it it's pretty nice the last thing is we did not want to we did not want the user to have to know the restrictions of SAS before using our program so we wanted to hide that from them as much as possible so if they were going to call things like things that would not be legal in SAS variable names with disallowed symbols we didn't want to expose them to that we wanted to fix it for them and make sure that they got to from UI to working program as fast as possible so we started to figure out what you're just talking about this we started by figuring out what kind of things people actually wanted to do with SAS so we got a whole bunch of programs people had actually submitted to the system we started extracting goals so taking chunks of code and figuring out what patterns people were using and what were their goals in running these patterns and building UX not around sort of replacing lines of code but replacing goals so that the goals of different parts of code work and we can ask you a few questions and generate code that does that whereas if we're just trying to let you graphically program line by line it's going to be really, really onerous after we went through that we extracted about 16 different patterns that SAS programs were made out of and then we started designing components for implementing those and during that we started reducing the set of goals and patterns to sort of a minimal set of UI components we noticed at some point there was a lot of confusion and we were like we can't just merge these together alright so here's our pictures of our early notes I had to go back on my phone and start finding all these as we were taking pictures of scribbles of things we discussed as we started building how we wanted the sections to interact with each other where we wanted UX to be some pseudo code and how that would relate into UI then talking about the API like what API components we needed anything in there you want to talk about? yeah and then we graduated to whiteboarding we got a little bit clearer and then we can involve more people around so we did a lot of this actually we started with just like okay how can we build this component and we just like try ideas right and then try to code it and go back on the whiteboard and try it so this is what's kind of the user's work the design cycle basically so we would think, we discuss, we whiteboard something and then I start building it, I integrate it I make it pretty I test it and then does it spark joy is it good and most of the time it was like yeah it's pretty good no, right? if it fails or it gives value then we just go back in the same cycle until it's pretty good and then you're like yes this is the perfect component, I love it but I forgot accessibility I made the same toggle that was changing color and it was sliding in and out it was beautiful it was not accessible, not a single screen reader the whole screen reader can make a heads or tails no, it was gone off no, I was changing the label dynamically so yeah no there was a lot of iteration on this so it took about 6 months, 8 months just iteration different to build every component so that's basically our cycle oh, demo alright so let's show you what we ended up with and then we'll talk about its implementation alright so I won't zoom in no, we're gonna go full screen now because that's my interesting so the first thing that we ask you to do is put in a title our title is you can have whatever you want and we will actually rename your program when you download it to the magic name including your title so that it can run in our TRI the next the next section is you have to select which data set you want to actually use so we have a list of all the data sets we have do we have LFS in here that's the one everyone likes it's apparently the label for surveys the most popular one you can talk about this component if you want yeah so what you see right here is you select your survey and then it pops down a table that includes all the variables of that survey so let's say your staff can give you a survey like say the census but you get lots and lots of questions and each question is a variable and many of them like multiple choice variable so in this case you have this is a small one you have 163 years old some are like 2000 that's not an exciting one yeah those CHI things are pretty good they're pretty good okay yeah this can be a good one oh oh this one there it goes 1600 yeah so you get two columns here you get the variable name and then a link to more information about it so if I would click on this it would fetch all the information about that variable just some can you click on the one that's a multiple choice go to hell I don't know there we go yeah so this is what most of them look like the variable of the different possible answers so and you can see it can be a lot of them so yes that's a lot of data points right when you was talking about all the columns that's what it is so the idea is that I could search if I just want geo it would search a subset of variable and I want to select a few of them and I could just click on those and they get added at the bottom and select the variable list and then we can interact with the program so you want to continue you can start writing a program the way SAS works is you start on data sets and you operate on data sets so the first step is to create an initial data set and it's going to work on these two variables so it will be called step one and then we can decide what we're going to do from this data so if we want to calculate something and create a new variable we have new variable new var and we could say we're going to set it to the value of an existing variable and then we get sort of that linked in or we could do some sort of math between other variables if we wanted so select these multiply that one by four so if we're creating variables we can convert between variables so we could oops let's not put that there they'll just overwrite it a new variable just give this one a legal name excuse me can you keep the action towards the top of the screen? sure sorry is it all so from here we can convert the value of this into different things so if we have things that are letters we can convert them into numbers or we can use suppose custom sass converters so that gets lots and lots of fun so getting into sass as a very expressive data converging system we'll just leave that there the whole time yeah so the whole time while you're doing this we'll put it in full screen mode oops my compiler failed this conversion must convert character with data we're just going to get rid of that yeah we're going to remove that so the whole time while we're doing this this VUGIS system is reactive every time we make a change it sends it back to our system to be to compile them so if we start changing this number our sass code also changes one of the more interesting functions we put in is a lot of the patterns we found is what most researchers were doing is they had a column when they were done their program they would take the values people put in like they take the values of the column and then translate them back into what that actually meant in terms of answers so for example this variable it stores like 1011 but what it actually said to a person filling out the survey was eastern regional and so at the end of the program people often sort of wrote that in so we'll call this text this so we wrote a step in it that would actually generate the sass code that did that so since we have accessed all the variable data we can create sass code that fills out the column text on the result table with actually the text values we saw a whole bunch of example sass programs like real ones that people were submitting so people submitted programs with this code in it sass is not that ever so we need to do that the other thing that is on all of these clauses is we can add code comments anywhere one so where is that code comment going it's going at the top of the program comment one so we can add a comment to this calculation does something and it'll just fill it out as we do that we can also add conditionals around everything so if you're really if you want to change this it's equal to that so then we can automatically build this for advanced users so in data sets we have a thing just to limit data so in these data clauses it'll essentially go over every row and run these to generate your output table you can have limits so that you can say exclude these rows from the result hopefully that we also provided a box just for people to dump in custom sass so if they knew what they were doing they can still use this interface and then the meat of this is at the end when you generate sort of these data tables is you have to run statistics on them a statistical process to calculate down to what you need and so you can choose what statistical process you want let's say frequency out data set so as to choose we want to run it on step one this would be tweak we'll calculate the frequency of this variable and then it will actually we have the sass code for calculating the frequency here at the end you can just click download program and it will download the sass program with the file sort of name correctly to run in RTA so that's sort of what we ended up with you can actually write sass programs so let's talk about the implementation so yeah the concept was basically at first we were like okay we'll just do a couple application but then it came up that well actually it's we have a ripple cited as all this data all we want to do is generate a program and a program is stored in JSON so it's basically a JSON structure that we define which component that you have what are your values and we want to store that in a text field like or in a dupal field so what happens on the display side of things is that it will instead of displaying like a text area of you just replace it with our view app and every time we change something we just write to that field so my journey into this I didn't know any of my view before that maybe a tiny bit oh yeah quickly oh yeah so that's the best way I think you should learn something new like this is to find something very challenging to do with a lot of current cases that are really tricky I really like the rectiveness of view for this because it was since we're building a data-driven application use a data-driven framework so it was good it was very fun the feedback was super fast I liked it yeah key points okay the application is starts with a JSON config so we define different type of component which component can include other components and it's busy scaffolding in JSON and then we have a central storage layer for here oh yeah one thing I need to mention is I built a lot of custom component like I could build get a lot of view library from the internet but because I had to iterate it was very specific type of workflow I use a lot of custom components for control but also use a third party for all the tediousness of building a table viewer or drop down select list that's computed so this is basically what this is the meat of the project you have the dupal backend that has endpoints like it has a variable and then you have a compiler and the view app in the middle here it's a core it's called ux is central storage layer of the app that it's all a business logic stays there and then it interacts with the UI when you change something in the UI or you change something in the store it will interact with the UI it will change the UI so you have the meat of this the store in the middle, the UI layer the display layer so whenever you change something for example I change a variable it does a call to the compiler here's my new program and the compiler on the dupal site compiles it and send my new back to the compile code this is the meat of the app and if I drill down a little bit more in the view app it has a new store like with the JSON I was talking about and then it has the store is divided in some modules that allow me to do more a single responsibility principle saying this module talks to the compiler this module deals with sample programs or just the surveys or just the program itself is where it stores my program and then you have another layer of components here you see the white one the white circles are all the custom ones I built and then there's a few of them in our third part like I mentioned I get a table view or a depagination or like the multi-select combo box and then there's another layer in view called plugins I use a few of them when you saw when you were deleting something you had a confirmed dialog and the other one is the ITNM that's another thing I really love with the view process is that ITNM is very similar to like a T function in Drupal so if you use the Drupal way of things it's basically the same thing here you can see how you can reuse components inside it as you saw in the demo he was adding like calculations and he was adding like a process inside another section you can see here you get with the blue box around you get process template and another process template below is I'm using different type of components as you can see in the previous slide I have UI components that are purely containers and others that are processes like by processes I mean like what's a calculate or conversion and those are processes but others are containers like the this process template the job of a process template was just I don't want to redo the UI every time so it's basically it's a section it has a header it will include this UI with the button to delete and reorder things around the comment button so I'm reusing these patterns so I have these containers you have these in the middle you have input elements input elements is all the buttons the text boxes and these I kept them as dumb as possible at first I was trying to make them complex very versatile but the problem is you need to pass too many parameters into them so I was like ok no you're super dumb all they do is they know their internal state so you just push what the new state is to distort and distort you with a distant logic so the back end so what is this attached to so at the end we built what I'm going to call a partially decoupled application the VJS component was essentially behaves like a CK editor WYSIWYG it binds onto a text area and it puts some data into it we have a massive amount of data for all of those variables and things that were in the drop down list those are all nodes and then there's an actual compiler that compiles the JSON that's generated by the view front end in the SAS code the Drupal components for the most part are very simple we used migration to create all the pieces of content when you click on the variables that is actually just loading the node view pages of those variables in that dialogue so very simple the end goal of this is to let people log in and write their own programs and save them so programs can be nodes so we actually made a field widget and field formatter for that view program so that you can make a text editor be the RTRA SAS editor and then when you're viewing that field that text field you can make it render through the compiler so you see the compiled SAS and then I wrote the SAS compiler as a Drupal service it's good I can get through this quickly so the what we were trying to figure out how to go from JSON to actually SAS was looking at a few things but in the end we realized that we really just needed to actually write the compiler and there's a lot of precedent for compilers in PHP we all use them every day twig is actually a full on compiler it compiles full twig templates and then the PHP code is executed to actually make the pages show up and so this we do a similar thing with we do something similar with ours we actually wrote a full compiler to go from JSON object to SAS code and all component compilers are essentially architected this way and go through these steps the first step we do is lexical analysis so that for sure in the JSON object there's not that much lexical about it but it is sort of breaking it up and making sure that all the components we need to be there and you remove the superfluous stuff so while some compilers would remove spaces our sort of remove state information that gets stored in the array for the front end that the back end doesn't need all the values for options the user has put data into but then not selected all that gets removed and we actually get the core of the application then we do syntax checking so we go through the all of the data in the object to make sure that all the parts are complete enough to actually make a sensical SAS program with so every time they're referencing variables or referencing previous data sets we make sure that all those are declared in the correct scope then we do type checking which is where you run all of your program checks in there in our compiler you have a long list of different type checks we need and those are the things to make sure that your program is actually in things are in the right order while you're writing you have buttons to reorder the different clauses so you can actually make invalid programs quite easily but we won't let you do that because sometimes you need to reorder programs but we're not going to do that we need to let you try and then give you an error so that's the phase where we do type checking as we're getting closer to the end we there's an intermediate generation so almost all compilers will generate an intermediate code before generating their final code if you're and that's the case even if you're writing your final code is sort of x86 assembly or machine code before you go there you always compile into an intermediate code and then you get that to a compiler that will compile into your individual frameworks for what we're doing is that is breaking all the different sections into their templatable sections with all the arguments they need so that the last section code generation where we actually generate the SAS code for our project we can then take our intermediate code and then template it into the actual complete SAS code and that's about it that's the architecture of this application and it surprisingly works very well it's been a pretty successful project what this data was amazing we built mostly the back end and I built the front end we're trying to find some API that we could talk to each other but at some point we just decided to put it together and it just worked it was like wow, I would never expect that it was amazing there's also a really proud moment we had on it is one of the first program that was outputted on it actually ran in our QRI it was very simple but it actually ran that was sort of a good milestone a little over time but if you have any questions I'm sure at the back there's an application for a federal website how do you tackle accessibility using a Vue.js um yes so yeah the requirements we had 2.0 I think now it's going to 2.1 so like AA so it's quite stringent the idea is that the Vue.js outputs HTML anyways so I just need to make sure that every button or every text field has a label every button has it in text that explains what this button is and the tabbing there's a lot of work that's been around the tabbing fair Patrick's sake I just have to say that JAWS on IE is terrible we still have to score IE right and they test a lot in JAWS because a lot of people are still using JAWS it's one of the biggest screen reader on the market and IE is a weird thing plus JAWS doing weird things to the DOM makes things tricky currently this is not live yet we're going through rounds of XCV testing I'm hopeful we fixed a lot of issues quickly but there's a few lingering ones especially with complex combo box drop down option group that you can search on JAWS doesn't like those if you don't need to go as far as having it that's a change from WCAG 1 to WCAG 2 you can't say what JAWS because it's a requirement as long as it works in an XCV way as described yes the first one is what does JAWS do when it's a button because of the massive amount of content and then the requirement that in the sort of phase 2 that they wanted people to be able to save programs online go into a portal write a program in the UI and sort of save it, have multiple programs have various user accounts where they can sort of share them in things and so we have this sort of very highly dynamic system we also sort of have all of these very very traditional CMS requirements as well that's why we built this sort of hybrid system because no matter what we did we need to figure out something that would manage, I think we have like 40,000 nodes in the system to store all the variables and data sets and things so we needed a robust CMS to back it oh one thing I didn't talk about I sort of mentioned it on the slide we use the REST plugin for views for all of our web services because JSON API just dives with the amount of data we're sending back and forth so we discovered that like if you're going to send 1600 nodes back and forth in one web service call you need to really craft it in view sorry the views, no view to only return the data that you need so that you can make it as small as possible because even though view loads all the nodes anyways they have a major performance difference how did you do the test rounds for this project and how did you do the testing before that? so we went through a few rounds of testing and a lot of different people tested it it sort of deals with having really good clients the client actually before went to accessibility testing and the client had their teams test it to see if they could build programs with it and they actually brought it to one of the SAS development teams and had them write SAS programs in it and we got a lot of really good interesting feedback from them and they really they really ended up reinforcing one of our development values because a lot of their complaints are like people can do stuff here and it's illegal in SAS people do stuff here and it's illegal in SAS and then it was like people using our UI shouldn't care the UI should fix the purpose of our UI is to teach you SAS, is to make your program run to get your SAS program and maybe teach you the abstract concepts but not burden you with the minutiae and so that really helped and then they went through a lot of accessibility testing and that's really what has been in the bottleneck because as you can imagine the accessibility team it was fairly donking when this arrived and they were like it's only one page and they were like you've never seen anything like this before so it took a long time to accessibility test and it's still got a process that's ongoing and yeah, so how is testing slowly, slow it's like going bad as you'd expect would you consider something more of an automated testing? yeah, you wrote that I started playing with a test cafe I think I didn't really go far in there it's in the plans to eventually do a lot of automated testing but it was changing too much to actually implement tests so now once we're done with accessibility then I want to implement a lot of tests to make sure that whenever in the future we change something that it actually works I would test a fun with a test cafe and you could test it back in the back do you find yourself getting confused when you're talking about view and views? all the time we solved that problem so I call it view and views we call it view and views so BJS and then you know how to problem next question well thank you very much everyone