 My name is Ken I work at AI Singapore. Today I'll show you a tool which I developed on my own after I left DBS. I was in DBS technology team writing a lot of test cases for test automation. I was there for a year then I left the team you know just packed my bags went to Eastern Europe with my wife stay for a year and during that time I developed these two. You can use it for test automation you can use it for RPA robotic process automation and so on. So I just jumped right into the demo. Okay I think I need internet access so let me tether. Okay so I'll just focus the time on just doing demos then later on you'll ask me questions. Okay let's say this example I'm a connector. Oh I cannot see my wifi. Why? Okay here. Yeah I hope the connectivity is okay. Yeah so you can kind of use it to automate web interactions desktop applications and so on. So in this example I'll just use it to automate extracting information from M1 website. So the use case for this is my wife want to get a new handful number from M1 so after I develop the tool I simply write a script to do the automation like selecting her ver parameters and then kind of like script the content of the website something like that. So what this tool does is it converts the human language like Thai or syntax into JavaScript code so for one line of interaction let's say click here enter here type this so that one line becomes maybe 10 lines of javascript code that does the automation in the back end. So the foundation layer will be CasperJS PhantomJS and I write the integration layer directly to talk to Chrome. You can run the Halo's Chrome, Visible Chrome and so on that's fine and besides grabbing data from website you can interact with your desktop application as well. So after downloading this actually the actual run I did it for let's say 30 over pages for this case I just do one page as a demo. So what you see now the typing on the screen to auto application is done using visual recognition. So I supply yeah this is actually the it's not pre-recorded it's actually actual automation running yeah and then selecting images and so on. So by mapping this type of processes that people do in that you know their work you can kind of automate all these processes. The code for this example is a bit low but let me try. Okay for example the one to do the all look email sending that's all the code you need to that's the actual code already yeah the script to do the email sending. For the grabbing the details from M1 is a bit more complicated because we are trying to do something more than just getting the front end stuff for example we want to get information like the price you notice on the website just now you don't have price so you've got to write the normal java script code to get the price. So this is a script for getting the the M1 numbers okay okay I go ahead and do some other demos okay let me close this maybe okay let's say letter automation this is a use case from NUS they get the donation letters and so on they scan it as a pdf they want to extract the information who's the donor the address and then automatically print it as a letter to mail a thank you letter yeah so that's why you're seeing now is you know I'll skip the donation receipt part rigorous sensitive info I would just assume we have the info and then populate it in this letter and and try to print it yeah so things like this you can if you have a data table you have a spreadsheet you can do it as scale this type of stuff and no automate printing oh 100 letters and then after that of course you've got to manually you're putting a letter and so yeah so all this you can do okay yeah so let me go to the other let me just stop I think it's trying to quit and don't save okay okay so for this tool for those of you guys who are in test automation that's your developer you would like to do things like having CI and CD integration so that this tool does it as well let's say I want to do a test of something like this you know this is how a normal script look like there's a lot of commands but in general you have one step for each UI interaction step you want to do and each line converts into javascript code actually runs the automation so this example it goes to Yahoo Finance webpage do a search and then you know show the results yeah let me try to run it in the test automation mode so that you will have a xml file at the end which you can use for your Jenkins circa CI and other things you know so all these things like read search button to search tag actually becomes 10 lines of javascript code that does the back end automation so this is slightly different from cucumber and gherkins where for each English language line you got to write a matching code in the back end okay uh yeah so let me run show in the foreground so I intentionally add a negative test case so I search for Microsoft but I look for the word at apple right so intentionally to test this part then after you run it at the end you have a xml file which you can pipe it to your Jenkins integration it's a bit messy here so let me do it from here okay site samples oh yeah um then another thing for this is it has Python integration and R integration so okay somewhere there here but never just skip in the interest of time so it has Python and R integration by this I mean within a normal script for this tool you can actually write Python code and runs with your machine learning framework and do inferencing so let me do an example here something like that so this is a normal script let's say you go to a website do something you know you can pipe all this information to Python and get the data back all right just run same for R R is the language used by data scientists for statistical analysis things like that okay I'll do using a real use case maybe this one download profile photos so as you can see we want to grab the so-called duty base of researchers in Singapore doing AI and we want their profile photos right but if you go to a website all the images to a computer it doesn't know what are profile photos right you can grab all the images yes but you can't really detect out of the 10 images which ones are the profile photos some may be university logo maybe a style logos and so on so in order to solve that you know we can use this tool because it integrates with Python so let me and then use a any free open source Python library for face recognition I use the one that's very popular called face recognition and what it does is for example this professor right there's a few images on the website this one this one and all these are images so after getting all these images you know I pipe it to the Python integration and then do a face recognition using the pre-built deep learning model and get the results back so out of all this this four photos detected the script will be able to detect oh actually this is the face for that from my website so things like this you can do yeah so let me try to kill it yeah so if you're interested to find more just search tag ui then the first result you see will be the that project yeah and on the website you can have things like the tutorial which is show you a step by step how to do some of those things sample workflows a short slide deck and a video of my presentation at FOSS Asia three months ago yeah then these are the other stuff let me see there's also a Chrome extension for you to record your automation let's say you do a particular workflow you can record what you do and you'll spit out the corresponding script which you can run directly already and make adjustments yeah it can also understand 20 over human languages let's say i do a oh okay okay let's say i want to run automation that's written in chinese that's fine too most of the languages i actually use google translate to self-built because after i create a tool i just write script it goes to go translate get all the definition for 20 over languages and create the dictionary yeah so this one okay let's say chinese all right chai right so this file is written in chinese so this is how a automation script looks like maybe i just run in chinese right i just need to change the flag to chinese okay and i can run it um yeah in the visible browser so this example oh i got disconnected is it yeah somehow i lost connectivity oh i see the data okay sorry usually i set 200 so let me see whether i can yeah i can connect now yeah let me try yeah so yeah things like this running in chinese you can do as well yeah i think it somehow got confused i'll try to run it again so you can write a script in chinese english um hindi japanese korean that's fine yeah you know in this english human language looking like syntax which converts into code and runs let's say you have a script written in chinese but for some reason your users are based in let's say uh indonesia right so you maybe you do something like that i think it's indonesian for the language okay then yeah you can run the same thing i said this time the output will be in indonesian yeah so stuff like that you can do yeah so yeah that's all any questions yeah otherwise yeah just ask any questions all right so when you google tag ui you'll go to this page with all the documentation info it's free it's open source it's full featured yeah so just go ahead and use it if it's relevant to what you're doing yeah so any questions yeah really cool project oh thank you you have like a ton of stars on this project and i was wondering how did you get it so successful um okay yeah yeah of course open source is really cut through these days yeah trust me yeah it's it's very hard to open source it's like kind of being consumed by the business world and if your open source project becomes successful you have a problem because you have so many users that you can't support at as a part-time job yeah so i guess open source today in my own view is kind of like date because the only people doing open source and hot projects are the big organizations the small person it's very hard to do a project and sometimes you're mixed what if it's successful what i'm gonna do next right so at the time um when i make the initial release i think somebody shared on this hacker news uh and and this uh you just got trending yeah 200,000 uh 200,000 over stars then after that i got to do the usual marketing stuff which i look at the popular open source projects on github then i use a tool to scrape the users i use a tool to spam them email them i won't say spam but because those people that start those projects are actually looking for tools that are similar to this so in a sense yeah you know it's kind of like a line so um before i email a large batch or email a small batch to test out all right 50 then if those 50 the responses like there are more people that scold me they encourage me right i'll just stop emailing to that that the particular github project and i do this for a few github projects and you know over a course of one year become 2000 over stars and more and more users yeah so yes you got to do marketing as well it's crazy but to give free stuff you've got to promote it as well it's just crazy yes that's a good question it works for windows mac and uh nix order three os yes i yeah open source choose about three os this yeah i got a demo where you capturing the images of people in Singapore where did you get the names from was that also automatically extracted oh the names people have to provide yes the list of the researchers um yeah that part you still have to do it manually the url the homepage or the names you we have to curate from a list then from the list we crawl the info i forgot to mention you can do ocr as well yeah you have a pdf or text document you can do ocr to convert it to text directly within the tool yeah and accounting firm in singapore is using this to automate some of their iris process look into iris see what iris um let the pdf iris send to them to their clients and see what is the cost of action for those clients and then automate that part of the process it can just ocr as well uh i think doing test automation on staging system is too boring yeah for for for then so after a year i got really bored like i want to touch production system but in dps technically i can't it's a test automation right you only have staging environment to play with so um and then i left to see whether i can find a job in uh europe whether then we stay there for a year but it turns out there's a lot of terrorist stuff so i can't back yeah yeah even manchester is not spare right the second in 30 60 are not spare so i realized uh Singapore is safe first so i came back yeah and then the other motivation will be um i was never a developer for my 10 year plus career so i thought it's a good time to to try to learn some coding yeah so you know this thing is written in javascript php shell batch python and r yeah these things are all those languages i learned along the way yeah so it's more of a both self-learning and more of a curiosity yeah any other question yes that's that's correct if you see on a cheat sheet you see all the lists of the keywords is the usual ui steps like click tab hover you'll select rig and so on yeah then there's also the advanced steps like um integrating with your python your your visual recognition your r in javascript stuff you have that as well yeah not surprising but the people tend to script competitor website and they usually don't want to share that they're doing it yeah um and i know some users can write very amazing scripts that yeah i think the users use this better than me yeah uh generally think so yeah i'm good at just creating it but i think the users do a lot of interesting stuff yeah and some may use it to play games like you have a game that you've got to keep kicking to get increase the score you can write a script to automate that and look for the sudden outcomes on the screen to pop up then you do the next thing to click yeah you can do things like that yeah and those have been misused by for example spammers yeah of course yeah this is like a boss on matchstick right you can use it for good and evil yeah i don't decide for you how you want to use it i used to put a disclaimer please don't use it for evil but i think people won't listen anyway yeah um yeah sure does it work on pages with capture uh yeah that's a good question so the professional way to do it is to engage to capture solving services you send an api call to them they'll either solve it manually automatically for you or if their algorithm fail they'll have a team of human workers to do that for you yeah you paper per click yeah paper because you can do that yeah but you got to pay it yeah you have an api call just use the api uh step um then you know you can do a usual api call with hater uh whatever and the title get push put question whatever yeah you can do that yeah i think uh so you mentioned that they used your tool to spread data right so uh how about the concurrency part concurrency as in you need to handle yourself or your tool have already concurrency means waiting for somebody to appear yeah like they want to for example they want to create a website a whole bunch of websites in plan oh oh okay uh usually i recommend people to break up into batches it's supposed data table so you have a lesser csv file of one thousand records right you can actually just run it or you can speed it to half or maybe three let's say you have three thousand records you split into three files and run it on three pc in practice in theory you can run on a cpc but i realized chrome takes up a lot of memory so for my computer i run four sessions it's usually will start to slow down yeah yeah that uh the part is handled by user manually i don't want to make it too easy to do parallel run because uh you'll get me into trouble you actually yeah the use case for doing bad things in parallel run is usually more yeah yeah all right okay can you wrap this thing about the building blocks of this position oh sure yeah that's a very good point yeah um so under here how it works uh chrome depth to protocol i open web socket direction to talk to chrome directly using its native language because i don't want to depend on the chrome remote's interface project yeah because if you use other projects for open source project right when you have a bug upstream right you cannot point your gun to that guy to fix it right and he becomes a bottleneck to your project so as best as i can use everything like in build la feel myself but there are things which are used directly off the shelf like skoolie for the visual recognition part kasper js fandom js and slimer js is for the firefox part yeah this so these are the underlying tools but uh yeah one thing i forgot to mention uh this thing is so easy to run you can just stick it on any laptop and computer unzip and run you can put in a thumb drive to run as well you don't need to install developer nodejs environment or python environment you just uh you can just unzip and run uh any other questions yeah i think you're very cool how do you use any other meetups um do i go to any other meetups i usually yeah just quote yeah it should yeah certain go out here yeah i was telling my friends i got like got no life so i got spent a lot of time on this kind stuff yeah yeah thank you very much