 Okay. How long do I have? How many minutes do I have? 30. 30? Okay. Alright, nice to be here. Today I'm sharing a little site project that I have, which is built with electron. I call that Space Radar. This is just a different title. The things that I have learned building this app, and I know it's called Space Radar. It has nothing to do with the outer space or the radar that users bring away. Because I think there are already a lot of these applications out there. I just thought that this radar is like one of the names which must not be used. But when I actually googled it there, there is actually something really called Space Radar, which is like satellite in space. I should think radio waves. This is electron and I thought it's space and radar. And I think Simon last year gave actually a space simulation. So this is a different kind of space. So I'm sure, I think two mid-ups ago, I think Chee-Yan actually gave a talk on electron. I mean, it was kind of like getting excited on electron. Today it might be getting a little disappointed with electron. I wasn't around but I actually caught his video. So thanks, engineer. So how I started Electron was, I was actually at the Taiwan JS JavaScript conference. There was a workshop on electron and I thought, okay, let me just try it. This is really cool. I think it's a very comprehensive presentation. Anybody who wants to learn more about electron or building cross-platform desktop applications with electron, you can click on the link. So that was what I did. I was sitting there and I heard about electron, so let me start doing that. It turns out to be pretty simple. First you need to have a main JS and then you require the staff. And you see that you add event listeners, windows are closed, the app is ready, and you kind of open the new browser window. So the browser window is basically a Chrome browser and that is actually basically your application, your electron application. Next thing is index.html. So you're basically saying, open the browser window and load up index.html. If you see over here, there's a file directory name, index.html. So you're basically just, I got a typo there, but okay. It still works. So this is HTML, it still works. So I wrote, okay, a simple hello world app. This is HTML, web technology. And how do you write electrons? So one way is that if you have NPM installed, you have node installed. You can just NPM install dash G for global and you can, in that way, install that electron previews into your system. And you just have to write electron. Which basically means run the main.js in your directory. And I said that you could create an electron app in less than four steps. So by the third step, you are already running an electron app. So if let's say, do you want to let me do a demo? This is really hard resolution. Okay, so yeah, this was my project and actually I decommit the first app. Sorry, I'm a little nervous. And you can just run the electron. Tada, hello world. Okay, so let's go back. And this is if, you know, if you think, wow, you know, it's so awesome. I'm a JavaScript developer and I can actually create an app. So this is packaging value. This NPM module which helps you with packaging. And if you run this command, it basically generates a map application. That's the platform directory. You can actually target it for windows and just put platform into the window. And it will generate like the binary switch. You can zip it up, send it to a friend to run. So yeah, that's all. Hello world in less than four steps. So I think what I've... Electron is really fun as... So I did actually try Node WebKit in the past and basically using HTML and JavaScript, you could actually create... You could have an app which you can package for somebody to download and run it. And as I continue to play a little, you feel comfortable because you can actually use whatever the way you write Node code or the way that you develop on the front end and you use Chrome to view a web page. So basically, as a JavaScript developer, you now have an avenue where you can actually write native apps. So actually now, come to why I actually wrote this app. For example, I wanted to upgrade to... So an app store update and you say, you do not have enough space. So this is what I usually get because this is a MacBook Air and it's running SSD and it's really easy to view out your space really fast. So of course, the thing to do is I figure out what's sticking out on my disk space and it turns out that there's... One way you can do that is to run the Vue command and it's kind of an interesting question. Like, oh, okay, Vue. It's like any decent developer who knows but it turns out, wow, there's actually so many answers. And different permutations where you can Vue and you can grab it, you can sort it. And yeah, it's quite entertaining to read that track. But for me, I think it's like... I want to know where my space go to but every time I want to run a Vue command, what was that switch? So I have to go and google that and then I'll go back to this stack exchange trend. So basically this app is like scratching my edge. To actually have an app to visualize my disk space. And so when I was playing around with Electron I said, well, there's a really cool visualization library called D3. If you go to this gallery, you can see, wow, there's a lot of beautiful charts. And one of them, if you see over here, it's called the Sunburst visualization and over here you can see the dream visualization. And it turns out that these visualizations were actually useful for this space visualization. D3 being a JavaScript library has been used to create lots of interactive visualization on the web. So how do you use D3? Of course you can download it. In this case, I am a PNM historian. And one way to get started is that my... Who's that? My creator of D3 actually has this back page called Blocks where he actually writes templates on examples where it becomes a way that you can actually learn through the examples. So basically I just went to his example of Sunburst visualization and you'll be able to see the code. And as I mentioned that Sunburst was actually this research project and it looks like from really old computers but anyway, this is the summary of that that they asked users to use Sunburst and Cremap visualization and in general the participants preferred the Sunburst too which looks like flares coming up from the sun. So did I miss something? So imagine that I actually combine them together and so actually D3 is pretty awesome. I think it can be another top. How you can use that for transitions, how it handles data binding in a really cool way. So the other library which I will use is 3js but this time it's D3 so both of them are really cool. So the next thing is that... So I thought, okay, I have a graph component and now I just need to get the data from my piece like how much space is in what my file is taking, how much space and I could actually write... I was taking a break in Taiwan, I was probably on a train and I think I did not have internet and I just kind of wrote just 30 lines of code and you could actually traverse your file system and calculate the total file size with this. So this is my first take on finding out where this space is. So basically it says explore a directory and you use lstat which is you find what is the data... I mean the file system information on the file which could be a file or it could be a directory. So you have if a stat is file or it's directory and if it's a directory you do a read directory thing and you can traverse that and that becomes a re-question. So in this case I just use sync because it's convenient. And yeah, and there you go. I could start to... That is like this visualization app that in less than not a lot of code and you could actually write this visualization app. So of course years ago I remember in Windows 95 there's this free web app called Scanner 2. So I think that was when I had the inception that actually wow this is actually a really cool visualization tool. So I think just using this it's like you could create an app. So of course it turns out that if you want it to be an app you can put an app store, you need to do more work, you need to think of interfaces. And this is like just throwing the tree map in. So it's not happy ending yet. So it turns out that this scanning can be tough. The next challenge that I made is how do I scan my entire... When I tried this prototype, wrote a code in a really short time I scanned it with a couple of directories and it worked well. But the question is how do I scan my entire drive? So I tried a different approach. So first like just now you see the synchronous method and then I got back to the internet and said there must be somebody who writes some note module out there and sure there's a note du. And there's somebody familiar that I know who has contributed to the project. And it's great to actually use note modules but in this case it does not fit on my needs because this returns me like the total results and not the breakdown of the individual sub directories. And then I tried to use QSyncQ to give a bit of... So let me backtrack a bit to say that why are we not using synchronous method? So I mean if you're a note user you know that synchronous is basically blocked the JavaScript track until after the apportion is completed. So when you run it in... even when you run it in Electron that principle applies. Basically when you're running a synchronous job basically your UI will be frozen so that's why you don't want to use synchronous calls. So I thought actually note du was using this library called async so I thought okay I'll give async a try and one of the things that I did was to wrap my... the recursive file like scanning into the async queue. So this is basically how you would use async. So the queue will call back a function and all the tasks are completed and you basically wrap the previous explore function and you push the job that you want to be done next. And so basically that can be done in-secretly and one of the things that asyncQ does is that you can have a limit on the number of parallel jobs. So I set it to 10 instead and then I set it to different numbers just to explore. And that's how that... it solved the problem of not locking my UI but there was a problem. It felt like it was taking a really really long time to scan my piece. When I scanned my drive it took 40 minutes. I decided okay let me just let it run and see how long it takes. And I decided after that let me just throw away this library. I was just using asynchronous call. And so that was what I did. I changed the code to just basically just use without any modules. Just read directory L step and it gave me like a speed up of 10x so instead of 40 minutes it's 4 minutes. And I compared that with the speed of DU and DU was giving me around 4 minutes for my entire hard disk. So it means actually wow actually nope it's actually really fast. And only after that then I thought about it. You know it could just be the async queue implementation. And I think some people recommend Bluebird if you want it to be performant but I have not dive into that. So yes, it turns out that this is a common problem that there's a crack on the electron. And I think at the start where people were complaining even an active editor seems to be locked up when you are scanning files and VS code actually does a better job. So some commenters say oh actually VS code actually spin off another process and run that in that process. And so I got around my journey to figure out how I can best do that. And so this is surprisingly there you know this isn't really well documented I think this chart was actually done by someone else but you see the orange it's called a browser process. Strangely it's not the browser window it basically means the electron the first place where you run launch electron that's the browser process and that's a parent basically runs main.js and when you create a new browser window you call that the renderer process and the renderer process basically looks HTML looks JavaScript and those JavaScript has no integration so this is basically how electron works and this is and actually electron when you start to do you know beyond that hollow world you may need to communicate you can actually create new browser windows and you can actually communicate among them but the how it gets communicated is that the renderer process and the browser process can do IPC calls so you could actually send messages the renderer process and the renderer process can send messages by the browser process but when you want to send a renderer process another renderer process then you might have to do some code to hop around and so if you look at the APR there's this thing called remote so remote basically encapsulates some of those IPC calls so that you are basically saying that instead of you sending a message down and handling the code in main.js you can actually from a browser process a remote which is basically main.js and you can receive a message and pass it to another renderer process I hope it's not confusing but you could actually read up on this so what I did was basically that similarly that I basically said okay let me try running the DO operation which is scanning out all of these the IO intensive operation by spinning off a different process that would run and it could be 4 minutes long it could be 10 minutes long and so one of the things that I use IPC is that while it's scanning I'm going to send my progress updates through IPC calls back to the first renderer process so your UI can actually say it's still scanning you could actually preview stuff without waiting for the entire job to complete so a first one easy way that I did not have to do that just to say okay just please wait but I just kind of panned out this path of thinking that while it's doing something let me preview something in the main renderer process and so what I did so what happens is that I generate a JSON which basically represents your file structure, your directory structure so basically file size and if it's a directory then it will be directory with children and so on and the next root block that I found is that JSON is getting larger and larger and so when I'm trying to pass the object through IPC it's basically saying your message is too large like don't do that that doesn't work so I tried like many different so bad build is basically a way to have a different electron process but that's like running in an eye frame in your current browser a fault node process means that you can actually fault an electron process but treat it as a node process and the way which I end up doing was to fault it and make that electron window invisible and how I transferred the data I discovered that I could use local storage since they are both chromium and local storage has this property where you update the key all other browsers on the same domain would be able to get a changed event and I used that hack in some other places and I reused that and it turns out to work pretty well but local storage also have a limitation which you know size limitation this was a little IPC using UDS project that I try out turns out you can send like 100,000 messages a second so I tried out of them and yeah and yeah a lot more stuff and I think the final decision was that I would use local storage when something is less than 10 megabytes and when it's above 10 megabytes when it hits the storage limit since the nice thing about electron is that it has like node permissions and you can write you know a message to the file system so local storage will send a message over asking the other renderer process to read it and it turns out that it's really efficient right I did a lot of like and I could scan 170 gigs of my space and yeah and so this was like my first one of my first release so basically you can see that I think there was a bit of latency I don't remember why but when you are JSON passing a big JSON file it's but more or less it was very reactive and fast like 10x and almost faster than BU so some other things that I learned that JavaScript objects are expensive so some other things that you have to take note is that electron process has to be under 1.5 gigs of memory otherwise it will crash so I think that's default in I'm not sure what defaults are there but I think if node that could be node default and objects itself has a memory card so when I'm serializing over my whole the whole representation of my disk it was 150 max in JSON but when you just have to JSON pass that in any node process and you will find that well it's like 400 max in your heat memory or you know like almost 800 max in your RSS so when I'm doing that I have to make sure that I have to be careful like my charts is displaying data I have to de-locate them so that it gets garbage collection and then render the new graph that 120 max of JSON has actually 2 million objects so actually there's actually so many objects in your file system and if you try inserting 2 million keys in the JavaScript objects you will start to face the challenge and if you are doing node add skill then you will face this problem like one of the library which my colleague has written is Judy FFI Judy Structures is an efficient way of storing max of data so 2 million keys I think we tested that and it doesn't use any much heat memory it's not really V8 heat but it's actually using node's heat which is almost like the way you do things in C so other approaches which I tried was using flat buffers this actually serialization one of the hotter topics like flat buffers, captain proto, simple binary encoding these are serialization methods which they claim to be really fast because of zero copy like in C you can actually read from the memory directly without having to deserialize that so I thought could I try that for JavaScript and it turns out that captain proto couple of JavaScript implementations of that but it didn't turn out to be really usable for me or you know it was a bit complicated and it turns out flat buffers has a JavaScript implementation which was just which was much, although there's no documentation but you should try that it's really cool so when I use flat buffers it takes up like this 100 meg of heat memory I think I'll get to that in the next slide yeah so yeah I say why I say JavaScript objects are fast because when I use flat buffers although it's 0 seconds deserialization but for me to actually read to read that data structure there's certain cost involved that takes me 5 seconds while the JavaScript object maybe it takes 1 second to JSON pass it but you know it's like almost less than 1 second to actually traverse the entire node so come to another point which is electron ads surprisingly there are certain things that you have to take note of doing cross-platform I think there's no magic bullet but it turns out that when I tried to run that on Windows which I have not booted up for a long time but I have a PC at home and when I decide to try that it actually works so it's kind of pretty impressive so the next thing which after I released the first version there were requests that they wanted to trimack too and one thing that is challenging is that D3 uses SVG and SVG by itself is not the problem but when you're running thousands of objects in the DOM and the browser will have a reflow and that will make it really really slow so I end up having to implement it in cameras and emulating the DOM and so actually some people mentioned after I did that it looks like this app called DayZDIS and it turns out to be a 990 app it's really cool it's like a company doing that but for anybody to I mean it's a compliment in a way that there's something that I did in just a few days could be pretty powerful okay I'll skip all of this so I think this is a really nice book called Generative Design and I think these are more challenges these are open source projects so if anybody is interested in my free time I will work on that so one of the things that I've learned is useful this is one of the latest edition of my app and I think it will be more fun to actually review it if we have any more time alright and let me scan let's say my Dropbox and it's actually done already it's really fast and you can go to the directory and say what is this framework and I have a lot of pdm which I downloaded and you can actually go up this directory so one of the stuff is that the integration of Shell which is a right click and you see this pop-up window that's native things like you can actually locate directly in the file system right click on it and delete at least I'm going to delete now and one of the realization when I was doing this is that I could actually apply this not only to this space but to memory so this is my memory and I can see that my app memory was taking out most of that Google Chrome and I actually try with my colleagues it's like everybody's memory usage is like being owned by Google Chrome all right and there's a three map representation of that so I think the point is that people say how this has been done before but I think by me doing this I learned a lot of lessons that I could apply that to the memory which I don't see any apps out there doing that and I don't think there's any apps which you can download right now which have three maps and a subversive visualization so it's nice to have your own so these are friends so they just send me a screenshot oh wow it's in my dashboard that icon was designed by one of my colleagues so I added a code my postdoc who was again one of the main guys behind E3DS he says visualization leverages the human visual system to augment human intellect and I think that's actually a reference based on the code and I can't remember the name but one of the forefathers of like the internet there's a link to that if you're interested to take a look at it so I found that actually when I use this app I could actually find a lot of stuff which is sticking out of my space if you're using Google Chrome Canary you will find that it downloads every version of every time it updates and that takes out a lot of space if you have X code it downloads every new OS update and over time it takes out like really a lot of space and that could actually be nicely sorted with visualization so that's me, I'm Joshua if nobody knows me I work for a check company called Zofim and I found this so this is a little bonus I found that if you rename a Github repository they will create links so click on any of the links it will go to my Github repository and yeah, don't want me to upload this space thank you one question that's from me I don't know but real path, did you take that into account because I remember I was traversing some stuff and it was taking forever but it turned out that it was just going like it was following Simlinks Simlinks and Hardlinks no, only Simlinks sorry and that's what made it slow did you resolve your real path yeah, that was basically a contribution to that to the Node.DU so basically there's STAT so LSTAT will be able to if it's a symbolic link it is it is not a directory, it's a file so you do not, because you do not traverse into that did you say that was my contribution to Node.DU? yeah, yeah let's take a short break let's take a short break