 Okay So Oh, I will jargon. Thank you Hello, I'm sorry. How do I pronounce your name? Oh, you guys Good good, so I have a question. How do I pronounce your name? Miguel Miguel that's okay. Okay. Thank you. Actually I worked with Prima before Yeah, he told me he told me He was the he was the teacher at Generous and oh you want to general assembly? No, I I wasn't a student. I was a TA for like Yes, I knew him and then the Yeah, then when I saw like you putting his name beside it, yeah, oh Yeah, I got some talent from What do you are you a teacher instructor at the wagon I teach sometimes when I'm the product manager I see and Prima was gonna come today as well to Tag along with me during the presentation, but he had something urgent and he cannot make it so I'm alone unfortunately I see oh, I'm curious The wagon is a teaching school, right? Yeah, then like what do you do as a product manager? So I there's a number of things that I need to do like the role of a product manager for us It's a little bit broader, but it's going to events building partnerships Trying to make sure that there's features on the roadmap that are reflected on on the platform that we use internally Mm-hmm. Also, there's this strong product market inside That kind of stuff. I see I see also you take care of partnerships. Oh, so well interesting and You you are a developer yourself. I Wouldn't call myself like a full-time super strong developer I don't know how to get around I I did I did some work freelancing for about two years and a half The but I wouldn't say like I'm a senior developer super good like I I know some stuff And there's way more than I don't know that I know Cool And the wagon is it's in Singapore, right? Yeah, it's cool. Is it only in Singapore? We're in Asia. We're in Singapore and in Bali Mm-hmm I see How how are the students doing? Super good super good and here in Singapore we started like two three batches ago And this last batch was super strong people like really good people were very strong at coding and also very passionate about it Most of them have found roles already Yeah, so it's that's very worth Yeah Interesting. I Will assume that most of your students will also be Me career change kind of people Kind of yeah, like we have Three kinds of people like we have people who are entrepreneurs and they don't want a career change What they want is to build their own thing And either as their own small project or they want to they want to build it without developers We have career changes and we also have lots of young grads who want to go into tech Okay, okay. Yeah, but back back when I was helping working with Fremont and GA Most of the people are like yeah, you know that you reminded me there. There are some of the winners but most of them then was Was me career change that people who want to go into tech Yeah, it's super cool seeing all these people It's nice working to it is an entirely different industry for them. So I found it super cool. Yeah And you are you working as a developer right now or what's your full-time occupation? Yeah, I'm a developer. I work. I'm working at Shopify. Ah, nice, man You know, I it's I hope it's it is it good as it looks from the outside I Listened to interviews to the CEO to big people in Shopify and just the culture the way they talk about their product The way they talk about how they want to understand technology and how they want to build platforms. It's just fascinating And it was yeah, really that good insight It is it isn't from my personal experience. It is I they really live up to what they preach that's awesome, so sometimes a bit unreal like to The extent of things like they do and they really living up to what they say But it was it's super cool. Like even I've been there for seven months, but It's just that Sometimes you really would not expect like how how far they will take things Like what did you have an example in mind? Ah Like like recently we we went full remote for example, like we have like about six I think six thousand people all over the world and then they just suddenly okay, everybody's gonna go remote because They told me he believes that Going room like working remotely is the the way What is the the way things will be in 2030? And then he just decided, okay, we're gonna be like be bring it forward and then now everybody's like all of a sudden Everybody's remote now but they but it's it really lives up to how How to be believes like the company should always be forward-looking should be always ahead of the curve and all this stuff So it was a very big decision to like everybody inside But son the brother with the help of the HR team because talent here and then so they gave us a lot of support to help us type through this thing Which was like very impressive or so, but this one of the reason like how far they really go to What they believe? Wow, that's amazing. That's really cool That's really cool By the way, I really like your the trick you pulled to the last meetup with the fiber To recreate the enumerator. That was really cool. Oh, thank you Me took me a while to twist my brain into that Solution I thought it was ingenious. I was like, wow, I would have never thought of fiber, but it makes sense. It's really good Let me let me check Usually people on this plate. I'm not sure what's What's the reason this time around Oh, am I not emitting people? Okay, let's Let's start at 740 and we'll give them some time Yeah Yes, yes Um, yeah, uh, no cause I got this Okay, let me explain the thoughts yes Set up also need to zoom record. So then I Use the other computer to OBS. I use this one. So I need to separate some account so I have a CSS account and a personal account and Each computer happened to be the CSS account. Oh, oh wait wrong. That's that's not that's that's not my face but, you know but now is Is your top CSS still running now? It's Monday. So we are first Wednesday Okay Yeah, I find this like first. I Always the Ruby one to be the second Tuesday of every month Yeah, it's really easier. Yeah, I found that like a reference like that and Easier for people to remember Also easier to coordinate our people like the JS JS is middle. So they are like I think Third ah, they are always the middle of the month. So that we are like we won't clash one Yeah, we'll never have a crash We got clashed before last time when we started out and we never like say say properly then Clash already like a clash at how much we have we have a few much runs Okay, la much la cause we are like Say what Okay, la share audience True true our overlap is ah Yeah, I'll overlap quite quite overlapped. I've been diagram quite close. So it's still like quite quite okay Yeah, even the design of your local is about the same Ayala cost In in the official the official story is we spin off on them. We are like we are their sister We are their sister because they run very long really then oh AWS tonight, I don't wonder Okay, no, like I always had to make fun of the big meet-up. So yeah And then the people who shot like all for the three of you who showed up instead of going to the for example Instead of going to the fit my meet-up you came to my meet-up. Thanks. Thank you three of you But crash crash It's actually okay if we clash we were not like I don't mind crashing with a has come it up because the has come even I'm gonna come to me. So yeah, hi. Hi. Hi. You're not good. You're not from the house girl group, but it's fine Yeah, it's okay now people will choose what where they want to go. Yeah true I'm in and then a ruby with that since 2016 Oh It's okay, I'm a ruby there, but I don't I only attended like two ruby meet-ups before I start About the start of this year, I think in February something like that. Oh, sorry sir. No, okay. Yeah And before that I actually don't don't go for Ruby meet-up I always find it very tiring to go So But then but then the ruby meet-up like died. Yeah, like why are so sad like where are the ruby and genius Okay, I do but then now I think it's turning out well if I do the online meet-up then I won't be so tired I think online makes it easier for people to The people who will say very very tired then don't go then it's it's I think it's lower barrier to entry or so lah, I guess yeah I'm just gonna stick to online meet-up. Yeah, I was so not so tired Okay, I think let's let's start although we don't have a lot of people today Video means go meet-up now. It's a king man. Tell me one I Know it's recorded so all these thoughts will go go up to engineers SG And and I think people do do go then Watch this video son. Yes. Yes 15k subscribers Go on your score. Let me Let me try and share. Oh, no, I Need to share screen You press the green button Yeah, you also record it. You are you are doing the recording yourself in the aid Actually, I'm not Michael he started the recording and then he Made me a co-host you say that something else the record But then now that I'm the co-host everything is so complicated Oh, um, so basically share screen is the same if he's the one controlling the recording Then you don't have to worry because he's already recording. Then you just run it as normal. Am I? Okay, yeah, let me see whether this is the correct screen. Oh Hey, I don't know what I cannot share. He says that Open security privacy You want access? Oh, yeah, you you you you you you get out of zoom first you go to your system preferences and you allow You use a cat on it is a Mac thing. You allow the preferences. Then you come back in Okay, they updated the versions everything to mess with with soon. Yeah, I've done this I don't see so many times. I'm like, oh, yeah minute. Oh minute. Yeah. Yeah Oh Can I assign you Um, well, yes, he's permissioning himself. Hi Nice to meet you. I guess Actually the organizer of the CSS Nice, and I recently became a Constance colleague a month and a half ago Okay, so he was working at a shop if I then yes, so I know Ruby or rails experience So I have to crash course But it's been good cause console has been working on the same project as me and so basically he's just answering all my No questions. I'm like, how to render a template. What is it? What is the content for how do I put How do I yeah, yeah, basically how do I look at it? Oh, hi. He's back. Okay That's that's life. You gotta you gotta start somewhere Yeah, I got you if you'd get me outside of the basics of CSS. I also need to ask a lot of questions Okay, no I'm biased Can you give me the access to share my screen post-disabled attendee screen sharing? Okay, hang on Let's see how do I pass These back to you Security How did you make me a hose ah, but I leave then they asked me to choose something Okay, in that case, I'm just gonna oh, yeah, I can't really I just made the horse. Okay Okay Well, this is like You still show me the same thing Huh, I'm sorry guys Wait, I mean, I'm sorry. Oh Hose disabled at the need. No, this one Michael needs to do something. This happened to me for CSL. I also cannot share Okay The host though one more time. I pass it back to you one more time. Okay Wait, if I'm the horse Hey, I can share the screen though. You can't you can't yeah Okay, okay, wait, uh, if I'm the horse I can Security I can you have share screen under you allow participants to share screen Where's he? Is he's gone? Oh man, come back because because I've done it on now So if I give it back to him He should be able to Yeah The perils of zoom. Okay to be fair to be fair We are not experienced in zoom because uh within our team we use google hangouts. Just say which works super well by the way Yeah, yeah It has this nice feature Open it up. You can I open it up. You can share screen now. Then I give it back to you. Sorry Why is it still like that? You still don't have No, it still keeps asking me to open system preference security privacy to go on like that Uh, okay, wait, let me check which which permission it was for me. Uh, security privacy Security Privacy screen recording is it? Yeah, screen record. Yeah, you should give screen recording But I already take everything and let me see if I anything else Release access Oh, I'm sorry guys Uh No microphone got camera got. Oh, is it accessibility? Why? I unchecked but you can try I have the option to share my screen actually No, he is zoom He is not a zoom problem. He is an apple problem Let me see Yeah, can you see my screen? Oh, I can I can so weird I don't know. Okay Uh There is an alternative. What do you need yours for? I just need to show the slides actually. Oh, wait. I have alternative You can send us the link and then we should log over. Yeah, okay. Yeah, I actually have an alternative I'm using some broadcasting system to show my Camera feed so I can Add my browser here It's very capture. Oh, Winston is here. Hey Never he didn't miss much Uh, yes, we can see your telegram. So you probably don't want that to be on screen. Okay Okay Uh, what's going on? Uh Council I cannot share screen. Council's Mac doesn't let him share screen. It's not a zoom thing. It's a Mac thing Oh, uh, I think you you probably need to get out of He allowed permission to redo all just now. He already quit and come back. Really? Yeah Okay, wait, uh, just a bit more That's a bit more That's a bit more. I'm sorry. Let me change to get pinned in his image and pinned video We're almost there. We're almost there. Okay. Got it. Got it. Yes Okay, I cannot see myself now. Okay, let's go with this Okay, uh Yeah, so today, I think you'll be a very uh, this is the schedule for today. We need to talk. So it's uh, it's as usual the schedule sponsored shout out and all the rest wrapping around the two talks So the first one is miguel miguel and prima Talking about scraping with food traffic So Okay before that So, uh, then again, I want to thank michael for hosting our ruby Meetup session and recording Again, like for the he has been helping us for the past Few meetup. So like thank you and then Uh, I don't know whether you all know yet. We have a ruby sg telegram channel So please join if you haven't Okay, and then uh, I will skip this one So today's talk, uh, prima is not here. Unfortunately. So we only have miguel Talking about Going through this presentation. So food traffic To he'll be showcasing the food traffic jam And it's a wrapper for ferrum designed to run website testing and script information on websites So this jam is uh, released in adler wagon. So they're here to get some feedback Uh, yeah, so Miguel do you want to give uh I pass over the time to you. Yeah, sure. I'm gonna go and share my screen. Is that okay? Yeah. Yeah. Yes. Yes Okay, so I'm gonna go to my desktop and If I click on pre sent can you still see it? Well Yes, yeah looks looks okay. Okay. So thank you very much for the presentation and for the introduction We're going to talk about it stripping with food traffic and another jam called vessel And the reason for that is because we see options for food traffic to be Connected in some way with vessels. So that's why we're going to showcase it so we can get the feedback from the community about us I'm Miguel. I'm the product manager slash business guy for the wagon Singapore with a coding school We run a coding boot camp in Singapore for the last year and a half And Prima who's one of our senior developers and league teachers Has prepared a lot of the work that you're going to see today But unfortunately because of an urgency he cannot be with me. So I hope I can represent him and to the best of my ability The talk about about the talk today. We're going to talk very quickly about what is scraping in case Anybody here doesn't really know what it is or it's just a term that they've never heard We're going to talk about a given overview of some of the scraping choices that you have in ruby And then we're going to talk more in depth about food traffic and vessel I'm going to do a quick life demo that I hope it takes most of the 30 minutes that I have And then I'm of course very happy to take any questions from you guys and now that we're smaller groups It's also more intimate. Is that okay any questions before I get started No, okay, let's go So scraping basically Transforming data that isn't structured from all over the web that it's presented in layouts and in browsers And you cannot really compute or manipulate or aggregate it any way You have an application layer where your web scraping software lives You can access through any sort of interface You can access the whole website get the information and make it structured and then you can either dump it Into a database you can dump it into an excel into json your csv or any format of your choice So this is basically what what scraping is about. Why should you care about it? And why is it important? It's extremely useful if you're data mining so if you are uh, let's say A recruitment agency and you're looking for profiles all over the web and all over linkedin Then you can use scraping to get a lot of data into your product quickly Growth hacking if you're looking for contacts if you're looking for people for customers and different listings And you can you just just want to automate the task of looking through emails looking for People that you can actually address also very important for competitive research or price monitoring So retailers supermarkets airlines are using scraping all the time to Look at what other competitors are pricing and that feeds into their own pricing algorithm So that's that's the power of scraping and there's big businesses or big companies built on only on the on the notion of scraping Some things to consider if you're scraping around and I didn't know some of this not too long ago So this you can actually be in trouble Scraping public data may be legal like depending on the country depending on the domain depending on the website Could be legal. So you need to inform yourself You almost never in almost any country you're never allowed to use data that you scrape for I'm claiming that as it as it's yours. So if you're using let's say blog posts or anything like that to Inject it into your own content that doesn't work If you break a website because you overloaded through scraping you can be prosecuted and you can get up to 10 years of jail in some countries And these regulations are more great than black and white. It depends a lot on the use case. So be informed Just just stay safe. If you're scraping around Make sure that there's no API before scraping if you can use an API instead of scraping, of course, it's a better option Make sure that you look at robots txt on the website and that gives you guidance on what is scrapeable What not at what times what what do they prefer? What do they not prefer how to identify yourself as somebody who's scraping and they can realize that you're not harmful agents Make sure that when you're crawling through a website you set a conservative pro rate So you can ping in the website every one two seconds and you're just not overloading their servers Declare your intentions again and the user agent part. So use they know what you're up to and who you are And if you're thinking of a really massive if crawling practice a crawling project It's maybe sound to get a lawyer involved and and there's still this complexity itself What is legal? What is illegal? So you're you're always safe. No matter what you do So that on scraping What can well do we have available to us in really to scrape them? We have a number of things I'm more known some less known one of the most known is of course no goviri Which is kind of your your starter jamping to into scraping and you just basically get the html and you can start playing with it And a very popular one is of course lino not only for scraping but also for integration testing automation tasks And you also have mechanized which is kind of no goviri on asteroids. They didn't have a logo but it's also Maintained by aron patterson tenderloaf. So I felt like putting that picture over there to represent him was the best thing I could do for him and for us And we also have ferrum which is a relatively new gem created by Studio that it's on the tip of my tongue right now, but I'll show you guys later It's a it's a famous ruby agency doesn't come right now into my mind damn it But basically why I'm showing why I'm showcasing two gems that are built on ferrum today both Food traffic and vessel. They're both built on on top of ferrum. It's because it's really fast Uh, it's uh when when you benchmark it against selenium it really wins. So it's it's really really good And it has a wider api surface if you use it for chrome So since selenium is compatible with Almost every browser that it's around there There are api surface and the things that the weights that you can interact with the browser are more reviews compared to what you can do with ferrum since it only uses chromium and It the weights builds it reminds you a lot of the way pappeteer And works if you're familiar with pappeteer and javascript, but of course since it's ruby the syntax is a little bit simpler And you can also perform js evaluation on it directly So if if you have a jscript that you want to see or you have a number of javascript Be a set of javascript behaviors that you want to evaluate against the browser You can do them ferrum and that's pretty cool And it's multi-thread and thread safety So you can basically run lots of different tabs in parallel And if you are running something with web sockets, for instance, and you have different users interacting with the database You can have it you can test it on different threads and see how that reflects in the way you're using it And it works really nicely. So those are kind of the advantages of ferrum And full traffic, which is the jam one of the gems that we showcasing today. It's a dsl for ferrum Like it's it's just a wrapper that allows you to structure sessions more effectively So you can see an example over there Where you could just basically just require for traffic you get a session And then you can start creating new new tabs and start playing with that behavior or also the There's a little bit of a syntactical sugar that you can use just to go to websites or use or navigate around And interact with the website. That's something that we'll see later on the demo And if you want to scrape Seriously, and you just don't want to get like just some data from a website You just want to do a serious scraping operation Then you have vessel which is also built on on top of ferrum and it works really well This is kind of the basic way it works. You need a class. You need to set a domain a number of start URLs It's also thread safe. So you can have several URLs at the same time to start with And then you need a parse method Where you basically declare what is the selector that you want to get out of the website And then you can get that data out and throw it somewhere else And then you just basically run it with with the run method and a block that will be consumed by the parse method later on So so that on the gems Um about food traffic The way it's built right now. It's it's open source. We we haven't It's a it's in a very early stage. So the possibilities are big But for now, it's not meant for pure automated testing It's not supposed to work along our spec and it's not supposed to be taking care of that side of your application If you're interested on using ferrum for that As well, there's another gem called cuprite that you can play that plays along very nicely with kb bearer for that food traffic is mostly designed for low like pressure testing your application And seeing how it works when when you you want to see how behavior plays around life if you're using a framework like Stimulus reflex for using websites and you want to see how a use case Actually plays out when a human is playing with it in the closer way In the closest way to it with as a human would be interacting with the website then Food traffic is a very good option for that, but it's not Right yet for automated testing. You can't run small scraping scenarios, but it's still not optimized for that yet. That's Something that we would like to add a wrapper to vessel to in the future. That's why we want your feedback So if you see that there's any potential you're curious about the gem and you want to try it out You want to raise a pr? Please be our guest. We were very keen to improve it And um, these are some of the links I can throw them later on the chat if you if you're interested in them I'm very happy to copy paste in so you can just go directly to all the gems that I've been talking about um Any any questions before we go into the live code anything on the chat I'll really all right, then let's get started Let me just get out of here here this thing So and let me go into sublime All the stuff that I have around here So this is food traffic first and we're just going to simulate how to scrape this site Which is the site meant for scraping it's going to go to it for a second and basically they have a sandbox of examples where you can just play around with Scraping without risking yourself that you in a scraping website that you're not supposed to scrape So it's it's very good as a as a sandbox sandbox for for beginners We're going to be looking at how to use for traffic We we're going to be testing two scenarios how to run a login So something like that. I don't I don't see it over here right now Sandbox scrape this side sandbox This one on the hockey teams I'm going to be scraping this table over here and There's another one that tests if you are a user that needs authentication, but doesn't quite Show right now. Let me I have the url over here So let me just bring it over So this is just simulates that you don't need to log in and you need to put any Any username and any password in order in order to go forward, right? So this is the two scenarios that we're going to try out with food traffic um You in the one you start with food traffic and and you have the gem install you get access to session And start and this method takes a hash of options that allows you to set some stuff to set to stay safe So the process timeout is you want to make sure that all the resources are loaded and before taking any action So I have it at 10 seconds max if otherwise it keeps moving a timeout if nothing happens at all You have a slow-mo if you want to simulate actions with a little bit more time And at what resolution you want to display the window and there's a number of options that I will show some others some other Attributes later on But basically I get a window object over here. So I can just say window tab thread and I can Open a block and that gives me access to a tab object and over here basically I can just call the tab and go to Go to an address And that is going to basically just summon the website after that I can just go into the tab and use method like CSS to get all the elements that respond to a particular selector or add css to get the first one I can do the same with add x path Or x path if I just want something that it's a little bit more specific or in in case the The structure of the website. It's made a little bit more up They've obviously obfuscated the the css and the classes. So, um, let's try that out before we go here Let's just run the script run And then I can go to give it a start That gives me the login right so Over here I can just basically explore what are the attributes of the things so I have an input for type text and I have the password so I can go at that css with the name user that you see over there Back into the code and then once I have selected that I can type And I can chain actions and type so I could say something like my name I could put comma after that. I can just click on a particular action that I want on a key something like tap or enter So I can just say tap to move to the password I can put pass. I think it responds to basically anything and after that I can click enter So if now I rerun my script hopefully it selects it and There's something that went wrong. Of course life coding nothing's going to go as you wanted it to go Did I save the file? I went Going to the file tap threat Okay, I think I'm missing. I know what I'm missing I need to focus on the element and then I'm ready to type. Sorry about that So now you can see and there's a little bit of time passing between the actions And then I successfully logged in. So it's nice if you want to Play with with your application and see how it actually responds to you And the good thing is that you can also open different threads and different tabs at the same time so I can do another tap threat and do something different And for instance, maybe I want to go to the other place where I I have the forms the The teams that I was showcasing For the hockey teams and let's just run it So you can see like the two tabs are opening at the same time And this is running. Well, this is working so basically what I want over here is to look for a team And get maybe all the wins for a given team across the year So if I want the Boston Bruins, I could look for Boston And I get all the Boston Bruins and I can get all the data. Maybe I just want the wins So I want to simulate that behavior with traffic To do that, I basically need to target this first. So now I have name q can go back into the code and Over here. I have the same tab And at the css with that identifier, I can focus and I can type just Boston and then enter I can then just for safety sleep a bit to make sure that If there's any JavaScript that activated in the moment of that search It doesn't interfere in the next action because maybe this is almost just a little bit too late to not not enough and then I could just Select the table That I have over there And then I could iterate from that table and basically just run a bunch of csv To get it into this file that I have over here. So basically we delete that I think my blocks are working well But as you can see, um, here's where you can see that food traffic is not Perfect for scraping just yet because I need to start running rules. I I don't have a lot of methods I don't have embedded classes that I can build on but I can still do the work somehow So basically over here what I'm doing. I'm I'm appending to a csv. I'm Accessing the the rows within that table that I got over here, which is basically um this table On the hockey teams I want to get all these rows and I'm just getting the I'm just getting this name and I'm getting their wins And I'm doing that with adex bath in this case when I'm going through the rows instead of with that at cse And then on the csv. I'm just throwing the team names and the wins So if now I close And I run it fingers crossed it works Goes to the website. You can see that that behavior is going over there while that is working Uh, hopefully the scraping happened can go to data csv and No, it didn't look unfortunately Try again. I have this issue before I wonder if it's a matter of this So let's run it once again. Otherwise we move to the next case There we go. Now I got it. So it was just sleeping and now I just successfully parsed the data. So so that is How a little bit of how food traffic works, um, you can go into the documentation if you're interested This is another good thing that it it it leaves your browser open So you can keep playing with stuff once you're done running the test case or the scenario And it has some really nice documentation on how to how to play with it. So so that's food traffic I'm going back to This light set. I just wanted to show as well vessel Which is basically An a different a different way of working. It's a proper gem meant to scrape websites And the way it works We have the case to over here You need to start the class that you can call scraper or Spider or banana anything you want, but it needs to inherit from vessel and cargo You need a domain you need instead of start URLs And you need a parse method So we're going to be looking at Um delivery room And we're going to parse we're going to scrape some information from the restaurants By the way, if you check the robots txt, it's absolutely clean Like you you are allowed to scrape delivery at any time. They have no restrictions So basically what I need is deliver on the domain And The start URL that I'm going to get is this one I'm going to get restaurants from singapore and I'm going to pass two arguments on the terminal later on So I can just customize which cd I want and was possible code I want Just so you get the idea of what I'm going to scrape It's over here That is exactly the The url that I'm that I'm looking at I'm going to play with this parameter is coast That could be no we now could be anything else and the postal code in singapore And I'm going to try to get the information from this card You can see over here that it's it's really obfuscated and this is something that maybe firms not going to be great at But over here, I I can play with the expats until I get what I really want Basically what I'm trying to get is this Like I have some information on the restaurant and it's in an unaryl label for accessibility But this is the info that I want so Basically, I'm going to get that start url Which I already copy paste it. Yes I'm going to start the parse method And in this parse method the first thing that I needed to it's a selector I get the same thing as Since if this is powered by firm and so you can use x path or you can use css for select I'm just going to use x path And this is the thing that I need I know that I need this because whenever I go to this element in particular And I copy the the full x path. This is exactly what what it leads me So the developer tools kind of helped me out in getting a really complex query done And then for each of these if I did add x path It will give me just one, but I want all of them. So I'm going through all of them And On each of these I'm going to get The name distance rating of the category by manipulating a little bit of the information that I get under that array label and splitting it And then I'm just going to call a yield block I'm sorry. I'm just going to pass a yield That will be returning the name the rating and the distance And in a in a hash for me So this is all I need to start scraping Now to I need to start the I need to store my information somewhere so I can just say any choices This could be an empty array And then I can run my scraper And then scraper.run takes a block that needs to be consumed by parsley to run So over here. I just say resto. Just what I'm using And within the choices I throw the resto and Maybe I can just display them on the terminal So with this If everything works out I can now do ruby case to start and I can pass east coast and I think this is my postal code and hopefully this works Nice. So I just get the information over here for all the restaurants that I'm passing So that on on vessel Just so you guys know Both support kind of good. They have good features for stress testing So here on the options that you can also pass on on vessel I can Use stuff like cloning So I could just simulate how 100 browser ads would be Doing this at the same time to see how my my application would work under stress And I could also Just run it on headless Which are also good good things to know So so yeah, um, I hope you enjoyed it I hope you got curious And we're very excited to know about what you know if you want to raise a pr If you want to be if you want to learn more about the food traffic and and ferrum and the scraping ecosystem We're definitely very excited about it. Now. We're using food traffic and levagon products to run Some of the simulations not as I said no integration testing But we need to see how some of the javascript and the sprinkles of javascript that they have We have on the front and are working So yeah, I'm really looking forward to having your feedback Um, just one thing more just to close before saying thank you if you're hiring ruby or ruby on rails developers, we are Creating new ones every two three months. So please feel free to contact us And if you enjoy teaching or you enjoy mentoring or you enjoy getting involved with people who are giving their first steps in programming People like this some of them are like if in two of them are singaporean And they transform their careers with us now they're developing and they're they're working as rails developers in in singapore Please please get in touch with me. Um, you can partner with us in hiring events and job posting You can bring you our students to look up some projects. You can be a teacher for us full time part time Do workshops um be a speaker whatever you want That's my contact and thank you very much. Thank you very much for listening Any any questions? Uh, I have a question vessel and food traffic are both apis for ferrum Sorry, is it something like that? They're all they're both built on top of ferrum. Yeah Uh, then I'm curious like, uh, what is the reason you guys build food traffic Well, we wanted a DSL that was a little bit more comfortable for for doing the threading So this particular behavior that we have over here It's uh, it's a little bit more cum cum cum convoluted when you're just using ferrum plainly And that's why we started doing food traffic Also, we need to do a lot of scraping work for uh for growth hacking purposes And we wanted to start using our own solution and and we really like vessel But there's some some stuff that we think food traffic could build on top And that's where we're going with this Thank you anybody else if any question? um, I I haven't done a lot of uh, web scraping just a little bit, but I definitely can tell that this is Easier to use than so they nailed Definitely The same text looks cleaner and it's more straightforward so Yeah, it's It's what I can see Well, I'm glad that at least that came across that it's it's usable Okay, anybody else has any feedback? I'm going to throw the links On the chat For those that are curious Okay, um Yeah, if nobody have any questions Anybody else have any feedback or questions? Uh, if not, then uh, thank you miguel for your sharing today Oh, thank you. Thank you guys for listening. Thank you. Appreciate it Okay, uh Then shall we continue? a Okay, so I'm going to continue to use my camera That's why I screen sharing pool Yeah, we don't mind and okay, cool Uh, let me know if you guys can all see my screen because this is not uh Not the norm for me. I'm using my camera to share Uh, okay, then the next stop is uh for me. Actually, this is not like super ruby But um, I'm just going to talk about json web tokens And there will be a small part of the tissue like how how we can use Uh, jwt is in ruby But the gist the main part of the talk is actually Uh, understanding this more of the conceptual level Of like what is jwt and what is it useful? Yeah, so Uh, let me get my slides Okay, I hope you guys can see Because I cannot see I cannot see anything related to json And why is this so slow? I can see it. You're lacking a bit. You're lacking a bit, but I can see it Okay, I Okay, everything is not going well today Okay, so I just got to start my talk So today again, I'm going to talk about jwt and then uh, just a disclaimer that this topic Uh, actually can get very deep But all the details and examples in this presentation are pretty basic should be good enough to get you started But I'm but I'm sure you will be able to go deeper into Uh, all the algorithms and all the stuff to to be your jwt implementation Okay, so the first thing is what are jwt's So json web tokens is actually an open standard With this number rsc 7519 and then the standard is actually for securely securely transmitting data between two parties using the json JavaScript object notation which is json so You can find the rsc the official rsc details in this website Yeah, and then you will see a very old school website detailing all the all the standards, okay So a trivial example of how a jwt can be used because it's supposed to be used to securely transmit data Right between two parties. So I have an example here is John john wants to tell roland something Right, and then john can create a jwt with some data And then he can pass to roland and roland can read the data and and also verify that the data is not tampered with so then The question is like how how can roland read and also verify the integrity of The the the data that was given to him by john. So this is the main main question today, uh, how to securely transmit data Okay, so Let's go through the structure of a jwt. So a jwt is actually like three has three parts separated by two dots So the first the part that is In front of the first dot is actually called the header then the middle part is called the payload and then the last part is called the signature Okay, so let's go through them one by one Firstly the header of the jwt, which is the first part before the dot interestingly actually this is just a base 64 url encoded data of the Recalling metadata of this token. It's like a http header like that. It's a metadata and this metadata It just outlines or it just tells you how to read the token. How do you read the How to read this jwt? so for an example if we do a base 64 uh url Decoding of the header, we will just get a json Algorithm is hs256. You just say There is the algorithm that is used to used to encode this token and then the type is jwt. So because this is jwt, so this this type will always be jwt I No idea why they specifically put this type here Since this standard is for jwt, but then this is like a standard okay, so It's anybody can if anybody get hold of this token They can just like split the token by the dot Get the first part and then they run a base 64 url decoding and then they will be able to understand how to read this token Okay So the second part is the payload, which is the middle part of the jwt which is actually also a Base 64 url encoded message Of uh that john wants to tell Roland this time round is literally the message that john wants to tell Roland So if you have any message that you want to send over via jwt, right? You should be putting inside here so So after url Base 64 url decoding this payload We will we we can see that what john wants to tell Roland actually So he has a subject it's like an email like that. So the subject is 1234567 890 then the name is his name is john though He just wants to tell Roland that his name is john though and It should add a timestamp. So it just tells john tells Roland that when did he send this message So the middle part this payload is also a base 64 url url encoded of the message And then the last part is actually the signature. So this is the most Important part of the token and this is not basically for url encoded so So this is this last part of the this signature of the jwt It's actually a hash generated using uh using Be the the base 64 encoded of the header plus a dot plus the base 64 encode of the payload And then sign with the with a secret Which only in this case because john is Creating the token right so he can use his secret to sign So in this very trivial example Uh, let's say john and Roland have a secret that they share so so John can sign with this secret and then Roland on the other hand can Uh can decrypt Oh, john can encrypt with this secret and then john can decrypt with the same secret because they share the secret right so this is how this is how Roland will be able to check the integrity of the Of the payload of this jwt whether somebody go and sample with it or not okay so So this is an example of how you can prevent man in the middle with a jwt Preventing man in the middle basically it just says that uh like to maintain the integrity of the token So for example, you have a hacker and a hacker Somehow got your token Got john's token. It's like a postman So like the postman got the letter and then in the middle he can do whatever he wants before he gives you Roland right So if if the postman hacker banned he got the token He cannot change the payload without Uh screwing up the signature Because the signature is dependent on the header and the payload So if any of these things change The signature would change So so the first thing is hacker ban cannot change the payload Uh without the signature the integrity check using the signature failing neither can you change the header Without the integrity check of the using the signature to fail too so All in all because hacker ban doesn't know the secret that is shared in between john and Roland He can he wasn't able to generate a new Signature after he changes the payload or the header So that is how uh, you can How JWT maintains integrity of the of the token Okay, so i'm gonna Go to this website to show you an example of uh How this integrity thing plays around so So so this is a secret. Okay. This is my very very secret Okay, my very secret Sorry my very Okay, and then this is the header right and then this is the payload so the signature is actually generated by This base 64 url encoded of the header with a dot and the base 64 encoded payload And then signed with this secret here Okay, so for example, i'm hacker ban then all I see is this and I want to change like Change the the header a bit so from e maybe I want to change to r Then I can see that then now it fails. It's a signature Yeah, so no choice cannot change then try to change Change the payload to that out of x then it still fails the signature So cannot change also. So there's no way no way Hacker ban can change this token Without failing the integrity check Unless he has the secret so if he has the secret he can change any of this data And then we sign the signature With the secret that he also found out then there is the only way to to be able to Attack Or to add change this token change the message that john wants to tell Roland Yeah, so So if you if you are very into this encryption style, you realize that Or maybe sharing a secret It's not It's not the best idea like Roland and john share the same secret then then you can bring all your like public public private ta Encryption and all these encryption ideas into this token, but basically the idea But basically the integrity check is that One side can sign sign the token and issue it out the other side will be able to check using this signature yeah, so So This topic on then the on this part you can go very deep so that one you can go and explore Okay, but the idea is this this is how jwt maintains integrity Okay, so the next part is then oh Now, you know the concept. How can we use do it in ruby? So, uh, actually there's a ruby gem called jwt Uh, which is a ruby implementation of this standard That we mentioned Okay, so any anything that you can find in the standard you can Is implemented in this gem So anything you want to do there that you see on the standard that you want to do you will be able to find a method To do it in this gem. Okay, so so We've because the jwt actually there's no need to maintain you just want to encode and you just want to decode Encoding is for or john john can encode and send the jwt out Decoding is for Roland to be able to read to get the jwt and then decode and check whether Check whether is the integrity is uh, still there and then to understand what is the data inside Okay, so so the encoding is pretty simple You just need to put a payload the message that you want to send to John wants to send to Roland and then put the key Put a key that is supposed to be our shared secret that that you use to Encrypt the to create a signature Then you can also choose a algorithm that you want to encrypt With so here by default it's hs256 I also not I'm also not a very not an expert on all these encryption algorithms, but I just know that there's many many types so Likewise in this gem they offer many types of encryption too. So you can go there and check A big reason if you want to use if you are if you can tell the difference And then by default it's hs256 And then You can put some additional header fields other than the default That was shown just now the type is jwt and something else which I cannot remember Okay, so so in this example on the right After we run the jwt.encode you can see that oh, okay, then It's a jwt. There is That is the output which is also if you look closely there's two dots. So it's also split into three parts Okay So this is encode creating a jwt. It will be Then the next part is then given this same token there. How do you decode it? You can also use the same jwt gem So the first few you just pass in put in the entire jwt Then the key you're supposed to put in the The decoding key or so the shared secret That you want to decode with here and then And then this verification actually In this standard, there's more than just checking the signature. You also can check or whether the Whether the The token is expired that you can have some rules. You can you can put in some rules to check Whether to put in like more validation rules for this token so that one You will put it put a true or false here whether you want to Check all these rules or so Then on the right you can see that if you pass in the token On decode then it returns you the message Yeah, so So this is how you can decode a jwt in Ruby. So actually I'm just like telling you how to use this gem That's only to do main methods And then what are the other things that you can do with this gem which is also comes from from the rsd Is you can have many types of claims like just I mentioned expiry claim you also can put in like not before claim So it's not valid It's not valid if if The token Is if you got the token before a specific time that john like Stated or you can have an issue ad claim then Uh, then you can determine like for example if john issue two hours ago Then you don't want to accept any token that is two hours old Then at this point in time you can use the issue ad claim tool To like say that the token is not ready. So of course, there's a lot of other claims You can go and take a look inside the rfc. Uh, you'll be interesting because some Like you you will have a use case for all these claims to make your beef up your jwt implementation so So now that we are talking about talk about jwt, right? It's just a token. It now is just a token that can Transmit data securely between two persons Uh, then what what can you use for so a very basic use case is actually authentication Okay, so I have a me. I have a small diagram here About how you can use jwt to Do authentication So for example, you have your api server On your left side and then you have the user mobile phone Maybe on the right side or a browser on the right side So when you when a user signs in Then you check whether the username and password is correct or not from your database records and then if it's correct you can issue a A jwt using your and sign with your own server secret and then You what you can put inside is you can put the user id equals to one And then you can put a expired reclaim to say that this jwt is only valid for two hours Then you send the token back to the user device So the user's mobile phone or the browser and then subsequent calls This this user can make subsequent calls using the jwt that the server gave him Give him or her okay So in this case the user can do uh to look up profile its own profile data Okay, so because the user cannot edit the jwt, right? So when the user send the jwt Along with the request to the api server The the server will try and decode with the Will try to check the integrity of the token with the the key that he the server created the token with Okay, so it can decode properly integrity intact, right then The the server should be able to trust the user id equals to one like the id is not changed Because if it is changed then the token will be invalid really So then the server will be able to just use the user id to look up the user data and then send back And then send the data back to the user device So basically this jwt can be trusted Okay, and then for example two out two about five hours later And then the user device try to Request data using the same jwt Then of course the server will be able to decode the The token but then by checking the expiry claim Then the jwt is not valid anymore Then the then the server can just tell the user that oh this token is not valid anymore You have to then the server then the user will have to get a new token somehow Maybe just sign in again to get the token again Yeah, so this is a simple example of how How you can Use jwt to To do authentication Because the token Can you can validate the integrity of the token? Okay, so of course authentication is slightly more complex than this. This is like more like a api authentication So of course it's slightly more complex But then this is the general idea that How you can use jwt to pass around So So of course Some people Will pass more information than just use maybe you can pass permissions So like now you issue this permission this use this token has permissions to Do this do abc then as long as this token is valid The user can use this token to do abc Yeah, so There are a lot of resources to read online And then I know that there are many libraries that that Like for example device device is a very popular authentication like ruby authentication gem But it doesn't use jwt, but I know that there's a plug-in with device jwt to make you do be able to do authentication with jwt Yeah, so There are many libraries so you can go and take a look. I'm not very sure I only know that they exist because I just use the idea and just wrote my own Or my own use case Yeah, so Uh, just some simple summary. So jwt is actually a very simple idea It's just a way to pass message while maintaining integrity So like it's not limited to authentication is you have any ideas that you think that you need uh, you need The message integrity to be there then Then you'll be able to use this like make it work for your use case and then it's very lightweight Another good thing is very lightweight because it's just like a string, right? There's some characters Maybe I think got a hundred characters, but that's about it Yeah, and then the next thing that is very very good is it can be generated this this This token is generated on the fly. So you want you can like when the user needs it Then you will call then you call and quote with blah blah blah all the information and then you generate there and then And then you can just send it out. You don't need to save anything On your server on your database Okay, then the next thing is you actually can let you encode like a lot of information There's no limit to what what message that you can encode, right? So you can encode a lot a lot of information and send it over at one go like in one a one in one What is it in one token? Yeah, so Some some people when some like for example, if I bring out the use case of permissions Like the very olden days of way of doing is like you have a table with your tag permissions and then you have a id A user id tag to a set of permissions Inside the database then when you then when the user requests or something You check what is the user id then you look up the database to check whether the user got this permission Then if you have the permission then you can then you let the person do that You don't have permission you can then you let the person then you don't let the person do that thing But in this case actually You can save the looking up the permissions in the database Because the token is issued with the permissions inside the token You can trust that the permissions are inside the token is correct. So when the user Requests to do something you decode the you decode the token you check Oh permission is there then you immediately can let the person do the thing You don't even need to go to the database to look up. So this is one example of how we can save your Uh resourcing, okay But this very very but this last one is the most important part Is that do not ever encode sensitive information? because the header and the the header and The payload is actually just base 64 url encoded. So for example permissions is not sensitive You can you can uh let people read it doesn't matter like user id let people read it doesn't matter As long as they cannot break into your system to get any information But if they have like credit card number then that one saving inside the database don't put inside the the don't put inside the token so Uh, even if they even if you have to read up the database to get that get that Uh credit card number so be it because this is so sensitive, right? So sensitive information don't Don't put it inside the token Yeah, so Uh, these are some of the resources that I have uh gone through Oh, I think it's helpful for this topic Yeah I think yeah, so so it's a very interesting idea. It's a very very simple idea, but it's a very very powerful concept So I hope you learned something new today And yeah, thank you for your time Thank you Okay, anybody got any questions? sorry for the recursion Uh, okay, I think Yeah, I think that's about it for today Thank you, man. I got I implemented gwt once And I really I didn't really have any idea of what was going on I was just just using the library and making it work But I I have no idea of the logic that was going on behind and the thinking so thank you for explaining that Welcome Okay, so there's only three people here, right? Not sure whether anybody was That whether it's helpful to even do any job shutdown Okay, and then Yeah, so the next meetup is Oh Oh, I didn't update this. Okay. It's actually July second Second Tuesday, but then not confirmed yet our But I created a meetup event already. So I might I might change It but I'll update again. Okay so as usual For cuisine and Miguel if you you haven't joined our telegram group, please go ahead And then yeah, that's the end of the meetup Um Thank you very much Miguel for speaking and thank you very much for coming So, yeah, what is do you guys again? Thank you. Thank you for organizing and And keeping the meetup together keeping the ruby community together I'll come. Okay. That's it. Thank you. Thank you, Michael for recording or so. Yeah, no worries. Thank you. Bye. Bye