 I think we can start. Hi everyone. So it's the first meet-up about Advanced Ruby that we want to organize here in Singapore and we would like to continue later. This time we will be exploring how everyone's favourite libraries, Prokets, works. Yeah. First of all, we've got sponsors. Thank you to Pivotow Labs for sponsoring the venue and food and drinks. Thanks to Michael for recording this meet-up. I hope that someone will watch it and more people will come next time. And before we start, who watched the presentation that I posted? How many people watched it? Okay. Who watched at least 10 minutes of the presentation? Yeah, me too. I did watch it a whole lot. Okay. And now more difficult. Who at least tried to read the source code of Prokets or Prokets Rails? Okay. Okay. Since there's just two of us, you will replace me later explaining some part. Since this is a meet-up where there is no one presenter, I would like everyone to contribute by asking questions and by explaining the things that we will be reading here. I'm just going to go through some basic stuff with Prokets, how to use it and how it works on the surface. And then following that, I hope that we will go deeper and deeper. And I'm sure we will not understand even 30% because it's too complicated. But I hope that we will at least reach some level from which you either can understand the rest of Prokets yourself or you can realize that you don't want to know how it works. And you're just happy that it does. Okay. So one thing about Prokets is that it has a few versions now. So the version that you're probably using in your application is version 3.6. And if you're having some Rails 5 application or you're using the Rails master branch, then you will have the version 4.0 beta. So it might have happened that if you check the source code and I read the source code, we were reading something totally different because the difference between these two branches is around 500 commits and around a year of work. So it's quite significant. So in this code that I will show you, I will use the version 4.0 beta. And there is this second gem that I will actually start with. It's called Prokets Rails. So this gem is a layer between Rails and Prokets that allows these two libraries to work together. So I will maybe start with how I actually started digging into the code. So first thing that I did was... I don't have Prokets Rails. Let me just quickly download it here. Ah, yes. Not good enough? Or... It's too small? Okay. So the first thing that I did was I checked how Rails uses Prokets. So I went to do the Prokets Rails gem and I checked the railty. And I didn't learn much from here. It was a bit complicated. But what I found is... Is it here or is it in the task? Yes, this. So what I found is how the Rails assets pre-compile task works. So this is the entry point. This is what you call when you build these assets files in your production. So then I checked that what it does is actually... It's just one line of code. It's manifest.compile. So then I checked what is the manifest. And manifest, and here we're going directly to the Prokets library, is an object from the Prokets. So now we can go directly to that gem. It's here. We've got the Prokets and we've got... Manifest. So this is the object that we built, the manifest.new, and then we compile it. So now we need to check what actually this manifest does. Fortunately for us it takes the asterisk arcs as argument. So basically we have no idea what it means. So fortunately right after we see that okay. There is environment. There is directory and file name. So we've got some hints here of what the manifest can do. So we've got this initializer that I didn't really go deeply later. I just checked that this method called compile. And once again it takes arcs. So once again I don't really know what exactly these arguments mean. But fortunately documentation says that I can provide application.js. So that sounds like a plan. So let's start by doing this. So here I've got my Rails application and here I need the console. So I will do sprockets manifest.new. Okay this requires output file name. So let's call it manifest.js. Okay it created an object but it apparently doesn't have data that it requires. So we need to provide some data. But let's see what happens if I just do manifest.new and compile. Okay it requires environment. And this is my biggest trouble with sprockets. So it has this environment file. It's here. This file and this object of environment is apparently just a whole configuration for your application. So you provide the paths that you want sprockets to take care of. You provide some caching. And this file seems small but actually you need to go to base. And it's not so small and it's not so easy anymore. So it includes a bunch of other stuff. So anyway I figure out that I will not get deeply into this environment and I will just check how Rails uses it. So Rails does something like Rails.application.assets. And here we've got this environment file. Environment object. So I can provide this object as environment. So if I do something like this. Environment is first. Okay thanks. Okay now I do manifest compile. Okay it returned me nothing. Because this file needs arguments. And the argument might be some files like application.js. So let's see what it will do. Okay it rolls. No okay it works. So now what you did was. You see I've got this new manifest file here. And this file just contains the information about the files. So my file the logical path is application.js. But the path the actual path one will be application.dash. And then this hash of this particular file.js. And the same will be with more files. If I provide CSS or another files we'll have this. And except for that we've got these two new files here. So these are actually both the same files. One is just gzip the other is not. The other file is compiled application.js that I have. So my application.js what it requires is just jQuery, jQuery, ujs, turbo links, and then all the files that I've got in directory in this tree, which is nothing. So the only thing that it contains is packed jQuery and these other two libraries. So that's the extremely basic version of how rails and sprockets work together. It just creates this manifest file, manifest object, you compile it and that's it. But if it was that the sprockets wouldn't be so much both loved and hated by the Rails community. So there is more than that. As you see this file here is just JavaScript, right? It's not even minified. We've got all the comments. Basically it packed all the content of these files together and threw it into one file. That's one of the reasons why it's so slow because what this Rectas does is it takes all these files together to a memory and then it writes this big string that is like a few hundred kilobytes to one file. That's why it takes so much time because Ruby is not extremely fast, especially with handling such big strings. So that's the manifest and that's how we compile all the files, but this is pretty boring. I know how it packs all the files but what happens if I have coffee script or SCSS, right? This is just copying JavaScript to one file and that's it. So then I thought, okay, sprockets has two kinds of processors. First is preprocessor, which means that it does something to JavaScript or CSS before packing to file and then it got post processors which formats them after that. So example of the preprocessor is directive, it's not here, it's here. So it's directive processor. What it does, it takes your application.js or application.css file and these weird comments here that you've got, it actually parses these comments and requires all these files. It basically finds all the files that you want in your JavaScript or CSS and it grabs them all together. So let's try to just do this one thing. So we've got this sprockets directive processor object and it accepts a hash and this hash needs to contain data. So let's start with saying that data is empty and see what happens. And now it fails. It fails because I don't cure data, okay? So I broke something. Let's see. Oh no, sorry. We do not need to initialize, we will just directly use call. So we'll do call with data and now it crashes because there is no conversion of new to string and that's not very useful. So we actually need to check what exact input it requires here. So we've got this call which calls underscore call and then we've got a bunch of variables that it creates. So we need to provide a hash with certain predefined fields there. So let's try to just define all of these fields that we have and providing nil there. So we've got, let's use empty strings. What else do I need? Metadata which needs to be a hash and URI. Okay, now let's try to call it with options. Okay, there is some progress. That is direct processor worked. So it processed the file that I provided which means no files and it returned me a data which is exactly the same as I provided because there was nothing to do with that. So okay, now as data I will provide something else. So I will just read this file. Okay, we've got the content of the file is data and now I'm calling this and it found the content but now it checks the file name. So the file name is, okay, so it checks the environment. So I didn't provide the environment instead of creating my own, I once again used the Rails application assets and now I call it again. Okay, and now it's returning us something. So let me show you. So now we've got the dependencies that it grabbed and they're required. So required are the files that I provided in that application.js file. So it requires the whole, if you see here, application.js. So it requires like jQuery, jQuery.js but also it requires the whole tree and the whole tree of this challenge is empty. It's also the cable.js. So it requires that and based on that files then it builds the whole dependencies. So for example, jQuery Rails inside it requires a bunch of other stuff and the same with like, the same with turbo links. So we've got this big list here and it only requires, as you see, the data that it requires is the parsed file without the lines that actually meant anything for us. Let me show here. So here we've got this text. This is a manifest file, blah, blah, blah. And this is what we get here, exactly. But in the end here, we've got read sprockets, read me for details and then it's end of this data. So it means that this data was actually parsed by the directive preprocessor and that it was removed later. So that when we provide the output of this as data to another preprocessor, it will not grab this anymore. Okay, so this is how it parses the data but we do not have very meaningful output here. So instead, I will create a simple coffee script file to show you how it's parsed. So we've got just extremely basic coffee script function, the function called x returns 3. And now I will use sprockets coffee script preprocessor and I will call it with options. Okay, let me first create coffee script options. Okay, and cf options, assets, Java scripts, objects, coffee, what? Ah, yes. Okay, perfect. So we've got this and now let's try to call it with the options. We've got sprockets, coffee preprocessor, call, cf options. It requires some more stuff for me. It will be coffee script preprocessor. What does it want? I need to provide cache there. Okay, cache needs to support the fetch method so I can't provide nil but maybe I can provide just a single hash. Let's try it. Okay, now it worked. So now you see that it's returned us to things. First is data, which is what we will be parsing later to another preprocessor and then it requires additional maps. So you've got the source map that maps the generated JavaScript to our coffee script. So for example line number 5 and call 11 from the generated JavaScript will be mapped to coffee script line number 1 and call number 7. So this map later can be used to debug the coffee script. Okay, now we've got this original, now we've got this coffee script, JavaScript here, but it is not minified. So now this data needs to go further to the next processor. So we've got cf result and now we'll try some Aglifier on that. So we've got Spocked's Aglifier Compressor and we will call it with cf result. Uh-huh, of course it didn't work. So now what I need is Aglifier line 56. Okay, it needs something like metadata merge. Okay, and now you see that the data it returns this JavaScript but it's minified. Instead of we don't have the enters, we don't have white spaces and the name of the function was shortened from X to N. Okay, that's not extremely useful. Okay, if the function was longer, right, then it would be shorter now. Okay, I think these are the basics that I managed to cover and to read the source of Sprockets. If you have any hints or questions or if you want to come here and share something else that you learned from Sprockets. No questions? To which purpose? First one, here are the options. I don't know why filename was needed here. I have no idea. Yesterday when I tested it, I provided filename. Today I was just lazy and I wanted to check if it works without it. So I'm not sure why it's used for. We can actually check it, right? What? On the output. Okay, let's see what will happen. Where do we have options, merge, filename, test file, JS. Okay, it returns. I don't see it anywhere here. No, there is no test here. So I don't know. Oh wait, maybe it saved it somewhere actually. No, neither. So I have no idea what it does. Let's see. Okay, it uses it to generate the name of directory and finds the path and it doesn't use it for anything. Okay, no idea. Anyone has some hint? I have no idea what's the meaning. We can check it. Okay, so here should be some... No, there is no comment in this file. Okay, I've got no idea. If someone finds something, just tell me. Yes, but if I don't provide this file and I provide the content, it still works. Yes? Okay, let's check. So this file will have some, I don't know, it will have a method that doesn't work, right? Choir. So let's try now. Sorry, I didn't reload the content. Okay, now it's fine. Yes. No, it's not. So I think that we use the founding. Yes? We use the founding to get the directory. Yes. If we get the directory, the founding and the directory is used to... Let's say we require another directory. So you can try something. So I can provide the whole directory instead of just the file? There's a directory that requires a directory. Okay. So let's say... Okay, so let me create. Okay, so now I've got it in the new directory. What now? Now I should do require directory, right? In my... Instead of this, we'll do... How do they call it? Up here. Okay. What now? So you pass it in... Okay. Right here. Yes, I've got require directory, so I've got options. Now I should be able to... What happened with this? Okay, so I've got sprockets, directive preprocessor, call options. And it... Require directory argument must be directory. But it is. Okay. What now? Okay, let's try... It says that it must be a relative path, right? Okay, let's... Let's check in sprockets. Oh, in the documentation. Okay, that's not extremely helpful. All source files... Is it...? No, there's no more. Okay, let's do... Instead of the app here, let's try this. Let's see if that will help. Okay. Okay, work now. But I still don't know what we... But I still don't think we... Do we provide the file name here? No. So we didn't provide the file name. Oh, let's try if we can do it with Niel. If it will complain now. Okay. So it needs to have some file name. It needs to be a string. But it doesn't matter what it is, apparently. Now this is the code that I wanted. And add another directory. Add the sub for the directory for it. Then the file name is the... Just the file name of the directory. Well, I'm thinking if it doesn't have any files, it has an environment from real. So if I deposit, it knows where to app the files. The what? Okay. So probably we can... create a sub directory under the... JavaScript directory. And then... create a new file there. And then... Okay, tell me how to show it. How to show example of... to show it so that everyone can see... what it does. You have an... We just created an app directory under the app directory. We just created a file under the app directory. Where? In the app directory? Yeah. We just created an app directory under the app directory. This one? The app directory under the app directory. This one? Yeah. What to do with it? We created a file there. Okay. So I've got the file there. So I will try to just require that one file. Well, it's like... Can we just... compile the test of yes... copy instead of the... application of yes? Oh, okay. If I... If I try to compile this... just this file. And then in this file, we can require... from this file, we will require the parent folder. We will require that file. Require just like here. Okay. So... Okay. So this is copy script. So we need something like... require directory. Something like this. Okay. No idea if this will work. That's okay. Let's try it. So now... Okay. Options. Data. Sets. Of course. No. Let's again. Scripts. Okay. Now it works. And now I should be able to do... sprockets... directive... call... options. No. So we didn't even scan it. Because normally the directive processor... it shouldn't require this information, right? It shouldn't... return it. Yeah. Okay. Let's see what will happen... if I do... just require jQuery here. For example, if people find that... they want jQuery. Okay. Okay. We can... Okay. Now it should be fine. Okay. Got it here. We'll call it... Okay. It set... the dependency. Okay. But now I'm confused. So instead of... doing the require directory for the parent directory... it took it for the parent of the Rails root. So it just took the directory with all my projects. I'm trying to put it in the final name. Final name? Yeah. I didn't see it before. I mean... Here? The right side. The what? The right side. Final name. Okay. Okay. And what should I set? The final name to the data. This one? Yeah. Okay. Got it. And now I will call it and... Okay. Hey, that works now. Okay. So to provide the data, we needed to provide the path because if we didn't provide the final name, if we didn't provide the path, it assumed that the path that we are passing is the main path of the Rails application. And that's why when we did like the parent directory, it showed the directory of all my projects instead of the one here. Okay. So final name is useful. It's actually needed. Okay. Hey, this is also really cool. Okay. More questions, more things. Maybe someone knows how later Rails finds the assets because I've got back in my application and I can't solve it. Maybe someone knows. The problem that I have is that when I use asset URL or asset path, it doesn't find a certain asset that I want. And it happens only in production. So it all works locally. But when I do the same thing in production, it doesn't find the asset. The one. Okay. Let's then check in the Rails code how it does it find the asset and then maybe we'll figure out what is the difference between the production and the development while the same thing, the same asset is found in development and not in the production. Is it a path structure problem? I have plenty of those. So because everything is flattened, equivalency keeps the structure. We compile everything. Okay. So everything is put directly in public. It doesn't keep the directory directly anymore. Okay. I will check this. I will show this. I will show this problem, actually. So this bug was reported half a year ago. It says that there is no method find asset for nil class. And the problem is apparently that this jam primary Rails which inline CSS into your emails that it used some private sprockets API. And this API was changed in the new version. So whenever someone updates sprockets like this doesn't work anymore. And the discussion goes to April where I hope to find solution. And this is the last trace of that bug so far. Okay. I will check the different directories. I tried this solution but it didn't work. So yeah. Okay. I think if there are no more questions or someone wants to show something created with sprockets that you read in the source code. Anything that you found that is interesting. Okay. I think we will not try to try to explore it more. But what was suggested on the meta page is that we should choose the library that we want to learn next time. So I'm not sure if you want to next time if you come to work with sprockets or if you would like to try something else. I'm not sure if I want to learn more sprockets. I think I'm happy that it just works for me most of the time. And it's a bit difficult to explore more. Yeah. Before the asset of my mind he was very clear what to do when you want to use a library. Not those big libraries where you can find a gem. Those small ones that you need to use for doing small things. And usually they have a structure where they put their own assets. The image or the CSS. They say select two. The one that contains the select one. Let's say we don't need the gem. I do not want to pre-process the library to fit into the assets pipeline. I would like to say that this folder comes back to and put it in the rendered assets pipeline. That's how it should go. It doesn't work like this. No. So I would smash everything to find the assets pipeline. Because it's getting that structure. So what people suggest is you have to scan the library and change everything and use assets. Okay. So we say that if select two for example requires some image and the path of the image is like, I don't know, arrow.pmg. That it doesn't find it anymore. It draws something on the own organization of the library. Yeah. And then it's forget to smash everything to one order. So one possibility that you have is to copy all the images to public directory, which is not convenient, but it works. The other one is that's why we have all these jQuery dash rails gems that basically take the jQuery or some other library and parse it and add these footprints to the finance for you. And the third solution is called Rails Assets. So Rails Assets basically generates these gems like jQuery dash rails for you. So you put everything to gem file. Like you can have bootstrap. You just add rails dash assets dash bootstrap. You add rails dash assets.org as your source. And if it doesn't have this gem, I think it will generate it. So it will take the library. It will parse this files. It will generate the code that later will add these footprints to all the files names. I haven't used it myself, but a few of my friends use it and they say that, well, it's not something they would love to have, but that's good enough, right? What I have to do is, as we say, split around the vendor assets, the base CSS. But you have to tell sprocket to get all the digest. You have to make a version and it's not like this. So you have to reverse the process. Yes. Because you want to use partially asset pipeline and partially you don't want to use it. So I don't need to pre-process my library. It's omakase. It's like I give you, you either take it or not. You can't change it. No, I'm just kidding. Of course, I agree with you that this is a problem, but this is just like the sprocket said that either it works with everything or you don't use it at all. So what I suggest now when people have this problem is not to use asset pipeline at all. Just use some JS package. Use, I don't know what is trending now. What's branch or webpack or maybe it's also old now. Okay. Yeah. I'm using Bower. I'm an old man. I'm using Bower and I'm... Fail. Yes. Fail. Well, it works. It never failed. So I have no reason to... I have no reason to change it. Look forward the better ways. Sprocket or you use Bower Rails? I don't know. I think I use Bower Rails. Yeah, right. Yes. Oh yeah. So in my gem file, actually some of the files are in the gem file. The ones that I need to replace this asset. Some of the ones that I don't need. I can use either this Rails asset or I can just put them into Bower JSON. So it's like, for me it's a balance that I don't want to keep all my JavaScript in gem file. I want to like keep as little as possible. But because asset pipeline is better than no asset pipeline for me, then I still try to use it. Yeah, but the perfect solution, if you don't have to use asset pipeline and you don't want all the stuff that it gives you, just use some JavaScript, purely JavaScript. And the problem with the JavaScript is that it's off. You can't resolve it and then you don't have to use the asset pipeline. It will give the image. Yes. So it has this Bower Rails gem, has this rake task that will get all the JavaScript files and that will attach automatically all these digest. So you can actually put like, I don't know, this select two, for example, it will be in your vendor. You don't need to use the gem. It's like this Bower Rails will do it for you. But then because it's partially Ruby, partially JavaScript, it's still not like very pure solution, right? I don't really understand how the Bower Resolve works. How it works? Yeah, that's the result. I feel like, okay, I have and then it works, but is that my deal? Okay. Is that good topic for the next video? Bower Rails doesn't really work with Heroku. No. I don't use Heroku. They don't have any endpoint in Asia, right? They don't have servers in Asia. In Heroku? Yes. Probably, yeah. Okay. Yeah. That's why I don't use them. If I have clients in Singapore and server in Singapore, it makes sense, but having clients in Singapore and server in the US, that's really... Anyway, okay, so let's maybe quickly check what are the other libraries that you would like to see next time. And I think that the Bower Rails is number one, right? Any other suggestions? Any other ideas? Okay. But this is not like a library, it's just like part of Rails. Okay. Okay. What? Right. Right. Right. Okay. But then you are sitting here. Okay. Anything else? Any other suggestions? One from me is Sidekick. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. check that. Okay, so what I will do, I will add all of these as suggestions on github issues in the Ruby SG Meetups and then everyone can vote and let's say in like two or three weeks we'll see which has the most votes and then we'll prepare, everyone will prepare a bit of that library. Okay, well I think this is all for today. Thanks everyone for coming and I really hope that you will come and also also next month. Thanks.