 Hi, I'm Chris Hiller. This talk is possible tools the present and future of tooling and Node.js So let's get to it So again the title of this talk is possible tools the present and future of tooling and Node.js So my name is Chris Hiller. I'm a developer advocate at IBM. I go by bone skull on the internet I'm a Node.js core collaborator. I Help maintain mocha, which is a testing framework that some people use I'm also an open.js foundation cross-project council voting number And finally I am often a panelist on a fun podcast called JS party My avatar is displayed here. It's a smirking orange skull within a black circle so I'd like to start by defining what Node.js is I'm sure you're all familiar with it but Let's get it from the horse's mouth So I went to the Node.js.org website and I clicked through to the about page And I read the text and it said blah blah blah Node.js is designed to build scalable network applications blah blah blah blah blah so You know that meme on Twitter the the narrator meme, right? Where you you say you're doing something and then the narrator pops in and and says how it really is well Here's the narrator mean It was mostly not used to build scalable network applications So what do I the narrator? What does the narrator mean by this? Well the narrator means Node.js builds the web This shouldn't be a controversial statement. So we have data. We know that people Mostly use node to help build websites. They use it for what? Developer tooling right they use it for bundling testing linting all sorts of things so Here's the more controversial statement Node.js is a tooling framework Now it may not have been designed to be a tooling framework But that's kind of where we're at so There's kind of a problem here, I think And I've given this problem a name Now may not make too much sense, but it's a name call this the Node.js server tooling impenance mismatch and so you may have heard of a a similar term which is the object relational impenance mismatch and so this term I just made up and so The deal behind the object relational impenance mismatch is it's difficult to map the concepts of a Relational database to an object-oriented programming language in vice versa So if you've ever used an ORM You're probably you you've probably felt this problem You know we can think of objects in a programming language As a graph or a hierarchy While instead the data in a relational database is tabular So in a relational database, there's no notion of a class. There's no notion of encapsulation There's no concept of inheritance or polymorphism There's there's no pointer for example, so you might have a foreign key, but it's it's not like a real reference It's not a pointer So in the case of Node.js you have this core and this core was designed to serve network applications That's what it was built to do But that's not what it turned out people were gonna use it for and so Because of this, you know node lacks this foundation it lacks some first-class support for building tools Happily people went and they started building tools anyway But you know That's kind of where we're at now Unlike the object relational impenance mismatch, which is sort of a wicked problem if you will The Node.js server tooling impenance mismatch can be solved so If you were hoping for me to talk about you know why Why it is the way it is like why? It started as this this network This this platform for building network applications and ended up as it as a tooling platform Well, I'm not gonna I'm not gonna go into or speculate why Why it is the way it is why why node invested so much in the in the network allocation side of things But you know, this is where we're at right now and that's what I want to talk about this the title of this talk Does not say the past its present and future and that's what I'm going to talk about so What can we do about this You may not have heard of the Node.js tooling group but there is a thing and it is called the Node.js tooling group and It is a official group not not a working group, but it's official in in the Node.js org To the right here, you're gonna see some art. It's a Trojan horse and the joke here is that You know the tooling group wants to work within Node core to improve Improve things for for tooling authors and consumers of tools and so we're gonna get We're gonna get in a node core and we're going to you know fix this from the inside out so not really a declaration of war, but You know, we need to we need to change things at the node core level So let me talk a little bit about the Node.js tooling group So this group was formed in 2018 Myself and some others I had talked to some people In the community and we all shared a lot of the same frustrations and so we thought we should try to come together as one mind and and help solve this So some of the members in the group We have maintainers of NPM mocha. That's me. We have maintainers of Istanbul NYC Yarg's create react app. These are popular tools and libraries Some of us are node core collaborators It's an active group we've got features under our belt, you know, our members have have gotten Gotten things in we have current initiatives We have ongoing initiatives and and some future pine sky stuff and we're gonna talk about these later We have a slack if you want to chat it's node-tooling.slack.com Go and sign up for that and pop in and there are channels for various Tools like there's an NYC channel a yard channel an NPM channel, etc So a lot of us gather in there and talk about building these tools We have open meetings every other week meaning that if you want to come and participate you can come and participate And so the those meetings will be there will be an issue in that github repository, which is again node.js forward slash tooling Also, we stream these meetings on YouTube. So If you don't want to participate just want to watch You can just watch There's actually going to be a meeting here at the open JS collaborators summit not not here here Because this is open JS world, but after open JS world is this collaborator summit going on and we're gonna have a meeting there so might want to check out schedule and See what time zone that's in wherever you are and come check us out if if you're interested in building tools or if you're you know a consumer of tools and really Have some ideas about how things could be better But yeah, whoever you are and if you're involved in building tools with node, we'd love to hear from you So next I want to talk about a few things that we have done so far So these are some completed initiatives that are that our members have brought to to node We have a cursive file system operation. So make their and render We have native source map support, which is which is still experimental I think but maybe not soon native code coverage support That's that's improving again. These are these have landed, but you know, there's room for improvement on some of these We have flag introspection, which is useful if you want to If you're a tool that needs to spawn other node processes Or you want to pass node flags through using your your command line tool and Now this is not node core but in my mind. It's close enough npm workspaces are landing soon, which Sounds pretty cool. If you are familiar with yarn workspaces npm workspaces are similar And I'm excited about that because I have a mono repo that I could really use some Sort of workspace going on there. So That should land. I'm not sure when exactly maybe in the next major of npm So that's what we've done again The way we work is we kind of come together and we just say hey, you know, I Find this to be a problem. What do y'all think and Other people agree. Yes, this is a problem and how can we make this better? And so we brainstorm we think about it Ultimately, it's up to the individual. So somebody has to send a pull request. Somebody has to write the code And just get it done. We don't have have a roadmap per se. We have essentially a list of Initiatives that we'd like to look at and whether or not anybody is working on any of these at any given time, you know Who knows but You know, we don't we don't come in and say you there you in this tooling group must spend time doing this Particular initiative. That's not how we work. That's not how no JS works No roadmap just you know, whatever we all think is important. That's what gets done So now let me talk a little bit about our current initiatives in the No JS tooling group first one is Reloadable modules. Maybe there's a better name for it But here's a deal with with that So you can't easily reload an ES module like you would a common JS module So you can't go in and mess with the require.cash. There's no API for it You just you can't really do it without without hacks So what this does is it inhibits tooling it inhibits Module level mocks. So if you're using a library like proxy choir or rewire mock You're gonna have trouble with those ES modules because I mean, it's just it's not gonna work because once the module's in You can't change it It inhibits tools that need to watch files and reload them. So maybe that's a test a test runner or something And I think right now not a lot of people are really feeling this pain and as more Developers start Creating new packages using ESM and no JS there. This is gonna start to come up a lot more. So There is a little bit of urgency to to address this Because it's just it's not gonna work like you want it to it's not gonna work Like how it worked before with common JS And so to solve this We need to collaborate with the no JS modules working group You know, maybe even v8 the v8 team at Google, but yeah, so this is This is kind of going to be a bigger deal As more people start to adopt ES modules So we need to get this one solved Next one is my favorite argument parsing so You know process dot argv dot slice to Which is a really kind of ugly way to try to get at your your command line arguments so I think node can do better and I think it can do better with a very minimal API and provide a lot of value so This would have a bare minimum of features So very very simple stuff. Maybe like one option or something The use cases here that we'd be focusing on We we want this to be good for simple tools code examples, you know learning materials or One-offs where you have a server or something and you need to just quickly add a Flag or two now that quickly little snowball if you've tried it Trying to parse things and and all sorts of stuff. So I think a little nice Straightforward API could could go a long way here So I'm looking to get this done Right now. We have some some pretty solid ideas about how this should look and and to the right here. We see an example The array that you see foo bar baz That That array is similar to what you would get in process dot argv that slice to so By default, maybe it'll just use that but you could you could give it your your own array of command line options and it's going to return an object and object will have keys based on the the options and Values as appropriate or maybe they're just flags and then that underscore there you see that is a positional argument. So Basically, it's just an argument without dashes or anything in front And that's kind of a convention that other libraries have used now. This is not expected to replace The user land libraries that we're already using to parse arguments like yargs or commander those are much more full-featured and If you're reaching for those right now, you know, you might be able to get away with something like this but Again, those those are those are going to give you so much more than than what we can do here So it's not really intended to be an API that you would want to build on top of But more something that a user can just just pull in and use directly Without having to go and reach for a package on npm And the next one is is is a rather large kind of cross-cutting Initiative and I don't have a good name for it. So I'm going to call it ultimate mega hooks. And so the idea is You don't want a monkey patch Stuff in in node itself. You don't want to go in and replace things in the FS module for example and you know, there are User lane modules that do this and they do it by necessity because there's no other way to accomplish what they're trying to do and so The from several different areas several different teams within note have identified that look We need some way to hook in to built-ins. We need some way to either You know Change something that a function is returning. We need some way to tell if a function has been called You know, we want to spy on a function call that sort of thing And so there's because you know several different teams like I know diagnostic security and tooling we all recognized That this was a need for different reasons So if you're building a tool like a package manager like a yarn or an npm you Might want to do something It's a plug-and-play right so plug-and-play the idea is that it kind of downloads stuff on the fly for you but what it does to accomplish that is It it it mucks with nodes module resolution and and sort of gives you this this phony File system and so right now to accomplish that you have to go in and you need a monkey patch Built-ins and of course the reason you don't want to do that is because it can break stuff so, you know if if we have something in here a hook where you could go in and attach this hook and You know do it in a way that is not going to leak out or impact other libraries, you know, that would be ideal Another use case could be for application APM tools instrumentation if you're trying to go in and Grab some diagnostics grab metrics on your apps This would be a great way for those tools to instrument your code Sandboxing is another one I think this is the the security teams concern With the sandboxing you may be able to lock down certain things if necessary and then Again from the tooling side, you know, this could Provide a great way to actually mock built-ins in nodes. So maybe you have some tests that Want to touch the file system You can write, you know some sort of integration tests that way and use these hooks So you're not you're not be fouling the file system with a whole bunch of files and Yeah, so there's a lot of different use cases now This is and this is going to touch a lot of places in node core And so we are forming an ad hoc group somebody's actually gone and There is a meeting Collaboration session at the OpenJS collaborator summit later this week So check out the schedule if this sounds like something you're interested in maybe Helping work out the requirements or maybe you're interested in implementation But yeah, let's check that out and I'll be there. This next one is kind of an ongoing concern More file system operations. So when I started with Node.js, and I think a lot of people may have had this experience It was essentially missing some APIs to do things so You know, I came from the Python world and of course Python has it has everything in there And so I was missing a RM a Remove directory or Christopher move directory like a prune type of thing or a copy tree Type command if you know the sh utils module in Python So some of these are we have these these core File system operations, but they're not they're not they can't be used in recursive mode So I mentioned Maked her and rimmed her earlier where we had added recursive options to those methods and This would also be you know, maybe we should add file system operations for CH mod CH own CB file Maybe there's something else But that could be very helpful because these are these are really kind of I'm not going to say common, but they are pretty fundamental tasks that tools need to do Another one is glob support. Maybe be further out on the horizon, but the the glob user land package is kind of ubiquitous and You know, why do we why do we pull it in? Well, if you want your command line app to accept glob patterns and you want to do it in a Platform independent kind of way you're gonna need something like the glob package because you can't rely on the shell to to do it the same way across of course across shells or across operating systems and so Adding something like glob support to to to core would really kind of make it a lot easier for people just to pull it in and deal with files this way and Yeah, I'd like to see it get done. I'm probably further out, but Yeah, that's a good one so Next I want to talk a little bit about this is the possible part of the talk So let's talk about the future. This is where the navel gazing begins and I know you can't see me I'm not on camera, but I am gazing at my navel right now and I want to start with Windows parody So Windows parody another casualty of the server tooling impotence mismatch You know, this has a lot to do with how node was designed what it was designed to do and Especially you know where node.js is intended to be run. So if you are writing a network application Where you're gonna deploy it you're gonna deploy it probably to Linux and so because of this Linux is really the first class citizen and note the libraries that are That node is built upon are also Linux centric and so Windows support has kind of been bolted on and there's only really so much you can do because it is a different completely different different operating system, but This is problematic, of course because more developers are using node on Windows and any other operating system But there's also kind of a bias. So the node core developers historically and probably even currently are Using Linux or Mac as their daily development driver not Windows And so if we're not feeling this pain daily that that Windows Developers the developers on Windows might have when using node It's not gonna it's not gonna be front and center. It's so it's a little bit out of sight out of mind And we haven't we haven't heard from from either the two people that deploy node to Windows But you know, if you would like to get in touch, we'd love to hear from you Otherwise if Windows is your daily driver if you write tools You know node core really needs people that that develop on Windows develop tools on Windows Yeah core and the tooling group would love to hear from you The next one is also a bit of a cross-platform issue That's FS watch and FS watch file if you've tried to use them before they don't they simply just don't work very well outside of a few limited situations mostly on Linux and so What we have here is a little screen grab from npm and there's a user land package called Chocodar Chocodar. I don't know I'm saying that right, but what that is it basically fixes file watching on on several different platforms and so This graphic is the weekly download count from last week. So 23 point five million downloads of of this package Because FS watch and FS watch file don't work very well and so this graph is essentially up into the right so You know FS watch this is going to be kind of a tough nut to crack You know it may come down to to pulling ideas from from these user land packages and bringing them into node but you know as always if If you're interested in helping solve this, please get involved with the tooling group. Please get involved with node core You know anybody can send a pull request, but Yeah, we'd also love to work with you in the tooling group to help get this solved The next one is self-contained distributables and so this would be node apps without the node so in short it's a way to package up your command-line tool and Distribute it to your users and they don't have to have node installed. They don't have to have npm installed So there's a user land package called pkg by Zyte. It's a Vercel now The description of this package is this command line interface enables you to package your node.js project into an executable That can be run even on devices without node.js installed And so the way this this works is it basically has to compile mode and then compile your specific tooling into that package and then and then create a binary executable So there's definitely room for improvements here, and I think those have to happen on the node core side So certainly if you're compiling node and you are adding some extra JavaScript to it It's gonna be a lot of stuff in there. You're not using So maybe it's stuff in crypto. Maybe it's stuff in HTTP HTTPS Maybe it's worker threads. There's all sorts of things that you probably aren't using that are gonna end up in that executable And that's essentially just dead code So, you know, we could get those binary sizes down we would need to find a way to to best do that Building node is also not fast. How can we how can we improve that situation? The startup time of node can be can be very very poor compared to you know, obviously something compiled in you know with GCC or some or a REST program compiled down that startup time is kind of rough You know, maybe it'll make a difference. Maybe it won't It really depends what you're doing But if you are interested in this problem, and I've heard this from several other people Please come into the node.js tooling repo check out issue 32 and Add your comments add your use case What we really need I think at this point is to kind of come up with a strategy to solve it There's certainly a lot of different ways we could tackle the problem. Which way is the the best way is is That's still on the table. We need to figure it out. So I'd like to thank Benco for this idea Built-in command line tools So the problem is this You have a package JSON you have a script property in there and in that script property are some scripts and Those scripts are in a shell the things that you can do in there are pretty limited because That that script may run in different shells might run in command might run in PowerShell might run in ZSH might run in bash You can't do there's not a lot of wheel room in other words And so say you have a script and you want to remove a directory Well, how are you going to do that and you can't so you can't just Expect a shell command to be there expect the the flags to work the same So if you want to do that in a portable way, you have to go right now and you need to You know either wrap nodes built-in Remder in in the FS module and expose that as a command line tool or go and NBM install rim raft and so rim raft provides a C a lot and that's what a lot of people do They just go download rim raft and instead of calling RM dash RF You just call rim raft because it's in JavaScript and it just works So this is the problem because node runs in many environments. It is more portable and that's why people reach for rim raft And one solution to this would be you know people have suggested NPM is the is the right The right tool to solve this and I don't think so. I think this needs to happen in Node core So no JS prevent would provide a CLI based or a set of CLI's Based on its own built-in modules. One of those could be like rim raft So say for example, you needed to delete a directory and you wanted that in your package Jason scripts You could have a command like this below Where it says no dash dash require built this doesn't exist. This is just just brainstorming but no dash dash require built in and you give it the the some identifier in this case rimmed her and then you pass the the the flags to it and you know that would be that would Be guaranteed to work in any shell and so that would be a great a great way to To to you do this in a cross-platform way do it with without requiring more of user lane modules There's some precedent for this Python ships with executable modules, so This command below Python dash M simple HTTP server So what this will do if you run it if you all you have to do is install Python, but if you run this command it will Create an HTTP server and serve the files in whatever directory you ran this in and so It has these like built-in modules that you can just run and that's cool I think node could do something similar I think it would be really beneficial for tooling and It would provide kind of a standard way to solve some of the problems that people are Encountering and again and again in their in their package Jason scripts Finally the last place I think We should look for improving the tooling situation is JavaScript itself and I would love it if people in the node Tooling group and others who are interested in no tooling could participate in TC three nine And so if you don't know what TC 39 is it's the team responsible for designing ECMAScript JavaScript the language right And so, you know if we are to have representation if we're gonna make sure that the language serves our use case We need to see at the table and so We really we really need some representation if we if we can't You know send somebody to the meeting certainly we should get in somebody's ear who's are who's going to be there There's a couple of proposals that that I identified as being ones that You know our art kind of interesting and I think could really be beneficial for tools That first one is the binary AST proposal and so the idea here is to make Web pages faster we want to ship less bytes and to ship less bytes Instead of shipping a text at JavaScript file. We can ship a binary AST and so This AST is an abstract syntax tree if you're if you're not familiar with the with the acronym but To me that sounds like a standardized AST of some sort and so the current situation in In the JavaScript community is that there is no standardized AST. There is a kind of community standard in the AST tree. I don't even know if that's a specification, but So it would really help tool interop and you know this this this AST format is binary format To be able to to have tools work with each other a little more smoothly So that one is probably a ways off, but I think it has a lot of potential The next one people have been trying to solve for a long time and you may know of or have used the Node.js built-in domains, which is deprecated But then there was never anything really to replace it. And so the idea there is a Realm and a realm is a distinct global environment So think the VM module where you say, okay, these are my globals and this is a script I'm gonna run go the problem with the VM modules it leaks it When you give it globals, you're not giving it a new set of globals. You're giving it your globals So it the the code running in this the script running in the VM is using those same globals that you have in your globals and so this leaks and so people have solved this sort of sandboxing problem in other ways But the idea here is to to offer a better way to sandbox code It may be even replacing what people use iFrames for now So back a long time ago. We used to use iFrames for something else entirely and nowadays We're still using iFrames, but for a different reason And so and there's a lot of use cases for tooling here including Like you know test frameworks and so maybe a test framework wants to isolate some code under test and make sure That it's not that we're not sharing an environment or stepping on each other's toes And so I would love to see realms get solved also if you're familiar with Angular and you know What a zone is realms are kind of similar to zones as well So that's kind of my spiel about the present and future of tooling and node and Again, my name is Chris Hiller. I'm a developer advocate at IBM and here are some links So you can hit me up Via email. I am bonescull at bonescull.com I am bonescull on github. I'm bonescull with a zero on Twitter and That final link is a link to the JSParty podcast there actually will be a JSParty podcast at This here conference later today Go and check out the schedule and check out JSParty. It's pretty awesome Again, my name is Chris Hiller. I'm happy to have had the chance to present here at OpenJS world 2020 Thank you very much, and I'll be around for Q&A