 at the University of Auckland, and have been working there for a number of years now, 11 years actually, and have worked in this industry for more than 16 years, and I've seen many things, as things have improved. Anyway, let's get it started. It is the year 2017 and war is happening. Python developers and their environments have been invaded by New Contender, and so anyone here actually use Node.js or MPM, raise your hands, that's quite a lot of you, doing web application development, I presume? Yeah? Okay. This talk is probably, you'll probably get what I'm about to say, but maybe you might have a bear about how to start by introducing a little bit of history. So, how did this start? So, JavaScript is the main runtime language for all web browsers. If you're launching a web browser, you have JavaScript, and it's there, but Python isn't JavaScript. So, if you want to get Python code running in your browser, you basically have to go through this transpilation step or run various fun working machines that you may have seen before, and that Node.js and MPM working together has created this really powerful ecosystem for getting JavaScript applications to run, oftentimes without a web browser, because they are on the server. So, people might ask, well, if Python isn't necessary, why don't we just run everything in JavaScript? Oh, that, well, Python, we really want Python, we really love Python, so we'll just have these Python microservices that run on some server somewhere, and JavaScript can speed front-end, it could be Angular or whatever, and there are a couple problems with that. Not a problem if you have a full application, but the thing is that Python backends become, generally become incapable of producing any kind of markup that search engines that are not Google will be able to understand. So, that is a death sentence for real websites. I work at the university, I have these publications that should be indexable by these search engines, and if they are not indexable by these search engines because they are in a blob of JSON, that is not going to do us any good. And the other thing is that there is zero cohesion between the JavaScript bits and the Python bits, because there are these two separate systems. And that means that how are we going to be able to convey these, our JavaScript needs from our Python packages to our dependence? There just isn't a good way to do that. But more history. So, when I first started, we have this thing 16 years ago, we have these things called a common gateway interface, and generally they are just random collection executable, probably Python scripts that would run on some server, that would produce a markup, and then that is the front end. Because that is all you get, that stays HTML over HTTP, get and post request. And of course, as time went on, people realized that they want more stuff, so we have these big, gigantic frameworks that does everything for you, such as Zope and Plon and also Django. Then they realized that, oh, man, these things are so hard and so annoying to use, and I don't understand. So people come along and start working on things like Web2py, Flask and Tornado, which are much smaller, more manageable frameworks for beginners to start to grasp. So the front end, as mentioned, is what the user sees. They generate something, they provide an interface to the back end, which could be some kind of SQL database. And as time went on, this has became not the front end, because the front end has become the JavaScript that is running in the client's browsers. And that JavaScript originally was created by Bren at Netscape, who thought, wow, browsers would be great if there is a little bit of interactivity, like popping up dialogue boxes to annoy your friends or doing actual real stuff like formalization or popping up ads. That's really annoying. But nowadays, it does a bit more than that. And back to a little more about the Python application stacks that we have come to know as we are moving on with the time. They are usually constructed from a collection of packages, but historically, there were just these big giant monoliths that comes with everything. Sometimes they came to be, oh, we are plugging architecture, but it doesn't change the fact that they were giant monoliths. They are really annoying to use. So they realize that. So Zopimplong, Django has done the thing where they have started refactoring the project in the smaller chunks. Soap has broken up their component extra texture. They have their interface. They are no longer in this one package, but they are separate Python packages now. So it makes it much more easier for a beginner to start working on some of these stuff, and it promotes software reuse. So now you don't need to feel like you have to take this entire big thing before you can start using a software. You can just take this one small library and then start working with it. And of course, with web application, you also got these other scripts and whatnot that are probably a Python package. And people consider this an asset management problem. And installation generally, no one really thinks about that, because with Python packages, you can install some package, and it's there. There are other systems such as build out, but most people don't really use that. And as mentioned, what about JavaScript code? So I'm not really talking about the stuff that you can get from NPM. I'm talking about your Python integration bits, like the configuration bits to get Angular going, for instance. The problem is that we don't really think about that, and we end up getting a bunch of special snowflakes, which are everywhere. I mean, you can say that every project is special snowflake, and that gets that kind of true. But then every Python framework, everybody, Django has its own way, they have Django static files, and it has its own JavaScript registry, and they have CSS registry, and they realize that this is a bad idea, so they refactor everything into a single resource registry. And Flask projects might use jQuery and friends, which they might get through some public CDN, and then they caught that. I'm guilty of this too, because I have some of Flask projects, and I remember using this table rendering library and dropped the JavaScript in there, and they kind of forget about it. And sure, they're simple stuff, they kind of work, but what about portability? What about if we are building real library builders, we want other people to be able to use a library easily, but without telling them how we actually got this thing put together, there's no way for them to do this without going through a lot of hoops. And then as these things are generally the way things have been done are generally not portable, makes reuse basically impossible. So they become liability, basically. It's no longer talking about assets and liabilities, and we only got technical debt now. There's no dependency management, because we just stuck the JavaScript code in there and forget about it. And they're also untested. Not many Python programmers write tests with JavaScript. Well, at least that was then. I hope things have changed now. Things have changed. And of course, Bower and NPM are a thing now, right? Well, that's how we get the invasion. And these are some examples from certain Python projects. On the left, right there, we got one Plong library called mockup, which provides a bunch of JavaScript code that does end up in Plong 5. And we got a standard. But we got these funny guys here, and we're like, Bower or JSON, package or JSON, other things that are really just in the faces of MPM that are not really communicate through the package management. You have to get to the repository to get it. Same thing with Django. This is the Django repo on the right-hand side. And they also have Brunt file in there, which is like, okay, that's like a build script. Well, to be fair, there's a big file over there, too. But they're not things that can be easily conveyed. And to be fair to Django, their package or JSON is really slim. They only really contain the ESLint, the basic tools. I think the same thing for the grant. But they also have this funny guy up there called js-underscore-test. So here's a question. If we decided, if I want to extend that, Django some functionality, how am I going to be able to easily run those tests against the sub-picture? I haven't broken anything during my extension process. And the other thing is that I mentioned these configuration files. You end up with these really cool test guides that have passed that are really specific to the source structure of the Git project, which is oftentimes not the same as the one that you have installed onto your system, within your virtual inf or on your distribution. So these things are generally not reusable. And how do you even know they are there while by Git clone, right? So there's no portability. Every single Python projects have used their own special Snowflake way of talking to NodeJ as an NPM. And it's just no way to reuse these kind of things. And as a personal anecdote, I decided I wanted to extend Plunk Walkup. So since none of the Python tools can do it, since all they use are Git, it's just a Git protocol, I have to, okay, I guess I'll just use Git some modules to embed that thing into my project and then go from there. And I end up having to use Shell Script to start copying their package of JSON, locating everything. And then, of course, add my own dependencies because I have other libraries I want to use. And don't even get me starting the big file. And it turns out that the karma.com.json, which is the thing for starting the test development server for getting a test running, also only customizing because it needs to know where everything is. And then this suddenly gets to the headache. I would even verify that things are even working. And also, most people in the JavaScript community don't seem to test their artifacts, which is kind of weird to me because that's the actual thing that user is running, but they just run the test on the dev and it's done. But I want to be able to test the artifacts. And to do that, I have to basically figure out all these tools, start writing a bunch of stuff. So instead of writing Python codes, some of you probably have seen this blog post from last year, so that's where I was learning like trying to like, okay, how do I be a programmer again? It's like, okay. So we get to do this. We get to write endless configuration files and like scripts that are just trying to get our stuff working. And instead of working on the thing that we should be, so this is what I look like. Okay? Like just ask my colleagues, they'll probably tell you that. But we are programmers. Why are we writing these reBOSS non-reusable copy-pasta configuration files, like spaghetti code, whatever you want to call it, to pass the spaghetti. And also, why haven't we program a system to manage and build these kind of things? To build these configuration files, actually run the thing itself. Instead, we program something to trap ourselves in doing that role. Like, computers are way better than us doing, then I just don't get it. So this is a great sentiment. Like, it's war. Let's fight against invasion. Like, you can watch me go mountain stage. And yeah, that's what's funny, because I want to include it there. So it is very easy to forget how to stay calm. And fortunately, I have a cure. ComJS. And gaze on the fluffy bunny rabbit. That's your eye bleach. So what is ComJS? So ComJS is a framework for Python that builds on top of setup tools which would then let, which would provide a good stable interface from there to know JS and NPM in order to provide a managed and well-defined integration so that we can actually use know JS in a more control manner from Python. And it's written to be modularized, extensible, and with a core focus in building, actually a reproducible way to build these deployable artifacts that would go on to the web server, which would then go to clients. And of course, the whole process sounds suspiciously like compile, assemble, and link like a C ecosystem. So that's, you want to turn an acronym, that's the acronym. And the other funny thing is that actually habits of rabbits is actually a funny way to describe, an app way to describe how JavaScript is produced and consumed in the modern day development environment. So that's what we're going to talk about in a little bit later on in this video. I'm going to talk about this in a little bit later on in this video. And you're suspiciously like another standard, right? You've probably seen this before. No, no, no. That's not my goal. Because setup 2 is already a standard. And I'm just merely extending it. Rather than coming up with something completely new like the Pong guys have done, or other people, actually, or do build something that basically is a standard. So that's the common ground. So that's not a standard. That's not a new standard. That's an existing standard that I'm reusing. And of course, ComJS is a library, so it does provide a common set of helpers that will make using these decorations much, much, much more easier manner. And setup 2 is in this utility is actually a lot of features in there. I took full use of it. So this is a sample configuration. Yeah, it's readable. And it's called JSON, which actually defines the package.JSON for the Python package. So all the Python dependencies are captured there, which can be accessible by other packages, the dependents, I should say. And a bunch of entry points, which for those of you probably know that's the next talk, but I'm not talking, so I hope it will be fine. And there is this other funny thing called extras underscore comJS. I tried to capture this using entry points, which you might probably see somewhere further down, but the problem is that the JavaScript figuring out where these artifacts are, or where the JavaScript code is, can still be very cumbersome. But at least I want, there must be a way to manually provide this. And this is my best attempt, but this is something I'm also working towards in solving. And the other aim of ComJS is to get out of the way. You want to use something and then just put it aside. So ComJS is designed for that. And that ComJS, as ComJS is also licensed under GPLs, you don't necessarily want to import from it, and then basically have the GPL be in your application for a philosophical reason that you want to avoid it, because entry points are just basically text files. So it's actually, the dependencies relationships actually got inverted. So you're not going to be using the text files or using the declarations in your application rather than the other way around. So unlike one of the other projects called FanStatic, which was an attempt to formalize the asset management in Python, but you can see that you have to import from that library in order to do anything. And a lot of declarations for one end every file and then it gets annoying. At least they did try. It's an interesting thing to do. So you have to import from within Python. And the other thing is that ComJS aims to be modular. So there is a set of common-based classes. And there are specific functionalities, again, defined through entry points. Okay. So I guess I can show you a demo now if this would work. Oops. I'm already in here then. Okay. So in here, I have a bunch of Python videos since I haven't actually released this on Pipe yet. So there's a development version. So I basically have to install it like so. Hopefully this will work. Let's take time because there's a really old notebook. And do I even have Internet? Nope. Try this again. There we go. Complaining about lack of Internet access. My apologies. If not. So I have another one that I already pre-installed. So ComJS, so you have a bunch of other modules that you could have. So let's back here. So in a bare installation, we basically have just the bare minimum stuff. So you can see that for modules that are not already installed, it does not show up. You're not forced to have all of these tools be available in your system in one go. And the other thing is ComJS aims to allow Python projects to include JavaScript. So sure, you might have a Python backend with some Angular stuff, but there are probably some kind of configuration file that you want to pass on to other developers that are aiming to extend your project. So that JavaScript has to go somewhere. So it's inside your Python, get repo somewhere. So in order to provide, so I want to aim to provide a consistent invocation method through the command line or the API. So here's a screenshot. I guess I can just try to get this demo going for you again. So if we want, we can just install the rest of the packages. So this is a processing. Normally, we just do this through pip, but again, it is not possible. So let's go in here, run setup. PY, develop. So we fetch the dependencies, and then if we run this, hopefully this would work. Oops, because I forgot to install it. Oh yeah, the other thing is I forgot to mention is that the Node.js, because I haven't installed any of the Node.js dependencies. So one neat thing is that I also provide the MPM integration. So if I want, I can just show the dependencies like so, but I also need a bunch of stuff. So if I want to use the webpack version, I can just have that installed, and then it will just concatenate everything together. And I'm just going to install the dev package for the testing, which I'll demo later. And of course, it will just go in, do the thing, go through MPM, download the dependency, and I'm not going to wait for this, so I'm just going to go back to here and then, what's wrong with directory? So I do actually tell you, have log messages that will tell you what's wrong, and now that's running, there you go. Here's the artifact. So that's all of the random artifacts that I've pre-created from beforehand. And the other thing is that I want to provide something that makes it easy for people to run tests against the JavaScript that you may have in your Python packages. So that Python programmers can actually start running tests much like when you might test an application using unit tests, like Python, TackView, unit tests, and some of your package test suites. There's no such thing really for JavaScript with Node.js, so I want to be able to provide that at least for the Python programmers. So we have something like this. So you saw how I created the artifact previously. So if I run, so this would just create the artifact through the required JS to a chain, but what if we want to run a test? So it's a stick, another command in the first. So comma is a standard test runner for JavaScript, Node.js development, and we do that, and it should start invoking the test. And there you go, pass all the tests, and then once it's done, now you actually go through and build your artifact, rather than, and of course, coverage is a thing I forgot to show. Probably should just put in the tech that we fled, so I can stop hitting that. And so you can see the coverage. You can also enable the other flags to enable coverage for your tests, so you make sure that you actually run your test, rather than them not really running. And then the other thing is that since we are really doing this for Python, there must be a way to include all of these artifacts. As part of the final Python wheel, which would then be distributed to the end user, because as I mentioned, they don't necessarily want to have com.js installed. They don't want to have Node.js installed, really. So why do we have to force them to do that? So if we have a way to formalize the building of these things, again, this can be achieved through setup tools, which I've done. So it would look something like this, when you run Python setup, you know, be this wheel, and then you have the ability to automatically get this done. So one thing I probably should show is that, so this is the, whoops, I hope it's working. Yeah, so these are the relevant sections. So define the file name, for the artifact, the specific entry point that will produce the tool chain required to get the artifact generated. Again, all these things can be customized because they're just entry points. Very easy to do. Of course, not the coding part, but definition at least, make it so. And whoops. So currently, version 2.1 is out. That's been a year. I'm working on the next version, which actually involves a lot of API changes because I blame Webpack. Webpack is actually really painful to deal with. Hopefully soon. And so all the other project packages also require a little bit of refactoring. So anyway. So instead of finding a war, peace can finally be achieved. So we should be able to disarm the configuration files, which are very specific and possibly toxic, and turn them, transmute them into some kind of conveyable form that other packages can digest. And then the entire package workflow is completely streamlined and completely reproducible. You can just make sure that your environment, your Node.js environment is installed, as mentioned, dependencies are all there. And there's no longer, I need to worry about these foreign invaders that are in your Git repo when you can just give them a path to turn them into useful fellow citizens through proper immigration channels. Right? Thank you for your time. No, we got plenty of time. Yeah. This might be too big of a question to ask right now, but I, you know, I'm not an expert at Node or anything. And I guess I was just wondering if you could explain a little bit more about the syntax of the entry points and how that relates to the Node module and allowing you to import it. Yeah. This is a point. You're right. This is pretty technical. I don't want under the hood that I ended up having to really gloss over, but again, if you and I can chat later or anyone else for that matter, because I'll be around. So if you really want to know how I got the library, because I actually do have Nunger, which also have a library that uses that. And I bring all these together. I hope that's a good enough way to demonstrate this particular use case. Yeah. Other questions? Hey, I don't know too much about this, but I was just wondering if you'd explored going the other way around. So, you know, using JavaScript tools to, you know, deploy Python. And, you know, is that a more tractable route? That can be. There is actually, I think there are projects that are kind of doing this, but I personally find the Node.js MVM ecosystem to be really fragmented. Like, there is just no, nothing like set up to those entry points there. Like, I mean, there are Autosystems similar, but nobody seems to have agreed what things really are. And also the ecosystem for Node.js have a lot of churn. Things just change all the time. I gave up, basically. I think there's one at the back. Anyway, you can come talk to me if you really want more details. So, yeah.