Saturday 1:30 p.m.–2:10 p.m.

Using Node.js/npm with Python/setuptools - a lesson on staying calm.

Tommy Yu

Audience level:
Intermediate

Description

Python programmers are finding their web application development environment besieged by an invader: Node.js/npm. War against them seems like an enticing option, but it only brings discord and misery to the environment. This talk introduces a mediator that seeks to bring coexistence through enhancements of existing foundations and disarmament, so all Python packages can enjoy the resulting calm.

Abstract

I have been a Python web application developer for eleven years, and I have seen the landscape change over the years. Most interesting change is the maturing JavaScript/Node.js ecosystem and how this end up being a hugely disruptive thing. Many Python web frameworks and developers have came up with their own way of dealing with this, but often in ways that are mutually incompatible because they based it on something that is not common across any of the frameworks the have used. Plone has their own methodology, so does Django, and never mind every other small time developer who have their own sites on microframeworks like Flask, Web2Py or Sanic. The end result is that many Python developers are struggling independently against Node.js and NPM and all the associative frameworks, and never have a cohesive way to tackle this together.

The problem is of course deeper. Example: some Python developer realised that putting their JavaScript code on npm will reach a wider audience, and end up having two packages. Naturally, their shipped wheels will have the compiled webpack or AMD (requirejs) artifacts which their Python package will use, however if any other downstream users of that Python package use it as a library and want to extend the functionality, the JavaScript side of things may end up being absolutely inscrutable. Or that the downstream users have their own way of setting Node.js environment up but then adding that new package in a way that makes it play nice with their Python package integration may be difficult. Or not really, but end of the day is that traditional methods of doing things in Node.js with grunt/gulp and the like involves hand editing a lot of files, and combining with how Python works, especially with multiple virtual environments, dealing with relative/absolute paths can get out of hand very quickly.

Never mind the inability to share/run tests like how Python programmers can do python -m unittest some.package.make_test_suite. No such things exists for Node.js, especially in a way that also plays well with Python.

So last year or so I was working on a templating system that uses Jinja2 templates in a way that also allow client-side rendering to work (using mozilla's nunjucks library), however dealing with the bridge between these two languages got me thinking about how to deploy my templates and JavaScript hooks with Python packages in a way that is reusable (and tested), and all the solutions that were out there were either proprietary with one platform (and it doesn't work really well) to some very convoluted thing that requires as much effort as hand-editing configuration files, or worst, introduces even more dependencies (armaments). Still, none of those solutions allowed the sharing and propagation of the configurations I wrote for building of the AMD/requirejs (eventually webpack) artifacts down to downstream packages so they can add their own extensions (like their templates) to. So I ended up spending a lot of time thinking and building this in a way that doesn't introduce yet-another-complete-new-standard (https://xkcd.com/927/) and in a way that every Python package can use. Naturally this foundation ends up being setuptools.

The system that I ended up building is something that makes use of the entry point system as a registry (a valid use case) for the declaration of modules and hooks for JavaScript files that are to be exported as JavaScript/Node.js packages, so that tools can be built to translate them into a usable form for Node.js tools (like require.js and webpack) to understand and build it with the packages from npm. Naturally, this will require that a package.json (setup.py analogue in npm) be built, so I also extended setuptools to support the creation and management of this file for Python packages, so Python packages can ship a package.json which can then be composed together with package.json from a different Python package.

Naturally, I realised that all of that should not just live in my templating system, but it should be split out into its own thing, so calmjs was created as the foundation for the integration with Node.js. The next thing was to create an artifact build system (well, integration with an existing one) and requirejs was the logical choice for me at the time, because I had planned to initially target Plone which had requirejs to manage their JavaScript modules in AMD format. calmjs.rjs was created, along with calmjs.dev for development and testing purposes (so JavaScript code in Python packages can also be tested in a way similar to running python -m unittest).

Of course, the Node.js system moves very quickly, and webpack had ended up being completely dominant, so I also ended up building calmjs.webpack which is currently in progress, but I hope to have at least 1.0.0 release by end of November as I really need to get back working on the original thing I had set out to do more than a year ago.

Since this is a Python conference, I do want to spend some time in my talk to cover the integration with setuptools bit, such as talking about what the things in pkg_resources are, what are distribution and entry points at least briefly as that is the existing foundation that I have used for the system I've built, in a way that avoided introducing too much completely new and proprietary things.