(Re)simplifying front-end build process with a build service

Ever thought it would be a good idea to have a build process as a service, so that doing SASS, browserify, minification and caching of front end dependencies is as easy as writing a <script> or <link /> tag? We did, so we built one.

We are in the midst of developing a new way of building websites for the FT, which we call Origami, and because we didn’t want to build anything monolithic, or yet another framework, it involves a lot of small components. We also have an eye on the implementation of support for Web Components, so when it becomes practical to use those, we’ll be able to turn our components into Web Components.

This does, however, leave us with a problem. We have a lot of websites, and most either don’t have a build process, or have a process that’s incompatible with the build pipeline that we have standardised for Origami components (Bower, SASS, Browserify with debowerify and brfs, and Closure compiler). And even for new sites, we have teams building using a wide variety of server-side technologies some of whom are not familiar with the front end build tools which are mainly coming out of the NodeJS community.

We needed something that would allow us to jump start a project without setting up a build process: something we could use in hack days or legacy sites. Something that would auto-upgrade modules when new components were released. Something that would let us write a script and link tag into the head of an HTML page and be done with it:

<script src='http://build.example.com/bundle/js?modules=module1@^2.3.5,module2,module3@3.4.0'></script>
<link href='http://build.example.com/bundle/css?modules=module1@^2.3.5,module2,module3@3.4.0' rel='stylesheet' />

Existing solutions

We started the project not aware of anything else that solved this problem, but since then Steve Souders made me aware of Yahoo’s Combo handler, which Yahoo! has been using since 2008 to concatenate together multiple portions of the YUI framework into a single HTTP download. Tom Ashworth also created Pldn.io, which does a similar thing for anything hosted on cdnjs.

But this isn’t enough. It’s concatenation of pre-built libraries, which means you run the risk of duplicating dependencies (say you require two components that both contain a copy of Underscore or Backbone), and it doesn’t do any packaging of the resulting bundle.

Then there’s browserify CDN (and RequireBin, which is powered by it), which is more promising – it allows you to create bundles from a list of npm modules, which it then installs using npm and builds using browserify. This is pretty nice, but still falls a bit short of where we wanted to be:

  • it doesn’t do anything for CSS
  • being limited to npm means it can only package up public modules
  • fetching multiple packages in one request requires a POST and returns a data response rather than the raw bundle, so you can’t use it in a <script> tag.
  • Issuing an ID for a bundle ties the service to a single shared persistent data store, reducing the scope for it to be a shared-nothing architecture for maximum resilience.

These aren’t necessarily failings of Browserify-CDN, just aspects that make it challenging to use directly for our use case.

The Origami build service

So we built our own. To make things simple, we started with a standardised build process, which made some demands of components:

  • CSS must be written in SASS
  • JavaScript must be written as CommonJS compatible modules
  • Each component must have a single ‘main’ file for each language, and declare it in a bower.json file

The Build service resolves module names to Git repos using the public bower registry and our own private bower registry, which allows us to specify Origami components using short names even though they’re not in the public registry (though most of them are public, on our GitHub account, so feel free to go have a look if you want).

Here’s an example, which you get by loading the URL https://build.origami.ft.com/bundles/css?modules=o-grid:

That should show the source of the o-grid component, our CSS grid. As I write this, we’re on version 2.0.7, but, because I didn’t specify a version number in the modules parameter, the code above will change every time we tag a new version of Grid.

Granular upgrades

Because we defer installation to Bower, we get Bower’s built-in Semver support for free. That means a developer can choose to be as risk-tolerant or averse as they like when embedding build-service resources in their code. The following build-service paths will all load the same module, but with different rules on when to allow upgrades:

URL Would load…
/bundles/css?modules=module1 most recent tagged version
/bundles/css?modules=module1@1.2.3 locked to version 1.2.3
/bundles/css?modules=module1@~1.2.3 latest which is >= 1.2.3, < 1.3

Generally speaking, the best strategy is to allow upgrades up to, but not including, the next major version bump, because that’s where you can expect breaking changes to the component. That said, some developers prefer to be more specific than that, and others (especially where they are not integrating any of their own code with the component) are happy to not constrain the version at all.

We can’t yet track a branch, but we’re thinking of adding that.

Caching and availability

Obviously, once a bundle has been built, it will be cached, and next time the same bundle is requested, the service will be able to return the built version quickly. We can also reuse existing module installations if we’re building a different module set but which contains some or all of the same modules that we’ve already installed in order to build other bundles.

If the build service server doesn’t have a cached version, it returns a 202 Accepted response and starts building the bundle asynchronously. That means we can also put a caching load balancer, which we deploy on a global CDN, in front of multiple build service nodes, which all operate independently.

Linking this back to Origami in general, the Build service is an example of how we are preferring tools over frameworks, and decentralisation over monoliths. Time will tell if this is a good strategy. We’ll let you know!