While I’m working on shelf (the database for javascript developers), I found myself fighting build tools.
You know the feeling, you are writing the config file for webpack or rollup and you are trying to figure out the right settings and plugins for your specific use case.
But the docs aren’t helping and the tutorials are outdated. So you’re left with a frustration session of hit and trial.
I was trying to create a UMD version for shelf, the one you can drop into your HTML as a <script> tag. I’m using rollup because it’s what the cool kids are using these days.
It didn’t work. This is what my dist.js looks like.
;(function(global, factory) {
// boring module bootstrap code
})(this, function() {
'use strict'
// wut const axios = require('axios') const methods = require('./methods')
const api = axios.create({ timeout: 5000 })
const shelf = { ...methods(api) }
module.exports = shelf
})What are these require statements doing here? These should have been bundled in.
While this works in a React project with webpack, it wouldn’t work when imported directly in the browser with a <script> tag.
I followed the docs, copied a few settings from other libraries, but nope.
Debugging builds are harder than debugging your application code because you can’t really add debugger or console.log in your code to find out what’s happening. It’s a black box, config in, bundle out.
There are multiple places where things might be going wrong in my project. It might be my setup with yarn workspaces, or maybe I’m using the wrong plugins, maybe it’s the way I’m writing my code?
Suspect #1 - Yarn workspaces / Dependency resolution
To talk about yarn workspaces, we need to talk about dependency resolution.
When you write the following statement in a file:
// server/index.js
const express = require('express')
const lodash = require('lodash')The nodejs runtime tries to resolve the dependency by recursively going up the tree to locate this dependency in a node_modules folder.
For the above code, both express and lodash will be resolved from the nearest node_modules up the tree.
project
│
├── package.json
│
├── node_modules/ │ ├── express
│ └── lodash
│
└─── server/
├── package.json
└── index.jsFor a heavily nested structure, it will keep going higher up the tree to resolve the package.
project root
│
├── package.json
│
├── node_modules/ │ └── lodash │
└── packages/
├── server/
│ ├── node_modules/ │ │ └── express │ ├── package.json
│ └── index.js
│
└─── client/
├── package.json
└── index.jsFor the same as above, express is resolved from the server’s node_modules, but lodash is resolved from the parent folder’s node_modules.
Yarn workspaces
Now, imagine you are building a small cache module that lives as a standalone package. You want to use it with your server as a dependency but it isn’t published yet - you want to use the development code.
This is where workspaces come in. Yarn workspaces lets you link local dependencies that depend on one another. To make this possible, it installs all your dependencies at the project root level and creates symlinks for local packages.
project root
│
├── package.json
│
├── node_modules/ │ └── lodash │ └── express │ └── cache → packages/cache │
└── packages/
├── server/
│ ├── package.json
│ └── index.js
│
└─── cache/
├── package.json
└── index.jsSo, now when you require('cache') from your server, it gets resolved up the tree to the node_modules in the project root (which is a link to the cache folder).
What does this have to do with bundling?
Your build tool - webpack/rollup/etc. uses the same dependency resolution while creating your bundle.
So, as long as yarn set up the symlinks are setup correctly, your bundle should compile. You can test this by running ls -l in your root node_modules folder.
drwxr-xr-x sid express
drwxr-xr-x sid lodash
drwxr-xr-x sid cache -> ../packages/cacheThat looks about, right. But, if that’s not the problem, what is?
Suspect #2 - Am I using the right plugins? / Debugging the build
This step is really hard because you’re dealing with a black box - especially when you don’t get any errors, but also don’t get the desired result.
My favorite technique to config is to create the smallest possible test outside my current project.
It’s really hard to pin-point the problem in a big project, but if you start with the smallest “hello world” project that works, you have a passing test.
Now, you add one complexity at a time to bring it closer to your complex project. When it starts breaking, you have found the potential problem.
Find a fix to this problem on the small test and then jump to your project and put the fix in.
If it starts working, great, you’re done! If not, keep adding complexity to your passing test.
The goal is narrow down the problem from both ends.
Okay, let’s create the passing test first
// index.js
const shelf = 'hi'
module.exports = shelf// dist.js
;(function(global, factory) {
// boring module bootstrap code
})(this, function() {
'use strict'
const shelf = 'hi'
module.exports = shelf
})Alright, let’s test this out in the browser with a small html file
<html>
<script src="dist.js"></script>
<script>
console.log(window.shelf)
</script>
</html>ReferenceError: module is not definedThat broke fast! That module.exports isn’t supposed to be there!
Quick read of the docs tells me that rollup prefers ES6 Modules (import, export) instead of the old CommonJS format (require, module)
Let’s quickly fix that in our small test to make it pass again.
// index.js
const shelf = 'hi'
export default shelf// dist.js
;(function(global, factory) {
// boring module bootstrap code
})(this, function() {
'use strict'
const shelf = 'hi'
return shelf
})Tested it in the browser and it works 👍.
Let’s take this to our complex project and see if that works. Nope, still broke 😅, there’s more work to do.
Let’s add one layer of complexity to our small test - add a local import.
// index.js
import methods from './methods'
const shelf = 'hi'
export default shelf// dist.js
;(function(global, factory) {
// boring module bootstrap code
})(this, function() {
'use strict'
function methods(api) { // boring code }
const shelf = 'hi'
return shelf
})Great, that works! Now, we know that’s not the problem. Next step, let’s add an external dependency:
// index.js
import methods from './methods'
import axios from 'axios'
const shelf = 'hi'
export default shelf(!) Import of non-existent export
index.js
1: import axios from "axios";
^🤨 “Import of non-existent export”?
Errors are good, error means you have something to look for. Searching for the error gives a few stackoverflow answers but copying code from their didn’t work.
Another strategy I like is looking at other libraries. Copying their config might not work (different rollup version, deprecated settings, etc.) but it still helps because it tells what are the right words to search for.
I found about rollup-plugin-commonjs from React Router’s rollup config.
Convert CommonJS modules to ES6, so they can be included in a Rollup bundle.
Not exactly helping but gives me something to search for - ” axios rollup-plugin-commonjs”.
Jackpot! Turns out the library I’m using (axios) doesn’t use the ES6 format. It uses CommonJS!
Now, I can either use a different library that uses ES6 modules or use rollup-plugin-commonjs to convert it.
Adding this to the list of rollup plugins fixes the build! Now let’s take this to our complex project to find out if this was the problem.
$ rollup -c rollup.config.js
index.js → dist.js
created dist.js in 287ms
✨ Done in 0.58s.Yep! It worked!
Turns out the big problem all along wasn’t yarn workspaces or the way shelf is structured but the fact I was using a commonJS library in a project otherwise using ES6 modules 😅
There is no way I would have figured this out if I kept poking the config in my big project. (I didn’t even know these formats were incompatible until just now!)
To summarize, don’t keep banging your head on your failing build. Start with a passing build and try to narrow down the problem from both ends.
Hope that was helpful in your journey
Sid