Ember and Yarn Workspaces

Square Dashboard circa 2012 versus Square Dashboard in the near future

Building so much functionality into a single application allowed us to move quickly with new products and bug fixes. Until late last year, we didn’t have another option—we were using a bespoke Rails-based build pipeline that didn’t understand Ember applications. But the size of the application is a heavy burden: build and CI times are very slow, and the codebase is intimidating to new and seasoned developers alike.

We finished migrating to Ember CLI on December 1st, and we’re starting to break up the monolith into Ember-specific NPM packages called addons and engines for two reasons:

• Addons and engines allow us to modularize the codebase and create boundaries between logical domains, making ownership clearer and allowing us to share code between teams without creating a rat’s nest of dependencies.

• Engines support lazy-loading code, meaning our merchants won’t download code for a section of the application until they need it.

However, breaking up the application is not a panacea. If we simply moved code from the application into separate NPM packages like this:

A working directory with three folders: “dashboard”, “sales-reports-engine”, and ”shared-ui-addon”. The latter could contain components and utilities used by both the app and the engine.

… and install the addon and engine into Dashboard like this:

{
  "name": "dashboard",
  "devDependencies": {
    "sales-reports-engine": "^1.0.0",
    "shared-ui-addon": "^1.0.0"
  }
}

… developing code simultaneously across packages would be quite painful. By default, NPM and Yarn would copy files between packages. If you change a file in sales-reports-engine and want to see it reflected in dashboard, you’d have to completely reinstall the engine package.

#headdesk (from https://giphy.com/gifs/fml-headdesk-frustrated-cvHSpGveXf5rG)

What we wanted was a way to break up the app without hampering the development experience for everyone—especially as we encourage engineers to spend most of their time developing addons and engines, instead of adding to the main application.

npm link

One solution would be to use npm link, a tool to symlink packages together. (yarn link does exactly the same thing.) Using it is fairly straightforward:

**>** cd sales-reports-engine
**>** npm link

success Registered "sales-reports-engine".
info You can now run `npm link "sales-reports-engine"` in the projects where you want to use this module and it will be used instead.

> cd ../dashboard
> npm link sales-report-engine

success Using linked module for "sales-reports-engine".

After running those commands, you’d see a symlink to the engine folder in the dashboard package’s node_modules folder:

The “dashboard” package now has a symlink to the “sales-reports-engine” code in its node_modules folder.

This structure allows Dashboard to access files directly from your working copy of sales-reports-engine during development.

However, this is insufficient if you want to see changes reflected across all three packages at once. What you really want is:

Tying everything together requires symlinks all over the place.

This arrangement would take numerous npm link commands to construct, especially once we have more than three packages. And personally, I could never reliably get npm link to work—the tool works correctly, but I kept swapping the commands or forgetting whether I had run them already.

Fortunately, there’s a better way! 🙌

Yarn Workspaces

Support for workspaces arrived in Yarn in August, 2017, and we’ve found the feature to be stable and easy to use since version 1.3.2. And as of Ember CLI 3.1 (now in beta), workspaces and Ember are best friends!

To set everything up, you’ll need to move your packages into a “workspace root” with its own package.json file:

The “packages” folder could be named anything, but it’s a common folder name in other monorepos like Babel and React.

The contents of the workspace root package.json are very simple:

{
  "private": true,
  "workspaces": [
    "packages/*"
  ]
}

Now when you run yarn install anywhere inside the workspace root, Yarn will discover the dependencies between packages and hoist symlinks up to a top-level node_modules folder:

Yarn workspaces organize all the necessary symlinks in a top-level node_modules folder.

There is no step #2! Most engineers working on Dashboard probably don’t realize this is even happening.

If you’d like to know more about how the packages find their dependencies in the top-levelnode_modules folder, the Node docs explain the lookup algorithm.

What about in-repo addons?

Ember CLI’s original solution for simultaneously developing addons is in-repo addons, which have never required any npm link shenanigans. We have a few reasons for not using them:

• In-repo addons cannot have their own test suites. Test files go into the host application. We want our tests to be as decoupled as our application code.

• We want to be able to publish certain addons for use by other Ember applications at Square. In-repo addons are not real NPM packages so they can’t be published on their own.

• Similarly, we want to allow teams to escape from the monorepo in the future if it makes sense for their product. Once their code is in a “real” addon with its own dependencies and test suite, they can move the files to another repository without much effort.

NB: Naming is hard—our “real” addons are inside the same git repository, so aren’t they “in-repo” addons? 🤯 To disambiguate the two patterns, we use the term “monorepo addon” for addons in our Yarn workspace.

Are there any gotchas?

Not really! You must be on Ember CLI ≥ 2.18 for Yarn workspaces and Ember to work at all (Edward Faulkner’s commit has a great explanation for why that is).

If you’re on Ember CLI ≤ 3.1, commands like ember install will default to using NPM, but this is remedied by adding the --yarn flag:

ember install ember-animate --yarn

(Our very own Timothy Park landed the patch to fix this in 3.1!)

Migrating to Yarn Workspaces

You’ll have the best luck if you follow this three-step process:

• Move your existing yarn.lock file from the original application folder up to the Yarn workspace root.

• Create the new package.json file next to the lockfile as described in the Yarn docs.

• Run yarn install.

If you don’t move the original lockfile, Yarn will create a new lockfile and upgrade all your transitive dependencies. This could lead to unexpected breakage.

Switching our application to Yarn workspaces is one of the most quietly impactful decisions we’ve made in our journey to modernize the codebase. Instead of one gigantic application, we now have a (slightly) smaller app and over a dozen well-organized addons and engines that are much nicer to work with. If you have a monolithic application that may start to buckle under its own weight, I highly recommend giving Yarn workspaces a try.

P.S. One extra pro-tip: yarn upgrade-interactive and yarn outdated work fantastically inside of a Yarn workspace. 💯