My Learning Curve

Feed / Mark's Dev Blog

Blogged Answers: My Experience Modernizing Packages to ESM

Tuesday, August 8, 2023

Table of Contents

Introduction

For the last 8+ years, the JS ecosystem has been undergoing a slow transition towards using ES Modules ("ESM") as the default approach for publishing and using JS code. Similar to the Python 2->3 transition, this has been incredibly difficult and painful to deal with.

As a package maintainer, I want to make sure that my libraries are maximally compatible and usable in the widest array of environments I can feasibly support. Unfortunately, this also means that I've had to become familiar with the nuances and behavior quirks of a variety of different build tools and runtime environments

Early this year I started working on trying to update the package formatting for the Redux family of libraries to give them "full ESM compatibility". I think I've finally come up with a set of configurations that seem to work reasonably well, but it's been a struggle.

One of my biggest frustrations is that there is no single authoritative and comprehensive guide on "How to Publish a JS Package Correctly". I've repeatedly begged for some expert who actually knows what they're doing to write and publish such a guide. Ideally, it would cover things like what file formats to include, how to configure ESM/CJS interop and package.exports, dealing with TS types versions, ensuring tree shaking, checking for compatibility issues, how to properly support specific build tools and what they look for, and so on. I have found some guides (which I'll link below), but nothing that quite matches the breadth and contents I've been wishing for.

This post is not that "authoritative guide". It's a recap of what I've tried, and hard-earned lessons I've learned along the way. Based on the number of times folks have popped up and said "you're doing this wrong", I'm sure there's plenty of pieces I'm still missing :) But, I hope this information will be useful and informative even if it's not fully comprehensive and authoritative.

There's plenty of articles out there already recapping the history of the ESM spec, the arguments and decisions that have led to the current confusion and compatibility issues, and how we got into this mess. In the interest of keeping this a somewhat manageable size, I'll try to dig up links to a few of those and list them at the end, and focus primarily on my own experience and steps throughout this process.

Redux Packages Background

Packages and Configurations

At the end of 2022, I maintained and published these packages as part of the Redux org:

Each of these packages had their own development history, packaging setup, and build configuration. In general:

  • All the packages included ESM, CJS, and UMD build artifacts (with varying combinations of embedded process.env.NODE_ENV values, or pre-compiled to "development" or "production" versions)
  • All build artifacts used a .js extension
  • All the packages were being transpiled to ES5 syntax for IE11 compatibility
  • redux, react-redux, redux-thunk, and reselect were being transpiled with Babel and bundled with Rollup. RTK was built using a custom ESBuild wrapper script that did the bundling and primary transpilation, but also used tsc to lower ES2015 code to ES5.
  • All of the packages used the "main", "module", and "types" fields in package.json. None of the packages used the relatively new "exports" field for defining which build artifacts get loaded.
  • Most of the packages except RTK output build artifacts to different folders by type: dist for UMD, lib for CJS, es for ESM

Some examples from those package.json files:

{
  "name": "redux",
  "version": "4.2.1",
  "main": "lib/redux.js",
  "unpkg": "dist/redux.js",
  "module": "es/redux.js",
  "typings": "./index.d.ts",
  "files": ["dist", "lib", "es", "src", "index.d.ts"]
}

{
  "name": "react-redux",
  "version": "8.0.5",
  "main": "./lib/index.js",
  "types": "./es/index.d.ts",
  "unpkg": "dist/react-redux.js",
  "module": "es/index.js",
  "files": ["dist", "lib", "src", "es"]
}

{
  "name": "@reduxjs/toolkit",
  "version": "1.9.5",
  "main": "dist/index.js",
  "module": "dist/redux-toolkit.esm.js",
  "unpkg": "dist/redux-toolkit.umd.min.js",
  "types": "dist/index.d.ts",
  "files": [
    "dist/**/*.js",
    "dist/**/*.js.map",
    "dist/**/*.d.ts",
    "dist/**/package.json",
    "src/",
    "query"
  ]
}

RTK's setup was more complicated, because it has 3 separate entry points: @reduxjs/toolkit, @reduxjs/toolkit/query, and @reduxjs/toolkit/query/react. Note that RTK's package.json didn't list the two RTKQ entry points. Instead, there was an actual /query folder in the published package, with /query/package.json and /query/react/package.json files that in turn pointed over to the right artifacts in the dist folder. (This setup was the result of considerable experimentation, aka "it seems to work with Webpack and a couple other tools I think???").

While not directly relevant for the rest of the story, I'd like to give a shout out to two highly useful tools that I use as part of the publishing process:

  • release-it: automates the actual steps for publishing to NPM, including Git tagging and pushing
  • yalc: lets you do a full local "publish" of a package so that you can test out installing it into example projects. Avoids issues with symlinks (ie npm link), and tests out the real build and publish steps.

Issue History

In mid-2021, we received an issue reporting that RTK could not be loaded properly in both client and server code simultaneously. In early 2022, a similar issue reported that RTK could not be correctly imported in a .mjs ESM file, due to use of "module" but no "exports" field in package.json. Finally, another issue noted that RTK didn't work with TypeScript's new moduleResolution: "node16" option.

I'd previously asked around about the implications of adding "exports" to a package, and I'd been told that "this qualifies as a breaking change". That meant that I couldn't begin to consider doing it until the next major release for each of the packages. But, I had no idea when we'd get around to publishing majors. Redux 4.0 came out all the way back in 2018, and RTK 1.0 in late 2019. React-Redux 8.0 was more recent, in mid-2022.

The Redux core had actually been converted to TS in 2019, but we'd never shipped it. 4.x and its hand-written typedefs worked, and we had concerns about potential ecosystem churn from shipping a 5.0 major. We also had plenty of feature work to do with RTK.

We shipped RTK 1.9 in November 2022. After taking a couple month break, I finally sat down to start seriously working on trying to modernize our packages.

Early Attempts

I'd been stockpiling a list of tabs and articles around "publishing modern JS", ESM, and ESM/CJS interop for the last few years, knowing that this day would come eventually. (At last check, I had roughly 175 articles in that list!).

I reviewed several of those articles to figure out my initial steps. From what I read, I concluded that:

  • I needed to add "type": "module" to my package.json files, in order for Node and bundlers to detect the package as containing ESM files
  • I also needed to add the "exports" key to package.json, and add keys inside that listed the possible entry points and what build artifacts to use when imported in different environments

I'd seen mentions of using .mjs as a file extension to force Node to recognize a given file as being an ES Module. To be honest, I felt that looked ugly and ridiculous, and I did not want to use that extension at all.

My first attempt was RTK PR #3095: Migrate the RTK package to be full ESM. Per my PR description, it contained these changes:

This PR attempts to convert the RTK package from its existing "contains ESM and CJS modules, but not fully ESM", to be fully ESM with {type: "module"} and still support CJS:

  • BREAKING: Sets the main RTK package.json file to be {type: "module"}
  • BREAKING: Updates all entry point package.json files to use exports to point to the types, ESM file, and CJS file
    • I still have main and module in there because WHO KNOWS WHETHER THOSE STILL END UP GETTING USED BY SOME TOOLS OR NOT 🤷‍♂️
  • Updates the build script:
    • Fixed ESM compat on execution by replacing use of __dirname and fixing the Terser import
    • Switched all build targets to be "esnext", to ensure the output is untouched other than TS transpilation
    • Moved all CJS build artifacts to be nested a level deeper in each entry point in a ./cjs/ folder, ie, ./dist/query/cjs/
    • Added {type: "commonjs"} package files to those folders
    • Turned off the UMD build artifacts for now

The resulting package.json looked like this:

{
  "name": "@reduxjs/toolkit",
  "version": "2.0.0-alpha.1",
  "type": "module",
  "module": "dist/redux-toolkit.modern.js",
  "main": "dist/cjs/index.js",
  "types": "dist/index.d.ts",
  "exports": {
    "./package.json": "./package.json",
    ".": {
      "types": "./dist/index.d.ts",
      "import": "./dist/redux-toolkit.modern.js",
      "default": "./dist/cjs/index.js"
    },
    "./query": {
      "types": "./dist/query/index.d.ts",
      "import": "./dist/query/rtk-query.modern.js",
      "default": "./dist/query/cjs/index.js"
    },
    "./query/react": {
      "types": "./dist/query/react/index.d.ts",
      "import": "./dist/query/react/rtk-query-react.modern.js",
      "default": "./dist/query/react/cjs/index.js"
    }
  }
}

I did try testing out local builds in Vite, CRA4/5, Next, and Node, as well as running the publint tool. Things seemed to mostly work locally, so I put up the PR to see what would happen with CI

AND EVERYTHING EXPLODED!!!! 💣💣💣

I spent a few more hours fiddling with things, and reported my findings:

Well. The good news is I think the runtime code works.

Bad news is Jest is being a pain. In particular, something about the way it's importing redux-thunk makes the default import an object like {default}, which is not a middleware function, and thus the tests explode when we try to create a store.

I spent the last couple hours hacking around with the thunk exports and republishing it locally. Switching the thunk package over to not having a default export at all sorta helped, but now something about the "no dev middleware in prod" test is failing.

So, close, but can't even build this branch yet.

edit

where I left off yesterday was that:

  • redux-thunk has a default export, and Jest was now choking on that
  • I published redux-thunk@3.0.0-alpha that tried to convert it to ESM, but still had a default export included. That helped build with Next, but not our local Jest tests
  • I then did a local-only publish of redux-thunk that dropped the default export and only had named exports. That actually seemed to help, but one of our RTK tests that asserts NODE_ENV=production behavior for getDefaultMiddleware was breaking

Some folks in Reactiflux suggested that it might be worth investigating a switch to Vitest. I'd prefer not to do that if possible, on the grounds that migrating to a different test runner is not my priority or something I want to spend time on. On the other hand, it could also be something that would be beneficial in general and for this specific problem.

Migrating to Vitest

I really didn't want to burn time trying to migrate our entire test setup from Jest to Vitest. But, I'd heard plenty of positive comments about Vitest, including that it ran significantly faster and had better ESM support.

A couple days later I decided to give it a shot. To my pleasant surprise, the conversion was fairly straightforward.

I ended up with RTK PR #3102: Migrate RTK test suite from Jest to Vitest.

The main test setup made sense, and the jest.fn() -> vi.fn() swaps were straightforward. The biggest pain point I ran into was where we tried to mock the redux package to assert that configureStore was calling through to the core library. Had to do a bunch of fiddling with vi.mock() until something finally seemed to work. On the other hand, timer behavior seemed to work more consistently.

As part of this process, I found myself also needing to convert other auxiliary files in the repo to ESM syntax, such as Jest config files and build scripts with a .js extension.

I was able to get that PR passing, and merge it a couple days later.

Initial Alpha Testing

I published @reduxjs/toolkit@2.0.0-alpha.1 on January 21. This included the RTK packaging changes, as well as a similar change to redux-thunk, and modernized the build artifacts to no longer transpile any JS syntax and drop IE11 compat.

Of course, this did not work as well as I'd hoped :)

Mateusz Burzyński ( @andaristrake ) maintains several libraries, including Emotion and Preconstruct, and spends much of his time working on the TypeScript compiler for fun. He's an expert on many of the intricate nuances of packaging formatting.

When I announced alpha.2 on Twitter, Mateusz replied with several suggestions for tweaks (looking at both RTK 2.0 and Redux core 5.0 alphas):

no idea what dist/es/redux.mjs is for now if it's not even in the exports map

by using the types condition like this TS might always assume that this is a module, even if loaded/required from CJS target~. Since you don't have a default export... that's probably fine

I would include the module condition and point to the ./dist/es/index.js with it, this will allow the package to be only loaded once by bundlers, despite the consuming file's format (cjs vs esm)

get rid of process.env.NODE_ENV in your dist files, use development/production conditions to accomplish this stuff (probably best to make production the default and the development stuff an opt-in)

I saved those as an issue for reference.

Shortly thereafter, we received a couple of new issue reports back-to-back complaining of problems with the config in alpha.1/2:

When importing anything from @reduxjs/toolkit@2.0.0-alpha.2, typescript cannot resolve types when tsconfig.json's moduleResolution is set to "node16" or "nodenext". I found that adding the extension .js to the declaration imports resolved the issue.

The current config in the Alpha does not allow for consumption of the CJS bundle in modern version of Node/any tool that follows Node's module resolution spec, as it uses .js to refer to a CJS module despite the package setting "type": "module". If "type": "module" is set, .cjs is necessary in "exports".

Some bundlers do work around this and are more forgiving (whether or not that's a good thing is something else entirely), but this config will not work in Node and/or any environments that follow its resolution mechanism.

I was... not a happy camper:

I do honestly appreciate you filing the issue, but I am also legitimately getting angry at how messed up this whole situation is :(

I want to do right by my users and support the variety of build tools and environments I expect they'll be using for their apps.

But I can't do that if every single article and person is giving me contradictory instructions on what I'm supposed to do :(

Clearly this was going to take a lot of effort to figure out what was going on and catch possible errors.

Researching Better Configuration

Right at the same time, Devon Govett tweeted about improving Node+ESM support in a React-Aria package update.

I replied noting I was working on some similar efforts, and tagged Mateusz. Shortly after that I saw the issues get filed, linked and griped about them, and Mateusz again suggested removing type: "module".

I was feeling frustrated and begged him to publish a full blog post that would give details on his recommendations. Instead, he suggested we do a phone call and talk through things directly.

On February 27, Mateusz and I hopped onto a call along with Nathan Bierema (Redux DevTools maintainer). I saved the discussion notes in a gist:

Mateusz threw out a lot of thoughts around how ESM and CJS can get used, and questioned whether it even entirely makes sense to ship ESM at all. The information was useful in general, but it left me still feeling pretty confused about next steps.

Somewhere in that Twitter discussion, I got in touch with Andrew Branch (@atcb), a TypeScript team member who had implemented the new moduleResolution: "bundler" option for TS, and has been doing work on JS/TS ecosystem module behavior as preparation for writing. We set up a call on February 28.

Andrew gave me a rundown of how ESM works, how TS treats ESM and module import paths, and how Node and other tools determine if a file is actually ESM.

The TL;DR of that last part is roughly:

  • If you add type: "module", every file with a .js extension gets interpreted as ESM, period. .cjs files will be interpreted as CommonJS.
  • Alternately, if you don't have type: "module", .js files are treated as CommonJS. You can use .mjs to mark individual files as ESM.

There was also some discussion of whether or not we should be pre-bundling our TS typedefs or leaving them as individual someSourceFile.d.ts files in the published package.

Setting Up CI Checks for Packaging

Initial CI Setup

I'd suspected for a long time that I was going to end up needing to put together some kind of battery of example applications, each using different build tooling, to catch possible errors in packaging during PR CI checks.

After the alpha.2 issue reports, I reluctantly concluded I really needed to spend time setting up those CI checks before I did any more work on the actual package configurations.

As part of my initial testing, I had locally created a small example app that exercised all of RTK's entry points. It had a counter to exercise configureStore and createSlice from the core, a UI-agnostic RTKQ createApi endpoint, and a React-specific RTKQ createApi endpoint. I'd pasted that into several different project setups.

We already had our CI set up to pre-build a tarball containing the package contents from the current PR, and were using that to run our unit tests against the PR package version instead of our source code. I decided to try expanding on that to test these different apps against that PR build as well.

I copied the first couple example projects into a new $REPO/examples/publish-ci/ folder, and updated the GH Action workflow to matrix the folder names inside of /publish-ci/, install the PR build into each example, then build+test it:

I also wrote a small Playwright test that would check the page contents to verify it could change the counter, and that both API endpoints had fetched the right mock data, in order to ensure that the apps actually ran correctly.

I actually targeted the PR against 1.9.x on our master branch, to see how things worked with the existing package setup first.

I eventually ended up with example apps that covered:

  • CRA 4 (with Webpack 4)
  • CRA 5 (with Webpack 5)
  • Next.js (with Webpack 5)
  • Vite 4
  • Node in both CJS and ESM modes

There were plenty of other build tool combinations that probably ought to be checked, but this is a good start.

Are The Types Wrong?

Somewhere in this process, I had discovered that Andrew Branch had created a tool called Are The Types Wrong. It's a website that lets you pick a published NPM package version, or upload a .tgz file, and analyzes the package exports to report how TypeScript interprets the configuration and whether the JS files and TS typedefs match up correctly. It then shows all your detected entry points, and reports details on any mismatches and errors.

Here's an example of the report for RTK 2.0.0-alpha.2 ( https://arethetypeswrong.github.io ):

Are The Types Wrong - RTK 2.0-alpha.2 results

You can see that it detected all of RTK's entry points, and that most of the entry point + moduleResolution combinations look okay. But, moduleResolution: "node16" + some CJS environment apparently has multiple issues.

I really wanted to use this analysis in RTK's CI to help verify that any future PR changes would actually work correctly. I looked at the attw repo, and noted that Andrew had split it into core and website packages. But, the core logic wasn't yet published as a package.

I initially tried setting up a CI task that would clone the attw repo, and let me write a command-line script that would import the core logic and analyze the PR build artifact. That technically worked, but fortunately I was able to convince Andrew to publish the logic as an actual @arethetypeswrong/core package.

From there, I put together a CLI script that ran the core attw logic, collected the reports, and wrote it out as a console table to match the display on the website. I did this using the ink React CLI renderer (and probably spent a bit too much time fiddling with rendering tables in the console). The results ended up pretty good:

Are The Types Wrong CLI output

I then configured RTK's CI to call that as another check alongside building the example apps.

There was an existing thread asking for attw to add a CLI, so I offered mine up as a potential starting point. (Someone else later filed a PR to add a CLI. That CLI has now been published officially, and I need to get around to switching over to using that in CI instead of my homegrown script.

Packaging Updates, Round 2

I decided it was best to try updating the smaller packages that RTK depends on first.

I'd already had to publish a 3.0-alpha.0 for redux-thunk to alter its behavior. I decided I'd switch over and try making further updates with that.

Switching Build Tooling

redux-thunk is a single tiny source file about 20 lines long (plus some additional TS types). That made it a good starting point to mess with changing packaging.

I noted that it was still using Babel+Rollup for the build step. I decided I'd try using ESBuild instead. But, how should I use that?

We already had a custom ESBuild wrapper script over in RTK. I briefly considered copy-pasting that over to the redux-thunk repo, but decided it would be overkill.

I'd done some previous searching for other ESBuild wrappers. I did some more looking and decided to give https://github.com/egoist/tsup a shot.

tsup actually worked out pretty well! In a couple hours I had a simple tsup.config.ts file that generated the two artifacts I wanted. In this case the thunk code had no dev/prod conditional checks, so I kept it really simple - just a single ESM and CJS file apiece.

I also removed type: "module", and switched to using .mjs and .cjs for the artifacts to force ESM or CJS appropriately.

I updated the thunk package.json file to use those:

{
  "name": "redux-thunk",
  "version": "3.0.0-alpha.1",
  "main": "dist/cjs/index.cjs",
  "module": "dist/index.mjs",
  "types": "dist/index.d.ts",
  "exports": {
    "./package.json": "./package.json",
    ".": {
      "types": "./dist/index.d.ts",
      "import": "./dist/index.mjs",
      "default": "./dist/cjs/index.cjs"
    },
    "./extend-redux": {
      "types": "./extend-redux.d.ts"
    }
  }
}

Meanwhile, I ended up doing all the same Jest->Vitest conversion as the RTK repo.

UMD Build Artifact Changes

I also spent a bunch of time debating whether it was worth keeping UMD files or not. redux-thunk had shipped with a UMD bundle, primarily for use as a script tag (which I assumed was mostly being done in CodePens or similar examples).

I have repeatedly asked about whether to keep publishing UMD builds over the last couple years.

The closest I got to actual advice and answers were:

  • Fred K Schott: "HTML examples and code editors aren't even reasons to use UMD anymore, on their own. Ex: @CodePen ships with a built-in Skypack integration"
  • Marvin Hagemeister: "I think it's fine to skip UMD. All bundlers can consume ESM, and with sites like https://esm.sh that can be easily used in the browser."

So, I decided it was finally time to drop UMD builds from the Redux packages :)

Later, I decided the best replacement for UMD was to include another ESM-format build artifact that was pre-compiled to production mode and no longer had process.env.NODE_ENV references, so that it could be safely loaded in a browser. That way people could use it as a