Speeding up any and all js-projects with a single, global makefile

TLDR: Jump to complete markdown-file

When working on various projects, you often need to install dependencies. When switching branches, it's easy to forget this, which can lead to bugs. Consider a branch where one of the dependencies is updated. Each time you switch to and from this branch, you need to reinstall dependencies. Of course, the package manager is often smart enough to install only what is needed, so you usually run the package manager's install command before continuing.

However, running the package manager's install command is still somewhat slow, even if there are no changes. It often also runs some post-install scripts that usually only need to be executed once.

Random example-projects on my computer, all run right after a successful install:

$ npm install      1.54s user 0.18s system  45% cpu 3.802 total
$ yarn install     2.17s user 0.20s system 159% cpu 1.488 total
$ bun install     0.44s user 0.06s system  18% cpu 2.666 total

The example above is not meant to compare different package managers, as they are working on very different projects here. However, there is something much faster than all of these and can be used in any project: make.

Make is great in that it is fast to instantiate and can easily be told how to cache resources, thus only performing commands as they are needed:

make node_modules  0.01s user 0.06s system 159% cpu 0.062 total

0.01s is much faster than any other package manager. We can use a Makefile in each project, which I prefer for projects I control. However, it may not always be welcome to add Make, especially if other maintainers are not using it, even though it is great.

We can create a Makefile that can be invoked from any directory, acting on that directory.

With the contents of ~/myscripts/js_project_Makefile:

node_modules: package.json package-lock.json
    npm install
    touch $@

For this, I suggest adding this function to your ~/.bashrc-file (or ~/.zshrc etc depending on your shell of choice):

jsmake() {
  make --no-print-directory -f ~/myscripts/js_project_Makefile -C . node_modules
}

Now you can invoke the install-command a lot quicker. However, it does have some drawbacks:

• Only works with npm

• Does not run any of the scripts

Lets fix this now.

Works with any package-manager

The first point can be fixed by checking which lock-file exists, since each package-manager uses their own lock-file.

define find_lock_file
if [ -f bun.lockb ]; then \
  echo bun.lockb; \
elif [ -f yarn.lock ]; then \
  echo yarn.lock; \
elif [ -f pnpm-lock.yaml ]; then \
  echo pnpm-lock.yaml; \
elif [ -f package-lock.json ]; then \
  echo package-lock.json; \
else \
  echo ""; \
fi
endef

define find_pkg_manager
if [ "$(LOCK_FILE)" = "bun.lockb" ]; then \
    echo bun; \
elif [ "$(LOCK_FILE)" = "yarn.lock" ]; then \
    echo yarn; \
elif [ "$(LOCK_FILE)" = "pnpm-lock.yaml" ]; then \
    echo pnpm; \
else \
    echo npm; \
fi
endef

LOCK_FILE := $(shell $(find_lock_file))
PKG_MNGR := $(shell $(find_pkg_manager))

node_modules: package.json $(LOCK_FILE)
    $(PKG_MNGR) install
    touch $@

This allows us to run the correct package manager with $(PKG_MNGR), assuming we can identify the right one to use. While there are likely more package managers available, adding support for additional ones should be straightforward.

Running jsmake should now use the correct package manager, so we don't need to remember which one each project uses.

This is great because now I don't need to manually run the package manager's install script. However, I often run the scripts in the package.json section. Although I think it's better to use a project makefile for this too, we can create a few helpers to run these scripts with the correct package manager. This has the added benefit of running the install script before the main script, but only if necessary. With this change, I no longer need to remember to call the install script; I just run the script I want, and make will call install if needed.

# Runs a script for the package-manager
run: node_modules
    $(PKG_MNGR) run $(SCRIPT_NAME)

Make is not really made to take in arguments, so one needs to define key-value pairs. But we can fix this with our jsmake-function like this:

jsmake() {
    make --no-print-directory -f ~/myscripts/js_project_Makefile -C . run SCRIPT_NAME="$1"
}

now we can invoke it sort of like how we are used to:

$ jsmake dev

Since I quite often run a lot of different projects, with different names for all their scripts, I like to have an interactive search for scripts, which will run the selected script. fzf is great here, and we can add a tiny integration with adding this to the makefile:

# Searches for available scripts interactively (requires fzf), and runs them if found
runSearch: node_modules
    @jq '.scripts | to_entries | map("\(.key)\t\(.value|tostring)")[]' package.json -r | fzf --query="$(SCRIPT_NAME)" --preview 'echo {1}' --select-1 --separator '\t' | cut -d$$'\t' -f1 | xargs -I{} $(PKG_MNGR) run "{}"

and then change jsmake to:

jsmake() {
    make --no-print-directory -f ~/myscripts/js_project_Makefile -C . runSearch SCRIPT_NAME="$1"
}

Now we have interactive search for scripts, which also runs the install-script as needed.

Complete makefile for referance:

define find_lock_file
if [ -f bun.lockb ]; then \
  echo bun.lockb; \
elif [ -f yarn.lock ]; then \
  echo yarn.lock; \
elif [ -f pnpm-lock.yaml ]; then \
  echo pnpm-lock.yaml; \
elif [ -f package-lock.json ]; then \
  echo package-lock.json; \
else \
  echo ""; \
fi
endef

define find_pkg_manager
if [ "$(LOCK_FILE)" = "bun.lockb" ]; then \
    echo bun; \
elif [ "$(LOCK_FILE)" = "yarn.lock" ]; then \
    echo yarn; \
elif [ "$(LOCK_FILE)" = "pnpm-lock.yaml" ]; then \
    echo pnpm; \
else \
    echo npm; \
fi
endef

LOCK_FILE := $(shell $(find_lock_file))
PKG_MNGR := $(shell $(find_pkg_manager))

node_modules: package.json $(LOCK_FILE)
    $(PKG_MNGR) install
    touch $@
# Runs a script for the package-manager
run: node_modules
    $(PKG_MNGR) run $(SCRIPT_NAME)
# Searches for available scripts interactively (requires fzf), and runs them if found
runSearch: node_modules
    @jq '.scripts | to_entries | map("\(.key)\t\(.value|tostring)")[]' package.json -r | fzf --query="$(SCRIPT_NAME)" --preview 'echo {1}' --select-1 --separator '\t' | cut -d$$'\t' -f1 | xargs -I{} $(PKG_MNGR) run "{}"

.PHONY: run runSearch

Helper for invoking:

jsmake() {
    make --no-print-directory -f ~/myscripts/js_project_Makefile -C . runSearch SCRIPT_NAME="$1"
}