Easily test Gitlab pipelines locally

As I'm a huge fan of Gitlab, especially the CI part, I obviously have to write a lot of .gitlab-ci.yml files for Gitlab to know what to do. And sometimes you have a typo, invalid syntax, programs which work locally on your machine but when run in CI there are some commands, dependencies etc. missing or working differently. I just recently had this issue when using Bazel with rules_nodejs and Astro, a package had a npx command in it's preinstall script, which worked fine on my machine (as I had npx installed) but broke stuff in CI.

Technically Bazel is meant to have encapsulated builds but this doesn't work with rules_nodejs. They are aware of this, see this ticket on GitHub.

Now the issue is that my local Gitlab Runner is not a registered runner and it shouldn't be one. It should just run the pipeline locally to try it out, but this way there are no variables available like CI_JOB_TOKEN. Most of the time that's fine, but one project I worked on used this CI_JOB_TOKEN to fetch a Node/JS package from the "Gitlab Package Registry". I could thus not test this pipeline locally because it always needs to authenticate against Gitlab's package registry. So I came up with a hacky but working solution!

Install Gitlab Runner

Check out the Gitlab docs to find the installation guide for your platform and environment.

Install Docker

You can also use other executors like Shell with Gitlab Runner, but in my opinion using Docker is easier and cleaner as everything is done in its own container and can be cleaned up nicely when needed.
On Linux this should be as easy as telling your package manager to install docker. With zypper for example: sudo zypper install docker.
Or check out the official Docker docs.

Shell Function

Put the following function in your .bashrc, .zshrc, whatever your shell sources at some point.
I like to keep all my shortcuts and aliases in one place, so I put it into ~/.alias which gets source'd by my .bashrc.

gitlab-run() {
    local -;
    set -o xtrace;
    gitlab-runner exec docker --env CI_JOB_TOKEN_OVERRIDE="$CI_JOB_TOKEN" --pre-build-script 'CI_JOB_TOKEN=$CI_JOB_TOKEN_OVERRIDE;' "$@"
}

Explanation

• local -: revert all changed shell settings after the function is over
• set -o xtrace: prints the commands which are being run, makes it easier to debug and see what's actually being run
• --env CI_JOB_TOKEN_OVERRIDE="$CI_JOB_TOKEN": sets the CI_JOB_TOKEN_OVERRIDE variable in the job to the current value of CI_JOB_TOKEN
• --pre-build-script 'CI_JOB_TOKEN=$CI_JOB_TOKEN_OVERRIDE;': adds a pre-build script which takes the previously set CI_JOB_TOKEN_OVERRIDE and overwrites CI_JOB_TOKEN with it
  
  That "hack" is needed because the Gitlab internal environment variables have precedence over the self defined variables and thus CI_JOB_TOKEN will locally always be overwritten with an empty value.

Result

Now you can get a Gitlab token like a personal access token and run your pipelines locally using this command:

CI_JOB_TOKEN="<your token here>" gitlab-run <your task>

I hope this makes working with Gitlab CI a bit easier :)