Rails Mrsk Deployment - Migrating from Ansible-Capistrano like to Single-Host Docker deployment using Mrsk

With the new Rails 7.1 release around the corner, a new deployment tool by 37signals is also available called mrsk.

Currently, I have been running all my apps on cheapish Hetzner Cloud Vms, usually, one host per app, web-hosted via Passenger. Everything is co-located, which makes it extremely fast and surprisingly available (4-5 nines - only downtime is usually due to developer/CI error, with maybe 1 downtime by Hetzner per host every 2 years or so). But, having to run a long Ansible script on a stateful server has its downsides, like: “Did I document every dependency?” (Imagemagick, google chrome, pdf generation tools, etc.), “Oh, new Ruby/Node Version, I have to manually swap the Ruby version during deployment and then change the Nginx/Passenger version, too.”, “Oh, new Ubuntu/Debian version, mhmm should I risk lsb-release-upgrade and deal with the consequence or build a new machine and migrate all stuff over…”. Also, running special dependencies (Like PostgreSQL Proonga extension in my case) is usually difficult because you need a good deb/ppa matching your distribution or compile it yourself.

Having everything—apps and databases neatly packed in a Docker image is a nice idea, so I decided to give it a try. In the process of migrating, I experienced some issues, which I will try to document here.

My Setup - Loadbalancer + Single Host

I use Hetzner Cloud VMs and run a dedicated “Load-balancer” VM for all my private stuff – You could also use the load-balancer by Hetzner Cloud, but I want to save some bucks, and when I started my other projects and cloud setup, Hetzner Cloud load-balancer was not yet available. So maybe in the future I will migrate to the Hetzner Cloud LB. Because I had this load-balancer, I could tryout a new host with the containerized deployment, test it, migrate the data if successful without too much downtime.

In any case, it is a good idea to have a dedicated load-balancer, because Mrsk does not support SSL termination via Letsencrypt easily - because, if you have multiple hosts, all need access to the SSL certificates.

Because of the simplicity, I wanted to keep all services (App + Worker + DB + Redis) on a single cloud VM. For my private apps that is perfect and cheap. After a recent PR, Mrsk is now able to support that use case as well.

So for my use case, docker-compose with a remote Docker (using DOCKER_HOST=ssh://user@host) would have been a good solution as well. But I wanted to try out mrsk, which supports multiple hosts, so I can use it for my other projects as well.

Prerequisites

• To be able to run in a Docker container, your app needs to be stateless. That means, at best no local storage, and no static configuration. Every secret and connection config should be added via environment variables or credentials.yml.enc. Also, attachments should be stored in a cloud storage like S3. In our case, for a single host app, we don’t just need that, yet. We just have to use a Docker volume where appropriate.
• You need to think about SSL-termination: If you use a service like Cloudflare or the Hetzner Cloud load-balancer, you can just use them, and point them to your app. Otherwise, you might also need a Caddy/Nginx/Traefik web server that fronts your app.
• A cloud VM or local machine with Docker installed. If you order a cloud VM via Hetzner, I recommend to immediately activate the firewall and only allow SSH traffic. Internal traffic without Hetzner VPN is unaffected.
  • BTW: If you like to use Terraform to manage that (upload ssh-key, order server with ssh-key, and Cloud Init to install Docker and set unattended updates, add firewall rules and internal network), here is my example Terraform config as a starting point.

Adding Mrsk to a Rails 7.1 app

First, it might be a good idea to upgrade to Rails 7.1, as it brings some new improvements, like Default Dockerfiles, don’t require a SECRET_KEY_BASE during asset compilation, assume_ssl, Health status check on /up and more. Right now as of writing this post, Rails 7.1 is not released, yet, but you can try to just use the main branch:

git_source(:github) { |repo| "https://github.com/#{repo}.git" }
gem "rails", github: "rails/rails", branch: "main"

Then, after upgrading the app as usual, you can add mrsk to it:

gem 'mrsk', require: false

bundle
mrsk init

This will create a config/deploy.yml. You will also need a Dockerfile and a .dockerignore file, which are generated by a new Rails 7.1 app.

Dockerfile - Multi-stage build with caching of assets

The new Rails 7.1 default Dockerfile is a good starting point, because it uses a multi-stage build, so you don’t need nodejs, gcc etc. in your production image. You can just generate a new Rails 7.1 app with your defaults and copy over the Dockerfile. In addition, I also found the Evil Martian Docker Guide very useful and applied some of the suggestions there to further speed up the build process, by using caches.

From them, I got the tip about using BuiltKit’s mount=type=cache options to speed it up by a lot. In the example below, I also facilitate Docker’s build cache for storing Vite and Sprockets assets between builds, by rsync’ing from the cache before and back after the build.

# syntax = docker/dockerfile:1

ARG RUBY_VERSION=3.2.1
ARG NODE_VERSION=14.21.3
ARG YARN_VERSION="^1.22.19"
FROM ruby:$RUBY_VERSION-slim as base

WORKDIR /rails

ENV RUNTIME_DEPS="curl gnupg2 libvips libvips-dev tzdata imagemagick librsvg2-dev libmagickwand-dev postgresql-client" \
    BUILD_DEPS="build-essential libpq-dev git less pkg-config python-is-python3 node-gyp vim rsync"

FROM base as build

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=tmpfs,target=/var/log \
  rm -f /etc/apt/apt.conf.d/docker-clean; \
  echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache; \
  apt-get update -qq \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends $BUILD_DEPS $RUNTIME_DEPS

ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
    /tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
    npm install -g yarn@$YARN_VERSION && \
    rm -rf /tmp/node-build-master

ENV RAILS_ENV="production" \
    BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle" \
    BUNDLE_JOBS="4" \
    BUNDLE_NO_CACHE="true" \
    BUNDLE_WITHOUT="development,test" \
    GEM_HOME="/usr/local/bundle"

COPY Gemfile Gemfile.lock ./
RUN --mount=type=cache,target=~/.bundle/cache \
  bundle config --local deployment 'true' \
  && bundle config --local path "${BUNDLE_PATH}" \
  && bundle install \
  && rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git \
  && bundle exec bootsnap precompile --gemfile

COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

COPY . .

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/


# reduce node memory usage:
ENV NODE_OPTIONS="--max-old-space-size=4096"
ARG DATABASE_URL="postgres://postgres:postgres@db:5432/postgres"

# Mount node_modules and tmp/cache/vite as cache volume:
RUN --mount=type=cache,target=/rails/node_modules \
  --mount=type=cache,target=/rails/tmp/cache/vite \
  --mount=type=cache,target=/rails/tmp/cache/assets \
  --mount=type=cache,target=/rails/tmp/assets_between_runs \
  mkdir -p /rails/tmp/assets_between_runs/vite /rails/public/vite && rsync -a /rails/tmp/assets_between_runs/vite/. /rails/public/vite/. && \
  mkdir -p /rails/tmp/assets_between_runs/assets /rails/public/assets && rsync -a /rails/tmp/assets_between_runs/assets/. /rails/public/assets/. && \
  SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile && \
  rsync -a /rails/public/vite/. /rails/tmp/assets_between_runs/vite/. && \
  rsync -a /rails/public/assets/. /rails/tmp/assets_between_runs/assets/.


# Final stage for app image
FROM base as app

# Install packages needed for deployment
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
   --mount=type=cache,target=/var/lib/apt,sharing=locked \
   --mount=type=tmpfs,target=/var/log \
   apt-get update -qq && \
   apt-get install --no-install-recommends -y $RUNTIME_DEPS cron

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
   --mount=type=cache,target=/var/lib/apt,sharing=locked \
   --mount=type=tmpfs,target=/var/log \
    curl -sL https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
    echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list && \
    echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list && \
   apt-get update -qq && \
   apt-get install --no-install-recommends -y google-chrome-stable

# Copy built artifacts: gems, application
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build /rails /rails

# Run and own only the runtime files as a non-root user for security
RUN useradd rails --home /rails --shell /bin/bash && \
    chown -R rails:rails db log tmp
USER rails:rails

# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD ["./bin/rails", "server"]

Also add the default Docker entrypoint to bin/docker-entrypoint which migrates the app before running it.

#!/bin/bash -e
# bin/docker-entrypoint

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

Find all the files in this gist:

GIST with Docker files

In general, if you run “docker build” and it takes a lot of time to submit the “context’, you can try to add more files to the .dockerignore.

Building your first Docker image might be a bit tricky, so it is a good idea to try out the build process locally by just running docker build . and fix the errors.

Building using Mac M1/M2 with amd64 issues

Building the Docker Image using an M1/M2 might fail, if you need amd64 specific stuff. In my case: Google Chrome for headless browsing. So I decided to just build the Docker image on the target cloud VM by Hetzner using a remote build. Usually, you can easily tunnel Docker to another host, by setting the environment variable DOCKER_HOST=ssh://user@host. But with mrsk you will instead need to set the remote build options in the config/deploy.yml. It worked great, but unfortunately, the caching of later app stages does not work very well and it reinstalls all the deb’s during every build. That is one part, I am looking forward to being fixed in the future.

Running Mrsk

Configure Docker registry

First, you need a Docker registry. As I already used Gitlab for code hosting, I decided to just use the Gitlab registry. For free users, you have a storage limit of 5GB which should be enough for the last couple of tags if none of your base dependencies change too often. To generate a “password”, head over to the Gitlab registry, and in the project under “Settings” -> “Access Tokens” you can generate a new token with the scopes read_registry and write_registry.

Then add the registry configuration to your config/deploy.yml.

# config/deploy.yml
registry:
  # use Docker reg, gitlab etc.
  server: registry.gitlab.com
  username: yourUsername
  password:
    - MRSK_REGISTRY_PASSWORD

And add the token as MRSK_REGISTRY_PASSWORD to your .env file (which should be in your .gitignore).

MRSK_REGISTRY_PASSWORD=YourDeploymentTokenByGitlab

Mrsk commands

Mrsk has a very extensive CLI api interface. Just explore it with mrsk help. You can start/stop/rebuild individual apps and services (“accessory”).

Here is the list of commands that I used:

# initial setup, install docker, services etc.
mrsk setup

# build Docker image, tag it, upload it to registry, pull it restart it and
# restart with zero downtime
mrsk deploy

# show app logs:
mrsk app logs -f

# show logs of specific service:
mrsk app logs -r job

# Docker ps everything:
mrsk details

Solutions

The videos, both by DHH and Drifting Ruby, are great resources to get started with Mrsk. Besides that, I came up with the following solutions to my problems:

Running everything on a single host

If you want to run everything on the same host, AND also don’t use 0.0.0.0 as the publishing host, and only advertise in the internal Docker network, I had to hack a little bit:

# config/deploy.yml
servers:
  web:
    hosts:
      # set to the public ip of the host
      - 123.123.123.123
    options:
      # add the host ip, so the containers can reach each other without docker-compose by using
      # postgres://host.docker.internal/.. etc.
      add-host: host.docker.internal:host-gateway
env:
  clear:
    DATABASE_URL: "postgres://postgres@host.docker.internal:5432/app"
    REDIS_URL: "redis://host.docker.internal:6379/0"
accessories:
  db:
    image: "postgres:13"
    hosts:
      - 123.123.123.123
    # you can also just use 5432:5432 BUT then the host is reachable via public internet if NO FIREWALL ENABLED!!
    # I just use the Docker main ip here.
    port: "172.17.0.1:5432:5432"
    # ...
  redis:
    image: redis:7.0
    hosts:
      - 123.123.123.123
    port: "172.17.0.1:6379:6379"
    # ...

As you can see, we use the same IP-address under hosts for each service. Then we add the Docker option --add-host=host.docker.internal:host-gateway to the Docker run command. This will add the host internal IP to the app container’s /etc/hosts file. So the container can reach all the other containers published ports via host.docker.internal:PORT. Then we can use the Environment Variables used by Rails to configure the database and Redis connection.

Cronjobs

If you are using whenever to generate the crontab, generate it during the build of the app stage. You can even do it before copying all the app code, so you don’t have to generate it, if the schedule.rb did not change.

# ... gems
# Install node modules
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

# Crontab
COPY config/schedule.rb config/schedule.rb
RUN bundle exec whenever > crontab

# Copy reset application code
COPY . .
# ...

Now, I added a dedicated entrypoint for running the Cron binary. The point is, cron needs to be run as root in a container, but our app is running as the rails user. Otherwise, you might get a Docker and cron is broken: can't lock /var/run/crond.pid error. So for this task, we need to start the container using the root user.

#!/bin/bash

# chmod +x bin/cron

env >> /etc/environment
crontab -u rails ./crontab

# run cron in foreground and enable logging
exec "/usr/sbin/cron" -f -l 2

Now we can add a cron service/app server:

# config/deploy.yml
servers:
  web: ...
  cron:
    hosts:
      - 123.123.123.123
    cmd: >
      ./bin/cron
    options:
      add-host: host.docker.internal:host-gateway
      user: root

And mrsk deploy the app again. Check the logs of the cron container with mrsk app logs -r cron.

Conclusion

For now, Mrsk seems to work ok for me; the only downside is the remote build, which takes some time because it does not cache correctly in my remote build situation. In comparison to running a full cluster on K8s or Nomad/Consul, it is conceptually much simpler and easier to understand, especially, if you don’t need dynamic scheduling (meaning: you don’t need to scale up/down based on load). All it does is run Docker commands on a host, so you need to understand Docker, especially network/volumes. Also, don’t forget to enable the cloud-firewall or be careful about the publish‘d ports. Otherwise, it might be very easy to expose your database to the public internet. At the same time, if you only deploy to a single host, docker-compose with remote DOCKER_HOST would be a great fit, too. But on the other hand, Mrsk can also “scale” to multiple hosts easily, if needed in the future. I am looking forward to seeing how it will evolve, and will probably migrate another app over.

Resources List

• Mrsk Github
• Intro video by DHH
• Longer video by Drifting Ruby
• Docker on Rails guide by Evil Martians
• Gist with my Docker files as starting point
• Gist with example Terraform config

(blog image generated by Stable diffusion prompt)