Migrating Gitlab Omnibus instance to new server

Since almost the beginning, we are using Gitlab as our main git repository server. After running on source for several years, we’ve migrated to the Omnibus installation. Since then, we’ve experienced almost no issues when upgrading between instances and we are using more and more of the provided services: Mattermost, CI, Registry.

Recently, we’ve finished a migration to a new server (simple Hetzner Cloud VPN, in case you are interested). For that case, I’ve needed to migrate all data, including the Mattermost Chat history to the new server efficiently.

1. Setup and config adjustments

1. Install Gitlab omnibus on the new server, e.g. apt install gitlab-ce
2. Create a ssh-key identity on the new server ssh-keygen and append the /root/.ssh/id_rsa.pub into the authorized_keys of the old server’s privileged user.
3. Upgrade Gitlab to the latest version on your old instance. Migration is only possible between same version
4. Copy the contents of /etc/gitlab to the new server, including the gitlab.rb. Now would be a good time for changing defaults inside.
   • Use the built-in command gitlab-ctl diff-config to see, what settings changed between versions. I would recommend to copy over all default values and comments, to future diffs are much more clean.
   • E.g. we switched back from S3 container-registry to local registry to same a good amount of money¹
   • We’ve also moved from a proxied nginx with external Letsencrypt to Gitlab provisioned Letsenrypt.

# the old host, must have direct ssh access
# generate ssh-keygen on new host and copy over into authorized_keys
gitlab='root@1.2.3.4'

echo "$gitlab - syncing content of /etc/gitlab"
mkdir -p /etc/gitlab/
rsync -a $gitlab:/etc/gitlab/. /etc/gitlab/.
gitlab-ctl reconfigure

2. Doing the migration

Best thing is, prepare everything as a script which you just run or copy & paste the individual commands. Here is the script I’ve followed on the migration. We take the thing appart and I link the full script as a Gist.

1. Seeting the login of the old server for convenience into the variable $gitlab.

#!/bin/bash
# ATTENTION: COPY&PASTE RUN EVERY LINE BY HAND AND VERIFY THE OUTPUT

# the old host, must have direct ssh access
# generate ssh-keygen on new host and copy over into authorized_keys
gitlab='root@1.2.3.4'

Preferred: Copy over ssh server identity, otherwise you will receive tons of warnings when using the ssh interface of your future Gitlab server.

# copy ssh hostkey, so that the ssh connection from clients still work
cd /etc/ssh
scp $gitlab:/etc/ssh/ssh_host_* .
service ssh restart

2. Stopping all processes that access the database on the target host

Don’t use gitlab-ctl stop without arguments. We need the postgresql daemon.

# Stopping everything on remote system
echo "$gitlab - Gitlab Stop"
ssh $gitlab "gitlab-ctl stop nginx"
ssh $gitlab "gitlab-ctl stop sidekiq"
ssh $gitlab "gitlab-ctl stop mattermost"

3. Create a on demand backup

echo "$gitlab - Make Backup"
ssh $gitlab "gitlab-backup create SKIP=lfs"

4. Copy over backup

# copy over last backup
backup_file=`ssh $gitlab "cd /var/opt/gitlab/backups && ls *.tar | tail -n 1"`
echo "Sync - /var/opt/gitlab/backups/"
mkdir -p /var/opt/gitlab/backups/
scp -p $gitlab:/var/opt/gitlab/backups/$backup_file /var/opt/gitlab/backups/$backup_file
chown git:git /var/opt/gitlab/backups/*.tar

5. Copy over registry

The registry is not included in the Gitlab backup, copy it over, if you are using local hosted (skip this step if you are using S3 for container images):

mkdir -p /var/opt/gitlab/registry/docker
rsync $ssh:/var/opt/gitlab/registry/docker/. /var/opt/gitlab/registry/docker/.
chown registry -R /var/opt/gitlab/registry/docker

5. Importing backup

echo "local - Stopping gitlab"
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-backup restore

6. Mattermost

Syncing the Mattermost data dir (including all attachments and Mattermost config and plugins).

# Mattermost data dir
echo "Sync - /var/opt/gitlab/mattermost"
mkdir -p /var/opt/gitlab/mattermost/
rsync -a $gitlab:/var/opt/gitlab/mattermost/. /var/opt/gitlab/mattermost/.
sudo chown -R mattermost /var/opt/gitlab/mattermost/

Creating a live Mattermost backup on the remove server.

# Mattermost sql
echo "$gitlab - Creating mattermost backup"
ssh $gitlab 'sudo -u gitlab-psql -- /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql mattermost_production | gzip ' > /root/mattermost-db.sql.gz

Importing the db:

echo "local - reimport mattermost db"
sudo -u mattermost /opt/gitlab/embedded/bin/dropdb -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production
sudo -u gitlab-psql /opt/gitlab/embedded/bin/createdb -U gitlab-psql -h /var/opt/gitlab/postgresql -p 5432 mattermost_production
gunzip < /root/mattermost-db.sql.gz | sudo -u mattermost /opt/gitlab/embedded/bin/psql -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production

7. Finish Backup and restart

gitlab-ctl reconfigure
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true

Don’t forget the Backup cronjob if you’ve used any, keep in mind that the default backup does not include Mattermost history nor the container registry.

0 2 * * 1-6 /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1 SKIP=lfs

Full Gist

Bonus: Problem when upgrading the latest Gitlab on old server broke because of ulimits rule

Before migrating, we’ve upgraded the Gitlab to the newest version. That broke the Gitlay daemon, which did not start again.

$ gitlab-ctl tail gitaly
./run: 11: ulimit: error setting limit (Operation not permitted)

Problem was, that Gitaly start tries to change ulimit rules and fails if that is not possible, which was, in our case, because we used a Proxmox/LXC container. See this Gitlab issue.

Solution: set gitaly['open_files_ulimit'] = false in /etc/gitlab/gitlab.rb and reconfigure.