Migrating Gitlab to RHEL8

This post is part of a series

Summary

This is the third sever (that I run in my home lab) that I have migrated to RHEL8, there are more steps than for the other two but I would say that this is still a simple process

Overview

  • make a backup of the GitLab
  • install a fresh RHEL8 VM and copy the backup files over
  • perform fresh install of GitLab
  • restore the backup
  • run a reconfigure

Details

So lets go into a little more detail on each step

Make a backup of the GitLab

There are a couple of tools that are provided by GitLab to do the backup and then there are some additional steps like backing up the certificates that you are using

Update before you start

When you restore you configuration to the freshly installed VM, that VM will no doubt be running the latest version of GitLab. So you should make sure that the version of GitLab you are running is the same version, so most likely the first step is to upgrade your GitLab instance.

If you are using the RPMs provided, this is just a yum update alway

Backup the GitLab instance

GitLab strongly recommends you separate your application data from your configuration and as such, two tools are provided.

Backup the config

The first tool will backup the config - gitlab-ctl backup-etc (see https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration)

The output of this will go into /etc/gitlab/config_backup/ unless otherwise told to. In here you will find a tar file that contains the following files

Contents of GitLab config backup tar file

server # tar -tf /etc/gitlab/config_backup/gitlab_config_1599056458_2020_09_02.tar 
tar: Removing leading `/' from member names
/etc/gitlab/
/etc/gitlab/gitlab.rb
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/trusted-certs/
/etc/gitlab/ssl/
/etc/gitlab/ssl/<fqdn>.key
/etc/gitlab/ssl/<fqdn>.crt
/etc/gitlab/ssl/<fqdn>.key-staging

As you can see, the default certifcate is included in this tar file and if you are using the default cert then thats fine. In my case Im not using it. I get my certs from letsencrypt and use these. I then specify the location in the config file

server # grep "ssl_certificate" /etc/gitlab/gitlab.rb

nginx['ssl_certificate'] = "/etc/ssl/fullchain.pem"
nginx['ssl_certificate_key'] = "/etc/ssl/privkey.pem"
# nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
# nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"

Therefore I do an additional backup step to make copies of these two files

Backup the Application data

Again a tool is provided for this gitlab-backup create (see https://docs.gitlab.com/ee/raketasks/backup_restore.html#creating-a-backup-of-the-gitlab-system)

The tool backup up these files …

  • Database
  • Attachments
  • Git repositories data
  • CI/CD job output logs
  • CI/CD job artifacts
  • LFS objects
  • Container Registry images
  • GitLab Pages content

Backup files will arrive here /var/opt/gitlab/backups/ by default

The GitLab backup tool is fully featured, far too much to go into in this post, but for example you can set a backup_keep_time to prevent your disk filling up and you can also define cloud storage locations to upload backups to. As I say thats beyond the scope of this article

So we have the application data backed up, we have the application config backup up, what next?

Backup your host SSH keys

If you are intending the freshly installed VM to look exactly like the old VM - so that other systems that interact with it wont notice the difference, then it goes without saying that you will need to copy the SSH Host Keys to the new system, so be sure to back those up also

 cp /etc/ssh/ssh_host* /backup-location 

(The authorized keys file for the git user will be regenerated when you restore your backup)

Create a new VM and copy backup files over

Install a new VM with EL8 and copy over the backup files and the backup-etc files. Replace the SSH Hostkeys and restart sshd

Fresh install

Now we have out fresh EL8 VM, next we need to get GitLab installed. If you want to trust the GitLab repo setup script then you can just do

 curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash

Otherwise you can just copy the .repo file from /etc/yum.repos.d over manually

Install GitLab

 EXTERNAL_URL="https://<your fqdn>" dnf install -y gitlab-ce

Restore tha Backup

Ensure your application backup file is in the correct location and has the correct ownership

 cp *-ce_gitlab_backup.tar /var/opt/gitlab/backups/
 chown git.git /var/opt/gitlab/backups/*-ce_gitlab_backup.tar

Stop a couple of sevices

 gitlab-ctl stop unicorn
 gitlab-ctl stop puma
 gitlab-ctl stop sidekiq

Just before I start the restore, I extract the gitlab-etc tar file and place the main config file in place and place my certifcates in the right location

 cp etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb
 cp fullchain.pem /etc/ssl/fullchain.pem
 cp privkey.pem /etc/ssl/privkey.pem

Run the restore

 gitlab-backup restore BACKUP=1598537437_2020_08_27_13.3.1-ce

Once the restore process completes, then copy over your secrets file and run a reconfigure

 cp etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json
 gitlab-ctl reconfigure 

Up and Running

You should now be up and running, test to see of you can still clone repos and make commits.

If you are getting strange errors, verify that the git/.ssh/authorized_keys is correctly setup - if its not then you can run gitlab-rake gitlab:shell:setup but this step should not be necessary

I accept, it looks like a lot of work but its really not that bad and I intend to convert these steps into a gitlab-restore-playbook.yml so make this process seemless

Hope you found this useful, OSG

See also