Skip to main content

Infrastructure and GitLab Installation Runbook

1. Objective

Build a production single-node GitLab Self-Managed instance at:
with its Container Registry at:
The initial host is Hetzner Server Auction ID 3051857: The two drives are configured as software RAID 1. The approximate usable capacity is one drive, less filesystem and boot overhead. GitLab is the primary workload. Only one constrained Linux CI job may run on this machine at a time.

2. Required inputs before beginning

Do not start installation until the following are recorded in the Jira implementation epic: Also record the proposed maintenance window and the person authorized to use Hetzner Rescue/KVM if SSH access is lost.

3. DNS preparation

Create these records, initially with a TTL of 300 seconds:
Add equivalent AAAA records only after IPv6 is configured and firewalled on the host. Do not publish an AAAA record that routes to an unconfigured interface. Verify from two independent resolvers:
All results must equal the server IPv4 before requesting Let’s Encrypt certificates.

4. Install Ubuntu 24.04 LTS with RAID 1

4.1 Enter Hetzner Rescue

In Robot:
  1. Select the exact server with auction ID 3051857.
  2. Open Rescue.
  3. Activate the Linux rescue system.
  4. Reboot the server within the activation window.
  5. SSH as root with the temporary Rescue password.
Before touching disks, identify the hardware:
Compare the IP and drive information with Robot. Stop immediately if the machine or disks do not match the change record.

4.2 Run installimage

Start the Hetzner installer:
Select Ubuntu 24.04 LTS Base. In the generated configuration editor:
  • Confirm both NVMe drives are present as DRIVE1 and DRIVE2.
  • Set software RAID on.
  • Set RAID level to 1.
  • Set the hostname to code.naudecoreventures.com.
  • Keep the EFI/boot entries generated by installimage; do not manually remove Hetzner’s generated ESP configuration.
  • Use a simple filesystem plan. The intended logical layout is:
A typical non-UEFI portion of an installimage layout resembles:
Treat this as intent, not a blind paste. Retain any EFI lines generated for this particular server. Save the configuration and let installimage complete. Record the final generated configuration in the Jira ticket, excluding passwords. Reboot:

4.3 Verify RAID before installing GitLab

After booting the installed OS:
Every active RAID-1 array must show both members, normally represented as:
Do not continue if an array is degraded, resync is failing, or a disk is absent. A running resync is acceptable only if both members are present and progress is healthy; let it complete before load testing. Configure emergency-only swap behaviour:

5. Create named administration access

Replace ncvops with the approved named operations account only if NCV has a different naming standard:
Paste only approved Ed25519 public keys into:
Test in a second terminal:
Keep the original root session open while hardening SSH. Copy the supplied hardening file:
Open a third new SSH session and verify key login and sudo again. Only then close the Rescue/root session.

6. Apply the operating-system baseline

The supplied script can install the baseline packages and UFW rules:
Alternatively, execute the equivalent steps manually:
Enable services:
Configure UFW:
In Hetzner Robot Firewall, permit the same inbound ports. When possible, restrict SSH/22 to NCV office/VPN/admin source addresses while leaving 80 and 443 public. Apply Robot Firewall changes only while a verified session and Rescue fallback are available. Do not expose these internal services publicly:
  • PostgreSQL.
  • Redis.
  • Gitaly.
  • Prometheus internal endpoint.
  • Container Registry’s internal port.
  • Docker daemon/socket.
Check listening ports:

7. Configure hostname and time

Keep the OS clock in UTC:
Add a stable hostname mapping if it is not already present:
GitLab itself is configured to display Africa/Lusaka time.

8. Install GitLab EE in Free mode

The Enterprise Edition package is recommended so NCV can activate paid features later without migrating editions. Without a paid license, it operates with the Free feature set. Add the official package repository:
Install GitLab:
Record the exact installed version because restores require the same GitLab version and edition:

8.1 Apply the GitLab configuration

Back up the generated file:
Copy the supplied baseline:
Replace every CHANGE-ME value. Do not enable SMTP until the real non-secret settings are known. Apply configuration:
Retrieve the initial root password before GitLab removes the file after 24 hours:
Sign in as root, immediately:
  1. Change the root password.
  2. Change the root email to a monitored recovery address.
  3. Enrol 2FA.
  4. Store recovery codes in NCV’s approved password vault.
  5. Create a second named administrator and enrol that account in 2FA.
Never store the root password in Jira or source control.

8.2 Configure SMTP securely

Edit non-secret SMTP values in /etc/gitlab/gitlab.rb, set:
Keep smtp_user_name and smtp_password out of gitlab.rb. Store them in GitLab’s encrypted SMTP settings:
Enter:
Reconfigure and send a test message:
In the console:
Exit with quit. Verify SPF, DKIM, and DMARC for the sender domain with the mail provider.

9. Validate TLS and the Container Registry

Verify certificates:
A 401 Unauthorized response from /v2/ is normal before Docker authentication. From a developer machine with Docker:
Then use a temporary test project and push a small image. Delete the test project after verification.

10. Apply GitLab administrative settings

In Admin, configure and record screenshots/evidence for these settings:

Accounts and authentication

  • Disable new user account creation.
  • Require 2FA for all human users, with a short controlled grace period.
  • Keep password authentication enabled as emergency recovery unless a tested external identity provider replaces it.
  • Set an NCV support/contact address.
  • Restrict or disable user-created top-level groups.
  • Restrict project creation to approved roles/groups.
  • Review session duration and inactive-user policy.

Visibility and access

  • Default new project visibility: Private.
  • Default new snippet visibility: Private.
  • Disable public snippets unless explicitly needed.
  • Do not allow arbitrary external users.
  • Do not make source projects public to facilitate desktop updates.

CI/CD

  • Disable instance runners for projects by default until explicitly assigned.
  • Prevent untrusted projects and forks from using the co-located runner.
  • Set a sensible maximum job timeout, initially 60 minutes.
  • Configure artifact expiry defaults; release binaries belong in Package Registry, not short-lived job artifacts.

Package controls

  • Source projects remain private.
  • Only release-only projects may enable Allow anyone to pull from package registry.
  • Protect container tags representing production releases.
  • Establish cleanup policies for temporary CI images while retaining immutable release images.

11. Create the NCV group structure

Create:
Recommended visibility:
  • ncv: private.
  • ncv-releases: private by default.
  • Individual release projects: private with anonymous package pulls enabled, or public only after verifying that no source, internal notes, credentials, or customer data are present.
Role allocation: Protect main, release/*, v*, and candidate-*. Developers must not push directly to main or create stable release tags.

12. Install Docker and the constrained Linux runner

GitLab recommends separate runner hosts for production security and performance. Co-location here is an intentional low-volume starting compromise. Treat this runner as trusted-code-only. Install Docker from Ubuntu’s supported package and GitLab Runner from the official GitLab repository:
Confirm cgroups v2 so Docker limits are enforced:
Expected:

12.1 Create the runner using the current workflow

In GitLab:
  1. Open the intended group or Admin runner page.
  2. Create a new runner.
  3. Configure tags linux-small and trusted.
  4. Do not run untagged jobs.
  5. Lock it to trusted NCV projects/group.
  6. Set it as protected if only protected refs should access protected variables.
  7. Copy the runner authentication token beginning with glrt-.
Register it:
Do not use deprecated registration tokens. Back up the generated runner configuration, then merge in the limits from config/gitlab-runner-config.toml.example, preserving the real token:
The required constraints are:
Run a test pipeline with two jobs. One must run while the second queues. During the test, observe:
If GitLab latency becomes unacceptable, pause this runner and move it to a separate machine. Do not increase concurrency on this CPU merely because 64 GB RAM is available.

13. Install offsite backups before migration

Install the supplied files:
Initialize the Restic repository once:
Run and inspect the first backup:
Enable the nightly timer only after the manual run succeeds:
A backup is not accepted until it has been restored on a temporary host using the exact same GitLab version and edition.

14. Install health checking

Configure Hetzner SysMon or NCV monitoring to check:
Also monitor:
  • RAID state.
  • SMART/NVMe errors.
  • Root filesystem usage.
  • GitLab service state.
  • Backup age and Restic snapshot success.
  • TLS expiry.
  • Runner queue duration.
  • CPU steal/load and I/O wait.

15. Connect Jira Cloud

Use GitLab’s Jira Cloud app, not the old Jira Cloud DVCS connector.
  1. Confirm GitLab is publicly reachable with a trusted HTTPS certificate.
  2. In GitLab Admin, create the OAuth application required by the GitLab for Jira Cloud app.
  3. Install the official GitLab for Jira Cloud app in Atlassian.
  4. Link the ncv group or relevant subgroup.
  5. Create a branch, commit, merge request, pipeline, and test deployment referencing a real Jira key.
  6. Verify branches, commits, merge requests, builds, and deployments appear in the Jira issue development panel.
The exact configuration and developer naming rules are in docs/04-developer-guide.md.

16. Final installation acceptance gate

Do not begin Azure migration until all items pass:

17. Immediate rollback procedures

DNS rollback

Restore the previous DNS records or remove the new records. With the temporary 300-second TTL, propagation should be limited.

SSH/firewall rollback

Use Hetzner Rescue or KVM, mount the root filesystem, remove the offending SSH include or disable UFW, and reboot. Never improvise firewall changes without Rescue access.

GitLab configuration rollback

Runner rollback

Pause the runner in GitLab or stop it locally:
GitLab remains available independently.

Failed GitLab upgrade rollback

Restore a backup made on the exact version and edition being restored. Database migrations make package downgrade without a matching backup unsafe.

18. Official references