Infrastructure and GitLab Installation Runbook
1. Objective
Build a production single-node GitLab Self-Managed instance at:3051857:
The two drives are configured as software RAID 1. The approximate usable capacity is one drive, less filesystem and boot overhead. GitLab is the primary workload. Only one constrained Linux CI job may run on this machine at a time.
2. Required inputs before beginning
Do not start installation until the following are recorded in the Jira implementation epic:
Also record the proposed maintenance window and the person authorized to use Hetzner Rescue/KVM if SSH access is lost.
3. DNS preparation
Create these records, initially with a TTL of 300 seconds:AAAA records only after IPv6 is configured and firewalled on the host. Do not publish an AAAA record that routes to an unconfigured interface.
Verify from two independent resolvers:
4. Install Ubuntu 24.04 LTS with RAID 1
4.1 Enter Hetzner Rescue
In Robot:- Select the exact server with auction ID
3051857. - Open Rescue.
- Activate the Linux rescue system.
- Reboot the server within the activation window.
- SSH as
rootwith the temporary Rescue password.
4.2 Run installimage
Start the Hetzner installer:
- Confirm both NVMe drives are present as
DRIVE1andDRIVE2. - Set software RAID on.
- Set RAID level to
1. - Set the hostname to
code.naudecoreventures.com. - Keep the EFI/boot entries generated by
installimage; do not manually remove Hetzner’s generated ESP configuration. - Use a simple filesystem plan. The intended logical layout is:
installimage layout resembles:
installimage complete. Record the final generated configuration in the Jira ticket, excluding passwords.
Reboot:
4.3 Verify RAID before installing GitLab
After booting the installed OS:5. Create named administration access
Replacencvops with the approved named operations account only if NCV has a different naming standard:
sudo again. Only then close the Rescue/root session.
6. Apply the operating-system baseline
The supplied script can install the baseline packages and UFW rules:- PostgreSQL.
- Redis.
- Gitaly.
- Prometheus internal endpoint.
- Container Registry’s internal port.
- Docker daemon/socket.
7. Configure hostname and time
Keep the OS clock in UTC:Africa/Lusaka time.
8. Install GitLab EE in Free mode
The Enterprise Edition package is recommended so NCV can activate paid features later without migrating editions. Without a paid license, it operates with the Free feature set. Add the official package repository:8.1 Apply the GitLab configuration
Back up the generated file:CHANGE-ME value. Do not enable SMTP until the real non-secret settings are known.
Apply configuration:
root, immediately:
- Change the root password.
- Change the root email to a monitored recovery address.
- Enrol 2FA.
- Store recovery codes in NCV’s approved password vault.
- Create a second named administrator and enrol that account in 2FA.
8.2 Configure SMTP securely
Edit non-secret SMTP values in/etc/gitlab/gitlab.rb, set:
smtp_user_name and smtp_password out of gitlab.rb. Store them in GitLab’s encrypted SMTP settings:
quit. Verify SPF, DKIM, and DMARC for the sender domain with the mail provider.
9. Validate TLS and the Container Registry
Verify certificates:401 Unauthorized response from /v2/ is normal before Docker authentication.
From a developer machine with Docker:
10. Apply GitLab administrative settings
In Admin, configure and record screenshots/evidence for these settings:Accounts and authentication
- Disable new user account creation.
- Require 2FA for all human users, with a short controlled grace period.
- Keep password authentication enabled as emergency recovery unless a tested external identity provider replaces it.
- Set an NCV support/contact address.
- Restrict or disable user-created top-level groups.
- Restrict project creation to approved roles/groups.
- Review session duration and inactive-user policy.
Visibility and access
- Default new project visibility: Private.
- Default new snippet visibility: Private.
- Disable public snippets unless explicitly needed.
- Do not allow arbitrary external users.
- Do not make source projects public to facilitate desktop updates.
CI/CD
- Disable instance runners for projects by default until explicitly assigned.
- Prevent untrusted projects and forks from using the co-located runner.
- Set a sensible maximum job timeout, initially 60 minutes.
- Configure artifact expiry defaults; release binaries belong in Package Registry, not short-lived job artifacts.
Package controls
- Source projects remain private.
- Only release-only projects may enable Allow anyone to pull from package registry.
- Protect container tags representing production releases.
- Establish cleanup policies for temporary CI images while retaining immutable release images.
11. Create the NCV group structure
Create:ncv: private.ncv-releases: private by default.- Individual release projects: private with anonymous package pulls enabled, or public only after verifying that no source, internal notes, credentials, or customer data are present.
Protect
main, release/*, v*, and candidate-*. Developers must not push directly to main or create stable release tags.
12. Install Docker and the constrained Linux runner
GitLab recommends separate runner hosts for production security and performance. Co-location here is an intentional low-volume starting compromise. Treat this runner as trusted-code-only. Install Docker from Ubuntu’s supported package and GitLab Runner from the official GitLab repository:12.1 Create the runner using the current workflow
In GitLab:- Open the intended group or Admin runner page.
- Create a new runner.
- Configure tags
linux-smallandtrusted. - Do not run untagged jobs.
- Lock it to trusted NCV projects/group.
- Set it as protected if only protected refs should access protected variables.
- Copy the runner authentication token beginning with
glrt-.
config/gitlab-runner-config.toml.example, preserving the real token:
13. Install offsite backups before migration
Install the supplied files:14. Install health checking
- RAID state.
- SMART/NVMe errors.
- Root filesystem usage.
- GitLab service state.
- Backup age and Restic snapshot success.
- TLS expiry.
- Runner queue duration.
- CPU steal/load and I/O wait.
15. Connect Jira Cloud
Use GitLab’s Jira Cloud app, not the old Jira Cloud DVCS connector.- Confirm GitLab is publicly reachable with a trusted HTTPS certificate.
- In GitLab Admin, create the OAuth application required by the GitLab for Jira Cloud app.
- Install the official GitLab for Jira Cloud app in Atlassian.
- Link the
ncvgroup or relevant subgroup. - Create a branch, commit, merge request, pipeline, and test deployment referencing a real Jira key.
- Verify branches, commits, merge requests, builds, and deployments appear in the Jira issue development panel.
docs/04-developer-guide.md.
16. Final installation acceptance gate
Do not begin Azure migration until all items pass:17. Immediate rollback procedures
DNS rollback
Restore the previous DNS records or remove the new records. With the temporary 300-second TTL, propagation should be limited.SSH/firewall rollback
Use Hetzner Rescue or KVM, mount the root filesystem, remove the offending SSH include or disable UFW, and reboot. Never improvise firewall changes without Rescue access.GitLab configuration rollback
Runner rollback
Pause the runner in GitLab or stop it locally:Failed GitLab upgrade rollback
Restore a backup made on the exact version and edition being restored. Database migrations make package downgrade without a matching backup unsafe.18. Official references
- GitLab Linux package on Ubuntu: https://docs.gitlab.com/install/package/ubuntu/
- GitLab Linux package overview: https://docs.gitlab.com/install/package/
- GitLab installation requirements: https://docs.gitlab.com/install/requirements/
- GitLab TLS: https://docs.gitlab.com/omnibus/settings/ssl/
- GitLab Container Registry administration: https://docs.gitlab.com/administration/packages/container_registry/
- GitLab Runner installation and registration: https://docs.gitlab.com/runner/install/linux-repository/ and https://docs.gitlab.com/runner/register/
- New runner creation workflow: https://docs.gitlab.com/ci/runners/new_creation_workflow/
- GitLab backup and restore: https://docs.gitlab.com/administration/backup_restore/
- Hetzner installimage: https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/
- Hetzner software RAID drive replacement: https://docs.hetzner.com/robot/dedicated-server/raid/exchanging-hard-disks-in-a-software-raid/