> ## Documentation Index
> Fetch the complete documentation index at: https://deployment.naudecoreventures.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Infrastructure and GitLab Installation Runbook

> Hetzner, RAID, Ubuntu, DNS, firewall, GitLab, registry and runner installation

# Infrastructure and GitLab Installation Runbook

## 1. Objective

Build a production single-node GitLab Self-Managed instance at:

```text theme={null}
https://code.naudecoreventures.com
```

with its Container Registry at:

```text theme={null}
https://registry.code.naudecoreventures.com
```

The initial host is Hetzner Server Auction ID `3051857`:

| Resource | Value |
| - | - |
| CPU | Intel Core i7-6700, 4 cores / 8 threads |
| RAM | 64 GB DDR4 |
| Storage | 2 × 512 GB NVMe |
| Network | 1 Gbit, unlimited traffic |
| Location | FSN1-DC17 |

The two drives are configured as **software RAID 1**. The approximate usable capacity is one drive, less filesystem and boot overhead. GitLab is the primary workload. Only one constrained Linux CI job may run on this machine at a time.

## 2. Required inputs before beginning

Do not start installation until the following are recorded in the Jira implementation epic:

| Input | Required value/owner |
| - | - |
| Hetzner Robot access | Named infrastructure administrator |
| Server IPv4 | From Robot |
| IPv6 policy | Enable and firewall, or document why deferred |
| DNS administrator | Person with access to `naudecoreventures.com` |
| GitLab root recovery owner | Two named administrators |
| Primary admin SSH keys | Ed25519 public keys only |
| Let's Encrypt contact email | Monitored NCV address |
| GitLab sender address | Recommended `gitlab@naudecoreventures.com` |
| SMTP host/port/auth | Approved transactional mail provider |
| Offsite backup target | Hetzner Storage Box or approved S3-compatible target |
| Backup encryption password | Generated and stored in password vault |
| Jira Cloud administrator | Named person |
| Azure DevOps organization owner | Named migration owner |
| Windows build runner owner | Named engineer/administrator |
| Apple/macOS runner owner | Named engineer/administrator |

Also record the proposed maintenance window and the person authorized to use Hetzner Rescue/KVM if SSH access is lost.

## 3. DNS preparation

Create these records, initially with a TTL of 300 seconds:

```dns theme={null}
code.naudecoreventures.com.           A     <SERVER_IPV4>
registry.code.naudecoreventures.com.  A     <SERVER_IPV4>
```

Add equivalent `AAAA` records only after IPv6 is configured and firewalled on the host. Do not publish an `AAAA` record that routes to an unconfigured interface.

Verify from two independent resolvers:

```bash theme={null}
dig +short A code.naudecoreventures.com @1.1.1.1
dig +short A code.naudecoreventures.com @8.8.8.8
dig +short A registry.code.naudecoreventures.com @1.1.1.1
```

All results must equal the server IPv4 before requesting Let's Encrypt certificates.

## 4. Install Ubuntu 24.04 LTS with RAID 1

### 4.1 Enter Hetzner Rescue

In Robot:

1. Select the exact server with auction ID `3051857`.
2. Open **Rescue**.
3. Activate the Linux rescue system.
4. Reboot the server within the activation window.
5. SSH as `root` with the temporary Rescue password.

Before touching disks, identify the hardware:

```bash theme={null}
hostname
ip -br address
lsblk -e7 -o NAME,MODEL,SERIAL,SIZE,TYPE
smartctl -a /dev/nvme0n1 || true
smartctl -a /dev/nvme1n1 || true
```

Compare the IP and drive information with Robot. Stop immediately if the machine or disks do not match the change record.

### 4.2 Run `installimage`

Start the Hetzner installer:

```bash theme={null}
installimage
```

Select **Ubuntu 24.04 LTS Base**. In the generated configuration editor:

* Confirm both NVMe drives are present as `DRIVE1` and `DRIVE2`.
* Set software RAID on.
* Set RAID level to `1`.
* Set the hostname to `code.naudecoreventures.com`.
* Keep the EFI/boot entries generated by `installimage`; do not manually remove Hetzner's generated ESP configuration.
* Use a simple filesystem plan. The intended logical layout is:

```text theme={null}
software RAID 1 across both NVMe drives
small boot/EFI partitions as generated by installimage
8 GB swap for emergency use
remaining mirrored space mounted at /
```

A typical non-UEFI portion of an `installimage` layout resembles:

```text theme={null}
SWRAID 1
SWRAIDLEVEL 1
PART swap swap 8G
PART /boot ext4 1G
PART / ext4 all
```

Treat this as intent, not a blind paste. Retain any EFI lines generated for this particular server.

Save the configuration and let `installimage` complete. Record the final generated configuration in the Jira ticket, excluding passwords.

Reboot:

```bash theme={null}
reboot
```

### 4.3 Verify RAID before installing GitLab

After booting the installed OS:

```bash theme={null}
cat /proc/mdstat
lsblk -f
sudo mdadm --detail --scan
sudo mdadm --detail /dev/md0 || true
sudo mdadm --detail /dev/md1 || true
sudo mdadm --detail /dev/md2 || true
df -hT
swapon --show
```

Every active RAID-1 array must show both members, normally represented as:

```text theme={null}
[UU]
```

Do not continue if an array is degraded, resync is failing, or a disk is absent. A running resync is acceptable only if both members are present and progress is healthy; let it complete before load testing.

Configure emergency-only swap behaviour:

```bash theme={null}
cat >/etc/sysctl.d/60-ncv-gitlab.conf <<'SYSCTL'
vm.swappiness=1
vm.dirty_background_ratio=5
vm.dirty_ratio=15
SYSCTL
sysctl --system
```

## 5. Create named administration access

Replace `ncvops` with the approved named operations account only if NCV has a different naming standard:

```bash theme={null}
adduser ncvops
usermod -aG sudo ncvops
install -d -m 0700 -o ncvops -g ncvops /home/ncvops/.ssh
install -m 0600 -o ncvops -g ncvops /dev/null /home/ncvops/.ssh/authorized_keys
```

Paste only approved Ed25519 public keys into:

```text theme={null}
/home/ncvops/.ssh/authorized_keys
```

Test in a second terminal:

```bash theme={null}
ssh ncvops@code.naudecoreventures.com
sudo -v
```

Keep the original root session open while hardening SSH.

Copy the supplied hardening file:

```bash theme={null}
sudo install -m 0644 config/sshd-hardening.conf \
  /etc/ssh/sshd_config.d/50-ncv-hardening.conf
sudo sshd -t
sudo systemctl reload ssh
```

Open a third new SSH session and verify key login and `sudo` again. Only then close the Rescue/root session.

## 6. Apply the operating-system baseline

The supplied script can install the baseline packages and UFW rules:

```bash theme={null}
sudo install -m 0755 scripts/initial-server-baseline.sh \
  /root/initial-server-baseline.sh
sudo cp config/sshd-hardening.conf /root/50-ncv-hardening.conf
sudo cp config/fail2ban-sshd.local /root/50-ncv-fail2ban-sshd.local
sudo /root/initial-server-baseline.sh
```

Alternatively, execute the equivalent steps manually:

```bash theme={null}
sudo apt update
sudo apt full-upgrade -y
sudo apt install -y \
  ca-certificates curl jq git git-lfs openssh-server tzdata perl \
  ufw fail2ban unattended-upgrades smartmontools mdadm restic chrony \
  apt-transport-https gnupg lsb-release
```

Enable services:

```bash theme={null}
sudo install -m 0644 config/fail2ban-sshd.local /etc/fail2ban/jail.d/50-ncv-sshd.local
sudo fail2ban-client -t
sudo systemctl enable --now ssh chrony smartmontools mdmonitor fail2ban
sudo systemctl enable --now unattended-upgrades
sudo fail2ban-client status sshd
```

Configure UFW:

```bash theme={null}
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp comment 'SSH'
sudo ufw allow 80/tcp comment 'HTTP and ACME'
sudo ufw allow 443/tcp comment 'GitLab and Registry HTTPS'
sudo ufw --force enable
sudo ufw status verbose
```

In Hetzner Robot Firewall, permit the same inbound ports. When possible, restrict SSH/22 to NCV office/VPN/admin source addresses while leaving 80 and 443 public. Apply Robot Firewall changes only while a verified session and Rescue fallback are available.

Do **not** expose these internal services publicly:

* PostgreSQL.
* Redis.
* Gitaly.
* Prometheus internal endpoint.
* Container Registry's internal port.
* Docker daemon/socket.

Check listening ports:

```bash theme={null}
sudo ss -lntup
```

## 7. Configure hostname and time

Keep the OS clock in UTC:

```bash theme={null}
sudo timedatectl set-timezone UTC
sudo hostnamectl set-hostname code.naudecoreventures.com
hostnamectl
chronyc tracking || timedatectl timesync-status
```

Add a stable hostname mapping if it is not already present:

```text theme={null}
<SERVER_IPV4> code.naudecoreventures.com code
```

GitLab itself is configured to display `Africa/Lusaka` time.

## 8. Install GitLab EE in Free mode

The Enterprise Edition package is recommended so NCV can activate paid features later without migrating editions. Without a paid license, it operates with the Free feature set.

Add the official package repository:

```bash theme={null}
curl --location \
  'https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh' \
  --output /tmp/gitlab-ee-repository.sh
less /tmp/gitlab-ee-repository.sh
sudo bash /tmp/gitlab-ee-repository.sh
```

Install GitLab:

```bash theme={null}
sudo EXTERNAL_URL='https://code.naudecoreventures.com' apt install -y gitlab-ee
```

Record the exact installed version because restores require the same GitLab version and edition:

```bash theme={null}
dpkg-query -W -f='${Version}\n' gitlab-ee
sudo gitlab-rake gitlab:env:info
```

### 8.1 Apply the GitLab configuration

Back up the generated file:

```bash theme={null}
sudo cp -a /etc/gitlab/gitlab.rb \
  /etc/gitlab/gitlab.rb.before-ncv.$(date -u +%Y%m%dT%H%M%SZ)
```

Copy the supplied baseline:

```bash theme={null}
sudo install -m 0600 config/gitlab.rb.example /etc/gitlab/gitlab.rb
sudo editor /etc/gitlab/gitlab.rb
```

Replace every `CHANGE-ME` value. Do not enable SMTP until the real non-secret settings are known.

Apply configuration:

```bash theme={null}
sudo gitlab-ctl reconfigure
sudo gitlab-ctl status
sudo gitlab-rake gitlab:check SANITIZE=true
```

Retrieve the initial root password before GitLab removes the file after 24 hours:

```bash theme={null}
sudo cat /etc/gitlab/initial_root_password
```

Sign in as `root`, immediately:

1. Change the root password.
2. Change the root email to a monitored recovery address.
3. Enrol 2FA.
4. Store recovery codes in NCV's approved password vault.
5. Create a second named administrator and enrol that account in 2FA.

Never store the root password in Jira or source control.

### 8.2 Configure SMTP securely

Edit non-secret SMTP values in `/etc/gitlab/gitlab.rb`, set:

```ruby theme={null}
gitlab_rails['smtp_enable'] = true
```

Keep `smtp_user_name` and `smtp_password` out of `gitlab.rb`. Store them in GitLab's encrypted SMTP settings:

```bash theme={null}
sudo gitlab-rake gitlab:smtp:secret:edit EDITOR=vim
```

Enter:

```yaml theme={null}
user_name: '<SMTP USER>'
password: '<SMTP PASSWORD>'
```

Reconfigure and send a test message:

```bash theme={null}
sudo gitlab-ctl reconfigure
sudo gitlab-rails console
```

In the console:

```ruby theme={null}
Notify.test_email('approved-recipient@example.com', 'NCV GitLab SMTP test', 'SMTP works').deliver_now
```

Exit with `quit`. Verify SPF, DKIM, and DMARC for the sender domain with the mail provider.

## 9. Validate TLS and the Container Registry

Verify certificates:

```bash theme={null}
curl -fsS https://code.naudecoreventures.com/-/readiness
curl -I https://registry.code.naudecoreventures.com/v2/
```

A `401 Unauthorized` response from `/v2/` is normal before Docker authentication.

From a developer machine with Docker:

```bash theme={null}
docker login registry.code.naudecoreventures.com
```

Then use a temporary test project and push a small image. Delete the test project after verification.

## 10. Apply GitLab administrative settings

In **Admin**, configure and record screenshots/evidence for these settings:

### Accounts and authentication

* Disable new user account creation.
* Require 2FA for all human users, with a short controlled grace period.
* Keep password authentication enabled as emergency recovery unless a tested external identity provider replaces it.
* Set an NCV support/contact address.
* Restrict or disable user-created top-level groups.
* Restrict project creation to approved roles/groups.
* Review session duration and inactive-user policy.

### Visibility and access

* Default new project visibility: **Private**.
* Default new snippet visibility: **Private**.
* Disable public snippets unless explicitly needed.
* Do not allow arbitrary external users.
* Do not make source projects public to facilitate desktop updates.

### CI/CD

* Disable instance runners for projects by default until explicitly assigned.
* Prevent untrusted projects and forks from using the co-located runner.
* Set a sensible maximum job timeout, initially 60 minutes.
* Configure artifact expiry defaults; release binaries belong in Package Registry, not short-lived job artifacts.

### Package controls

* Source projects remain private.
* Only release-only projects may enable **Allow anyone to pull from package registry**.
* Protect container tags representing production releases.
* Establish cleanup policies for temporary CI images while retaining immutable release images.

## 11. Create the NCV group structure

Create:

```text theme={null}
ncv/
├── infrastructure/
│   └── gitlab-platform
├── platform/
│   ├── ci-templates
│   ├── ci-images
│   └── release-schemas
└── products/
    ├── <product-one>
    └── <product-two>

ncv-releases/
├── <product-one>
└── <product-two>
```

Recommended visibility:

* `ncv`: private.
* `ncv-releases`: private by default.
* Individual release projects: private with anonymous package pulls enabled, or public only after verifying that no source, internal notes, credentials, or customer data are present.

Role allocation:

| Role | Intended people |
| - | - |
| Owner | Very small platform-administration set |
| Maintainer | Technical leads/release managers |
| Developer | Engineers |
| Reporter | Read-only technical/management users |
| Guest | Avoid for private engineering data unless a precise use exists |

Protect `main`, `release/*`, `v*`, and `candidate-*`. Developers must not push directly to `main` or create stable release tags.

## 12. Install Docker and the constrained Linux runner

GitLab recommends separate runner hosts for production security and performance. Co-location here is an intentional low-volume starting compromise. Treat this runner as trusted-code-only.

Install Docker from Ubuntu's supported package and GitLab Runner from the official GitLab repository:

```bash theme={null}
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

curl -L \
  'https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh' \
  -o /tmp/gitlab-runner-repository.sh
less /tmp/gitlab-runner-repository.sh
sudo bash /tmp/gitlab-runner-repository.sh
sudo apt install -y gitlab-runner
sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner
```

Confirm cgroups v2 so Docker limits are enforced:

```bash theme={null}
stat -fc %T /sys/fs/cgroup
```

Expected:

```text theme={null}
cgroup2fs
```

### 12.1 Create the runner using the current workflow

In GitLab:

1. Open the intended group or Admin runner page.
2. Create a new runner.
3. Configure tags `linux-small` and `trusted`.
4. Do not run untagged jobs.
5. Lock it to trusted NCV projects/group.
6. Set it as protected if only protected refs should access protected variables.
7. Copy the runner authentication token beginning with `glrt-`.

Register it:

```bash theme={null}
sudo gitlab-runner register \
  --non-interactive \
  --url 'https://code.naudecoreventures.com' \
  --token '<GLRT-RUNNER-AUTHENTICATION-TOKEN>' \
  --name 'ncv-gitlab-host-linux-small' \
  --executor 'docker' \
  --docker-image 'debian:bookworm-slim'
```

Do not use deprecated registration tokens.

Back up the generated runner configuration, then merge in the limits from `config/gitlab-runner-config.toml.example`, preserving the real token:

```bash theme={null}
sudo cp -a /etc/gitlab-runner/config.toml \
  /etc/gitlab-runner/config.toml.before-limits.$(date -u +%Y%m%dT%H%M%SZ)
sudo editor /etc/gitlab-runner/config.toml
sudo gitlab-runner verify
sudo systemctl restart gitlab-runner
```

The required constraints are:

```text theme={null}
concurrent = 1
runner limit = 1
2 CPU maximum per job
8 GB memory maximum per job
privileged = false
only /cache mounted
no Docker socket passed into CI jobs
```

Run a test pipeline with two jobs. One must run while the second queues. During the test, observe:

```bash theme={null}
htop
sudo gitlab-ctl status
curl -fsS https://code.naudecoreventures.com/-/readiness
```

If GitLab latency becomes unacceptable, pause this runner and move it to a separate machine. Do not increase concurrency on this CPU merely because 64 GB RAM is available.

## 13. Install offsite backups before migration

Install the supplied files:

```bash theme={null}
sudo install -d -m 0700 /root/.config
sudo install -m 0600 config/restic-backup.env.example \
  /root/.config/ncv-gitlab-backup.env
sudo editor /root/.config/ncv-gitlab-backup.env

sudo install -m 0755 scripts/backup-gitlab.sh \
  /usr/local/sbin/ncv-gitlab-backup
sudo install -m 0755 scripts/check-gitlab-backups.sh \
  /usr/local/sbin/ncv-gitlab-backup-check
sudo install -m 0644 config/gitlab-backup.service \
  /etc/systemd/system/gitlab-backup.service
sudo install -m 0644 config/gitlab-backup.timer \
  /etc/systemd/system/gitlab-backup.timer
sudo systemctl daemon-reload
```

Initialize the Restic repository once:

```bash theme={null}
sudo bash -c 'set -a; source /root/.config/ncv-gitlab-backup.env; set +a; RESTIC_REPOSITORY="$RESTIC_REPOSITORY" RESTIC_PASSWORD="$RESTIC_PASSWORD" RESTIC_SFTP_COMMAND="${RESTIC_SFTP_COMMAND:-}" restic snapshots >/dev/null 2>&1 || RESTIC_REPOSITORY="$RESTIC_REPOSITORY" RESTIC_PASSWORD="$RESTIC_PASSWORD" RESTIC_SFTP_COMMAND="${RESTIC_SFTP_COMMAND:-}" restic init'
sudo bash -c 'set -a; source /root/.config/ncv-gitlab-backup.env; set +a; RESTIC_REPOSITORY="$RESTIC_CONFIG_REPOSITORY" RESTIC_PASSWORD="$RESTIC_CONFIG_PASSWORD" RESTIC_SFTP_COMMAND="${RESTIC_CONFIG_SFTP_COMMAND:-}" restic snapshots >/dev/null 2>&1 || RESTIC_REPOSITORY="$RESTIC_CONFIG_REPOSITORY" RESTIC_PASSWORD="$RESTIC_CONFIG_PASSWORD" RESTIC_SFTP_COMMAND="${RESTIC_CONFIG_SFTP_COMMAND:-}" restic init'
```

Run and inspect the first backup:

```bash theme={null}
sudo systemctl start gitlab-backup.service
sudo journalctl -u gitlab-backup.service --since today --no-pager
sudo /usr/local/sbin/ncv-gitlab-backup-check
```

Enable the nightly timer only after the manual run succeeds:

```bash theme={null}
sudo systemctl enable --now gitlab-backup.timer
systemctl list-timers gitlab-backup.timer
```

A backup is not accepted until it has been restored on a temporary host using the exact same GitLab version and edition.

## 14. Install health checking

```bash theme={null}
sudo install -m 0755 scripts/gitlab-health-check.sh \
  /usr/local/sbin/ncv-gitlab-health-check
sudo /usr/local/sbin/ncv-gitlab-health-check
```

Configure Hetzner SysMon or NCV monitoring to check:

```text theme={null}
https://code.naudecoreventures.com/-/readiness
```

Also monitor:

* RAID state.
* SMART/NVMe errors.
* Root filesystem usage.
* GitLab service state.
* Backup age and Restic snapshot success.
* TLS expiry.
* Runner queue duration.
* CPU steal/load and I/O wait.

## 15. Connect Jira Cloud

Use GitLab's Jira Cloud app, not the old Jira Cloud DVCS connector.

1. Confirm GitLab is publicly reachable with a trusted HTTPS certificate.
2. In GitLab Admin, create the OAuth application required by the GitLab for Jira Cloud app.
3. Install the official GitLab for Jira Cloud app in Atlassian.
4. Link the `ncv` group or relevant subgroup.
5. Create a branch, commit, merge request, pipeline, and test deployment referencing a real Jira key.
6. Verify branches, commits, merge requests, builds, and deployments appear in the Jira issue development panel.

The exact configuration and developer naming rules are in `docs/04-developer-guide.md`.

## 16. Final installation acceptance gate

Do not begin Azure migration until all items pass:

```text theme={null}
[ ] DNS resolves correctly from outside NCV.
[ ] GitLab and Registry have valid certificates.
[ ] RAID arrays show both members healthy.
[ ] Named key-only administrator access works.
[ ] Root SSH and password SSH are disabled.
[ ] UFW and Robot Firewall expose only 22/80/443 as intended.
[ ] GitLab services and gitlab:check pass.
[ ] Root account and second administrator use 2FA.
[ ] Public sign-up is disabled.
[ ] SMTP invitation/reset email works.
[ ] Container Registry push/pull test works.
[ ] One runner job runs and the second queues.
[ ] Offsite encrypted backup completes.
[ ] A restore test has been completed on the same GitLab version/edition.
[ ] Jira receives a complete test development flow.
[ ] No source project is public.
[ ] No permanent customer credential has been created.
```

## 17. Immediate rollback procedures

### DNS rollback

Restore the previous DNS records or remove the new records. With the temporary 300-second TTL, propagation should be limited.

### SSH/firewall rollback

Use Hetzner Rescue or KVM, mount the root filesystem, remove the offending SSH include or disable UFW, and reboot. Never improvise firewall changes without Rescue access.

### GitLab configuration rollback

```bash theme={null}
sudo cp /etc/gitlab/gitlab.rb.before-ncv.<timestamp> /etc/gitlab/gitlab.rb
sudo gitlab-ctl reconfigure
```

### Runner rollback

Pause the runner in GitLab or stop it locally:

```bash theme={null}
sudo systemctl stop gitlab-runner
```

GitLab remains available independently.

### Failed GitLab upgrade rollback

Restore a backup made on the **exact version and edition** being restored. Database migrations make package downgrade without a matching backup unsafe.

## 18. Official references

* GitLab Linux package on Ubuntu: [https://docs.gitlab.com/install/package/ubuntu/](https://docs.gitlab.com/install/package/ubuntu/)
* GitLab Linux package overview: [https://docs.gitlab.com/install/package/](https://docs.gitlab.com/install/package/)
* GitLab installation requirements: [https://docs.gitlab.com/install/requirements/](https://docs.gitlab.com/install/requirements/)
* GitLab TLS: [https://docs.gitlab.com/omnibus/settings/ssl/](https://docs.gitlab.com/omnibus/settings/ssl/)
* GitLab Container Registry administration: [https://docs.gitlab.com/administration/packages/container\_registry/](https://docs.gitlab.com/administration/packages/container_registry/)
* GitLab Runner installation and registration: [https://docs.gitlab.com/runner/install/linux-repository/](https://docs.gitlab.com/runner/install/linux-repository/) and [https://docs.gitlab.com/runner/register/](https://docs.gitlab.com/runner/register/)
* New runner creation workflow: [https://docs.gitlab.com/ci/runners/new\_creation\_workflow/](https://docs.gitlab.com/ci/runners/new_creation_workflow/)
* GitLab backup and restore: [https://docs.gitlab.com/administration/backup\_restore/](https://docs.gitlab.com/administration/backup_restore/)
* Hetzner installimage: [https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/](https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/)
* Hetzner software RAID drive replacement: [https://docs.hetzner.com/robot/dedicated-server/raid/exchanging-hard-disks-in-a-software-raid/](https://docs.hetzner.com/robot/dedicated-server/raid/exchanging-hard-disks-in-a-software-raid/)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.