> ## Documentation Index
> Fetch the complete documentation index at: https://deployment.naudecoreventures.com/llms.txt
> Use this file to discover all available pages before exploring further.

# NCV GitLab Delivery Platform Implementation Plan

> Ordered implementation plan with acceptance and rollback checks

# NCV GitLab Delivery Platform Implementation Plan

> **For agentic workers:** Execute tasks in order. Every task ends with an independently verifiable result. Do not proceed through a failed acceptance gate.

**Goal:** Establish `code.naudecoreventures.com` as Naudé Core Ventures' self-managed GitLab engineering and release platform, migrate Azure DevOps repositories and pipelines, integrate Jira, and implement immutable Experimental/Beta/Stable release channels.

**Architecture:** One GitLab Linux-package node on the supplied Hetzner RAID-1 server, one constrained co-located Linux Docker runner, external offsite backups, and separate Windows/macOS runners. Source projects publish immutable packages to release-only projects; channel promotion changes signed/verified manifests without rebuilding artifacts.

**Tech stack:** Ubuntu 24.04 LTS, GitLab EE in Free mode, bundled PostgreSQL/Redis/Gitaly/NGINX, GitLab Runner Docker executor, GitLab Generic Packages, Container Registry, GitLab Releases, Jira Cloud, Bash, JSON/JQ, Java/Gradle/Maven, Flutter, PowerShell, and optional Rust/native desktop updater helper.

## Global constraints

* Public GitLab URL is exactly `https://code.naudecoreventures.com`.
* Container Registry URL is `https://registry.code.naudecoreventures.com`.
* OS is Ubuntu 24.04 LTS.
* Both NVMe drives use software RAID 1.
* GitLab is the authoritative source/build/release platform; Jira remains the work-management source.
* No plaintext production secret is committed to Git.
* The Linux runner on the GitLab host has concurrency one and is restricted to trusted NCV projects.
* The supplied CPU is treated as a constrained pilot below GitLab's current single-node baseline in physical-core terms; runner separation is the first scale action.
* Windows native builds run on Windows; Apple builds run on macOS.
* Promotion between release channels never rebuilds or re-signs the artifact.
* Backups leave the physical server and are restore-tested.
* Azure is placed read-only only after repository, LFS, pipeline, package, and deployment verification passes.

## Jira delivery model recommendation (plan-only)

**Recommended project type:** Jira Software, company-managed, using the existing `DEV` project only after its settings, permissions, and issue-type scheme are verified. Do not create a second project for this program unless the existing project cannot provide the required controls.

**Recommended board:** One Kanban board filtered to the GitLab delivery component and the label `ncv-gitlab-deployment`. Use columns `Backlog`, `Ready`, `In Progress`, `Validation`, `Blocked`, and `Done`. Kanban is preferred over Scrum because the program is dependency-led, has infrastructure gates, and does not require sprint commitments. Apply WIP limits to `In Progress` and `Validation`; keep blocked work visible rather than hiding it in a separate queue.

**Issue-type hierarchy:** Jira Cloud's standard hierarchy is Epic at level 1, standard issues at level 0, and Sub-task at level -1. Therefore:

```text theme={null}
NCV GitLab Delivery Platform (program record in the project)
└── Epic = Workstream
    └── Task / Story / Bug
        └── Sub-task
```

Do not introduce a custom `Workstream` issue type for this plan. Represent each workstream as an Epic, use Tasks for executable delivery units, use Sub-tasks for bounded verification or configuration steps, and use Bugs for observed failures linked to the affected Task. Use issue links for dependencies such as `blocks` and `is blocked by`; do not encode dependency order only in text.

**Required fields on every Task:** owner, reviewer, due date, dependency links, acceptance criteria, verification evidence, rollback procedure, affected GitLab group/project, and a single program label. Keep secrets as vault references only; Jira descriptions and comments must not contain secret values.

### Proposed Workstream/Epic map

| Workstream Epic | Tasks | Purpose |
| - | -: | - |
| Program governance and access | 1 | Establish the implementation record, owner/reviewer matrix, access inventory, and rollback contacts. |
| GitLab platform foundation | 2–6 | Prepare DNS, install Ubuntu/RAID, apply the OS baseline, install GitLab, and configure identity, permissions, and project structure. |
| Resilience and execution infrastructure | 7–8, 16 | Establish offsite backup and restore evidence, install the constrained Linux runner, and add native Windows/macOS runners. |
| Jira integration and delivery visibility | 9 | Connect GitLab to Jira and verify branch, commit, merge-request, pipeline, and deployment evidence. |
| CI, release channels, and endpoint delivery | 10, 17 | Build immutable CI/release channels and evaluate the endpoint updater against security and recovery requirements. |
| Azure migration and operational cutover | 11–15, 18 | Inventory Azure, migrate the pilot, validate and cut over pipelines, migrate remaining repositories in waves, and complete operational handover. |

Each numbered implementation task becomes one Jira Task under exactly one Epic. Verification commands, evidence capture, and rollback checks become Sub-tasks only when they can be completed independently; the parent Task remains open until all required evidence is attached and the acceptance statement passes.

***

## Task 1: Establish the implementation record and input inventory

**Produces:** A Jira implementation epic, owner matrix, access inventory, and rollback contacts.

* [ ] Create a Jira epic titled `Implement NCV GitLab Delivery Platform`.
* [ ] Create work items for infrastructure, security, backup, Jira integration, Azure inventory, repository migration, pipeline migration, runner setup, release management, endpoint updater POC, and cutover.
* [ ] Assign a named owner and reviewer to every work item.
* [ ] Record the supplied Hetzner Auction ID `3051857`, monthly price, datacenter, IPv4 address, rescue access owner, and support phone password location.
* [ ] Record DNS administrator, GitLab administrator, Azure DevOps organization administrator, Jira Cloud administrator, SMTP administrator, backup-storage administrator, Windows signing owner, and Apple account owner.
* [ ] Create an encrypted secret inventory containing only locations/references, never secret values in Jira.

**Acceptance:** Every later task has an owner and the required administration access is confirmed.

**Rollback:** None; this task changes no infrastructure.

## Task 2: Prepare DNS before installation

**Produces:** Valid public DNS for GitLab and registry.

* [ ] Create `A` record `code.naudecoreventures.com` pointing to the server IPv4.
* [ ] Create `A` record `registry.code.naudecoreventures.com` pointing to the same IPv4.
* [ ] Create `AAAA` records only if IPv6 is deliberately configured and firewalled.
* [ ] Set TTL to 300 seconds during implementation; raise it after go-live.
* [ ] Verify authoritative resolution from at least two external resolvers.
* [ ] Confirm ports 80 and 443 can reach the server for Let's Encrypt HTTP validation.

**Verification commands:**

```bash theme={null}
dig +short A code.naudecoreventures.com
dig +short A registry.code.naudecoreventures.com
```

**Acceptance:** Both names resolve to the intended server IP from outside NCV.

**Rollback:** Remove or restore the previous DNS records.

## Task 3: Install Ubuntu with RAID 1

**Produces:** A clean, mirrored Ubuntu 24.04 LTS host.

* [ ] Activate Hetzner Rescue System and reboot.
* [ ] Connect as root using the one-time rescue password.
* [ ] Record detected NVMe device names and SMART status.
* [ ] Run Hetzner `installimage`.
* [ ] Select Ubuntu 24.04 LTS minimal/base image.
* [ ] Set hostname `code.naudecoreventures.com`.
* [ ] Enable software RAID and set RAID level 1 across both drives.
* [ ] Use ext4 and leave enough free workspace for GitLab backup generation.
* [ ] Complete installation, reboot, and verify the server boots from either mirror member.
* [ ] Verify `/proc/mdstat`, `mdadm --detail`, filesystem capacity, and both NVMe devices.

**Acceptance:** RAID is healthy, the filesystem is mounted from RAID, and the effective capacity is approximately one 512 GB drive.

**Rollback:** Re-enter Rescue System and rerun `installimage`; no production data exists yet.

## Task 4: Apply the operating-system security baseline

**Produces:** Key-only administrative access, firewalls, patching, time synchronization, and hardware monitoring.

* [ ] Create a named sudo administrator account.
* [ ] Install the administrator's Ed25519 public key.
* [ ] Open a second SSH session and verify sudo before changing root/password settings.
* [ ] Disable SSH password authentication and direct root login.
* [ ] Configure UFW for SSH, HTTP, and HTTPS; do not expose PostgreSQL, Redis, Gitaly, or Registry internal ports.
* [ ] Mirror equivalent allow rules in Hetzner Robot Firewall without locking out the active SSH source.
* [ ] Install and enable `fail2ban`, `unattended-upgrades`, `smartmontools`, `mdadm`, `jq`, `curl`, `ca-certificates`, and `chrony` or systemd time synchronization.
* [ ] Configure `mdadm` and SMART failure notifications.
* [ ] Configure Hetzner SysMon for HTTPS and optionally SSH.
* [ ] Keep the OS clock in UTC; configure GitLab's display timezone as `Africa/Lusaka`.

**Acceptance:** Password SSH fails, key SSH succeeds, only approved public ports listen, automatic security updates are active, and RAID/SMART checks report healthy.

**Rollback:** Use Hetzner Rescue System or KVM to restore SSH/firewall configuration.

## Task 5: Install and configure GitLab

**Files used:** `config/gitlab.rb.example`, `config/sshd-hardening.conf`, `scripts/gitlab-health-check.sh`.

**Produces:** A healthy GitLab instance at the required URL.

* [ ] Install GitLab EE from the official Linux package repository without activating a paid license.
* [ ] Set `EXTERNAL_URL=https://code.naudecoreventures.com`.
* [ ] Apply the supplied `gitlab.rb` baseline.
* [ ] Enable automatic Let's Encrypt certificates for GitLab and the registry.
* [ ] Disable public sign-up and require administrator-created accounts.
* [ ] Configure outbound SMTP and verify invitation/password-reset delivery.
* [ ] Enable the Container Registry at `registry.code.naudecoreventures.com`.
* [ ] Run `gitlab-ctl reconfigure` and `gitlab-ctl status`.
* [ ] Run `gitlab-rake gitlab:check SANITIZE=true`.
* [ ] Change the initial root password and place it in NCV's approved password vault.
* [ ] Create a second administrator account and verify recovery access.

**Acceptance:** HTTPS is valid, login works, email works, the health check passes, and `docker login registry.code.naudecoreventures.com` reaches the registry authentication flow.

**Rollback:** Restore `/etc/gitlab` configuration backup or reinstall the same GitLab version before production data is added.

## Task 6: Configure identity, permissions, and project structure

**Produces:** Standardized NCV groups and least-privilege access.

* [ ] Enforce 2FA for all human users.
* [ ] Disable user-created top-level groups and unrestricted project creation.
* [ ] Create private top-level group `ncv`.
* [ ] Create subgroups `ncv/platform`, `ncv/infrastructure`, and `ncv/products`.
* [ ] Create a release-only top-level group `ncv-releases`; make only its intended release projects public.
* [ ] Create `ncv/platform/ci-templates` and tag the first template version `v1.0.0`.
* [ ] Create `ncv/infrastructure/gitlab-platform` and commit this handbook after removing any secrets.
* [ ] Define roles: Owner for a very small administrative set, Maintainer for technical leads, Developer for engineers, Reporter for read-only management/audit users.
* [ ] Protect `main`, `release/*`, `v*`, and `candidate-*` refs.
* [ ] Prevent direct pushes to `main`; merge through merge requests only.

**Acceptance:** A Developer cannot push to `main`, create a stable tag, or alter group settings.

**Rollback:** Group owners can correct memberships and branch/tag rules; no code migration is required yet.

## Task 7: Implement offsite backup before migration

**Files used:** `scripts/backup-gitlab.sh`, `config/gitlab-backup.service`, `config/gitlab-backup.timer`.

**Produces:** Encrypted offsite backups and a proven restore process.

* [ ] Provision two encrypted offsite Restic repositories: one for application backups and one, with separate access/password, for GitLab configuration and decryption secrets.
* [ ] Store both repository locations and separate credentials in `/root/.config/ncv-gitlab-backup.env` with mode `0600`; escrow recovery credentials outside the GitLab host.
* [ ] Install Restic.
* [ ] Install the supplied backup script and systemd service/timer.
* [ ] Run one manual GitLab application backup.
* [ ] Run `gitlab-ctl backup-etc` and include `/etc/gitlab/config_backup`, TLS material, and SSH host keys in encrypted offsite backup.
* [ ] Verify both remote Restic repositories and perform `ncv-gitlab-backup-check --check`.
* [ ] Build a temporary restore host or VM with the exact same GitLab version and edition.
* [ ] Restore application backup and GitLab secrets/configuration.
* [ ] Clone a test repository, inspect a release/package, and run a login test.
* [ ] Record restore evidence in Jira.

**Acceptance:** An offsite snapshot exists and an exact-version restore has been demonstrated.

**Rollback:** Disable the timer and remove credentials if backup configuration is incorrect; do not proceed with migration.

## Task 8: Install the constrained Linux runner

**Files used:** `config/gitlab-runner-config.toml.example`.

**Produces:** One trusted Docker runner for Java, Flutter checks, Linux builds, and deployment scripts.

* [ ] Install Docker Engine and GitLab Runner.
* [ ] Register an instance/group runner tagged `linux-small` and `trusted`.
* [ ] Use the Docker executor with unprivileged containers.
* [ ] Set global `concurrent = 1` and runner `limit = 1`.
* [ ] Limit each job to two CPUs and 8 GB memory.
* [ ] Enable per-build Docker networks.
* [ ] Persist only the runner cache directory.
* [ ] Do not mount `/var/run/docker.sock` into general jobs.
* [ ] Do not allow forks or untrusted public projects to use this runner.
* [ ] Run a CPU/memory test job while observing GitLab response time.

**Acceptance:** One job runs; a second queues; GitLab remains responsive; jobs cannot read host GitLab directories.

**Rollback:** Pause or unregister the runner. GitLab remains available independently.

## Task 9: Connect Jira Cloud

**Produces:** Jira development-panel visibility for GitLab activity.

* [ ] Confirm GitLab is public over HTTPS with a complete trusted certificate chain.
* [ ] Install/configure the GitLab for Jira Cloud app using an approved Jira administrator.
* [ ] Connect the intended GitLab groups/projects.
* [ ] Configure project-level Jira issue integration where required.
* [ ] Create a test branch containing a real Jira key.
* [ ] Commit, push, open an MR, run a pipeline, and create a test deployment.
* [ ] Confirm Jira shows branch, commit, MR, pipeline, and deployment data.
* [ ] Publish the mandatory naming convention to developers.

**Acceptance:** A Jira issue displays live development information from the self-managed GitLab instance.

**Rollback:** Remove the GitLab/Jira app connection; repositories and pipelines remain unaffected.

## Task 10: Create CI templates and the release model

**Files used:** `ci/templates/*`, `scripts/publish-generic-package.sh`, `scripts/promote-channel.sh`, `templates/channel-manifest.schema.json`, and `templates/release-notes.md`.

**Produces:** Reusable CI and an immutable release lifecycle.

* [ ] Commit the templates, helper scripts, and `ci/images/release-tools/Dockerfile` to `ncv/platform/ci-templates`.
* [ ] Build, scan, verify, and publish `release-tools:v1.0.0` to the NCV Container Registry; record and later pin its digest.
* [ ] Protect template tags; projects include templates by immutable tag, not `main`.
* [ ] Create one `ncv-releases/<pilot-product>` project containing only release material.
* [ ] Configure its Package Registry for the intended anonymous/authenticated download policy.
* [ ] Protect the release project default `main` branch; store manifests beneath `channels/`.
* [ ] Configure cross-project publish credentials using a job token allowlist or narrowly scoped project access token.
* [ ] Publish a test immutable package.
* [ ] Create a GitLab Release with release notes and package asset links.
* [ ] Publish Experimental manifest pointing to the package.
* [ ] Promote the exact SHA to Beta, then Stable, without rebuilding.
* [ ] Verify each manifest, URL, package checksum, and GitLab release entry.

**Acceptance:** Experimental, Beta, and Stable point to the same artifact URL and SHA-256.

**Rollback:** Revert a channel-manifest commit to the prior known-good package; never delete the known-good package during rollback.

## Task 11: Inventory Azure DevOps

**Produces:** Complete migration inventory and wave plan.

* [ ] List organizations, projects, repositories, default branches, branch policies, tags, Git LFS usage, submodules, and repository sizes.
* [ ] List YAML and classic pipelines, schedules, triggers, environments, approvals, agent pools, artifacts, caches, variable groups, secure files, and service connections.
* [ ] List Azure Artifacts feeds and package consumers/producers.
* [ ] List webhooks, status checks, deployment scripts, and external integrations.
* [ ] Classify repositories into pilot, standard, high-risk, archived, and retained-read-only.
* [ ] Record every production deployment pipeline and its rollback method.
* [ ] Choose migration waves and freeze owners.

**Acceptance:** No repository or operational dependency is unassigned.

**Rollback:** None; inventory is read-only.

## Task 12: Migrate the pilot repository

**Files used:** `scripts/migrate-azure-repo.sh`, `scripts/verify-repo-migration.sh`.

**Produces:** A GitLab repository with all Git refs and LFS objects.

* [ ] Create empty destination project under `ncv/products`.
* [ ] Mirror-clone the Azure repository.
* [ ] Fetch all Git LFS objects.
* [ ] Push all refs to GitLab.
* [ ] Push all LFS objects to GitLab.
* [ ] Compare branch and tag sets.
* [ ] Compare the default branch tip SHA.
* [ ] Clone the GitLab repository into a clean directory and build locally.
* [ ] Keep Azure PR/build history read-only; do not pretend Git mirror migration moved it.

**Acceptance:** Refs, tags, LFS objects, submodules, and a clean build match the Azure source.

**Rollback:** Continue using Azure as canonical; delete or archive the incomplete GitLab project.

## Task 13: Convert and validate the pilot pipeline

**Produces:** GitLab CI parity with the Azure pipeline.

* [ ] Map Azure stages/jobs/steps to GitLab stages/jobs/scripts.
* [ ] Map Azure agent pools to GitLab runner tags.
* [ ] Recreate variables in GitLab group/project CI/CD settings; never copy secrets into YAML.
* [ ] Recreate secure files in GitLab Secure Files or approved vault delivery.
* [ ] Replace Azure service connections with narrowly scoped GitLab/deployment credentials.
* [ ] Recreate schedules and manual inputs.
* [ ] Produce JUnit XML and coverage artifacts where supported.
* [ ] Build and deploy to a non-production test environment.
* [ ] Compare outputs with Azure by file name, size, version metadata, and checksum where deterministic.
* [ ] Run smoke and integration tests.
* [ ] Run the release pipeline through Experimental and Beta; do not deploy Stable yet.

**Acceptance:** The GitLab pipeline produces functionally equivalent tested artifacts and deployment results.

**Rollback:** Keep Azure production pipeline active; disable the GitLab production job.

## Task 14: Pilot cutover

**Produces:** One product operating fully from GitLab.

* [ ] Announce a repository freeze window in Jira and team communication.
* [ ] Disable merges in Azure for the pilot repository.
* [ ] Fetch and push the final delta to GitLab.
* [ ] Run migration verification again.
* [ ] Change developer remotes to GitLab.
* [ ] Enable GitLab as the only production pipeline.
* [ ] Run a tagged release and promote the exact artifact to Stable.
* [ ] Confirm package download, release notes, deployment, monitoring, and rollback.
* [ ] Mark Azure repository and pipeline read-only; retain history for the agreed retention period.
* [ ] Record cutover evidence and owner sign-off in Jira.

**Acceptance:** All new commits and releases originate from GitLab and the known-good rollback remains available.

**Rollback:** Re-enable Azure merge/deploy, restore developer remotes, and remove the GitLab Stable channel pointer created during the failed cutover.

## Task 15: Migrate remaining repositories in waves

**Produces:** Azure DevOps retirement without a big-bang migration.

* [ ] Repeat Tasks 12–14 by migration wave.
* [ ] Do not mix unrelated high-risk production systems in one wave.
* [ ] Validate backup growth and runner load after every wave.
* [ ] Add a separate Linux runner before increasing concurrency on the GitLab host.
* [ ] Archive rather than migrate dead repositories unless legal/business retention requires active hosting.
* [ ] Revoke Azure service connections and tokens only after dependent systems are verified.

**Acceptance:** The inventory has no active unaccounted repository, pipeline, feed, or credential.

**Rollback:** Per-project rollback, not whole-program rollback.

## Task 16: Implement Windows and Apple runners

**Produces:** Native cross-platform builds.

* [ ] Install a dedicated Windows runner tagged `windows-x64`.
* [ ] Install Flutter, Visual Studio Build Tools/workloads, installer tooling, and code-signing access.
* [ ] Run the runner under a restricted service account; protect the runner directory.
* [ ] Store signing credentials only in protected variables/secure files and expose them only on protected refs.
* [ ] Build, sign, package, and verify one Windows release.
* [ ] Install a dedicated macOS runner tagged `macos-arm64` when Apple delivery is required.
* [ ] Configure Xcode, Flutter, Fastlane, keychain access, App Store Connect API key, TestFlight groups, and signing.
* [ ] Verify macOS runner user-session and keychain behavior after reboot.

**Acceptance:** Native signed artifacts are produced only on their matching OS runners.

**Rollback:** Pause the affected runner and continue non-native CI on Linux.

## Task 17: Endpoint updater proof of concept

**Produces:** A go/no-go decision for automatic customer updates.

* [ ] Implement the updater behind an application-owned interface so the package can be replaced later.
* [ ] Validate channel selection from server policy rather than allowing ordinary customers to select Experimental.
* [ ] Check current version and fetch release notes.
* [ ] Download to a staging path and verify file length, SHA-256, and publisher signature.
* [ ] Test optional update, mandatory update, restart-now, restart-later, silent idle update, and failed update recovery.
* [ ] Test power/network interruption during download and install.
* [ ] Test an application process holding files open.
* [ ] Test downgrade/rollback policy.
* [ ] Test standard-user installation permissions.
* [ ] Test antivirus/EDR and Windows SmartScreen behavior with signed binaries.
* [ ] Test Linux executable permissions and publisher authenticity.
* [ ] Capture update success/failure telemetry without collecting customer secrets.

**Acceptance:** Security and recovery review approves the chosen updater and rollback path.

**Rollback:** Disable auto-update via server policy and distribute the previous signed installer manually.

## Task 18: Operational handover and go-live

**Produces:** A supportable platform rather than an undocumented installation.

* [ ] Run all checks in `docs/09-go-live-checklists.md`.
* [ ] Confirm backup and restore evidence is current.
* [ ] Confirm administrator break-glass access.
* [ ] Confirm monitoring and alert routing.
* [ ] Confirm upgrade owner and maintenance policy.
* [ ] Train developers using `docs/04-developer-guide.md`.
* [ ] Train release managers using `docs/07-release-package-update-guide.md`.
* [ ] Train infrastructure staff using the operations and DR runbooks.
* [ ] Store this handbook in `ncv/infrastructure/gitlab-platform` and tag the approved baseline.
* [ ] Close the Jira implementation epic only after owner and reviewer acceptance.

**Acceptance:** Operations, development, release management, and recovery each have a named trained owner and a verified runbook.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.