Skip to main content

NCV GitLab Delivery Platform Implementation Plan

For agentic workers: Execute tasks in order. Every task ends with an independently verifiable result. Do not proceed through a failed acceptance gate.
Goal: Establish code.naudecoreventures.com as Naudé Core Ventures’ self-managed GitLab engineering and release platform, migrate Azure DevOps repositories and pipelines, integrate Jira, and implement immutable Experimental/Beta/Stable release channels. Architecture: One GitLab Linux-package node on the supplied Hetzner RAID-1 server, one constrained co-located Linux Docker runner, external offsite backups, and separate Windows/macOS runners. Source projects publish immutable packages to release-only projects; channel promotion changes signed/verified manifests without rebuilding artifacts. Tech stack: Ubuntu 24.04 LTS, GitLab EE in Free mode, bundled PostgreSQL/Redis/Gitaly/NGINX, GitLab Runner Docker executor, GitLab Generic Packages, Container Registry, GitLab Releases, Jira Cloud, Bash, JSON/JQ, Java/Gradle/Maven, Flutter, PowerShell, and optional Rust/native desktop updater helper.

Global constraints

  • Public GitLab URL is exactly https://code.naudecoreventures.com.
  • Container Registry URL is https://registry.code.naudecoreventures.com.
  • OS is Ubuntu 24.04 LTS.
  • Both NVMe drives use software RAID 1.
  • GitLab is the authoritative source/build/release platform; Jira remains the work-management source.
  • No plaintext production secret is committed to Git.
  • The Linux runner on the GitLab host has concurrency one and is restricted to trusted NCV projects.
  • The supplied CPU is treated as a constrained pilot below GitLab’s current single-node baseline in physical-core terms; runner separation is the first scale action.
  • Windows native builds run on Windows; Apple builds run on macOS.
  • Promotion between release channels never rebuilds or re-signs the artifact.
  • Backups leave the physical server and are restore-tested.
  • Azure is placed read-only only after repository, LFS, pipeline, package, and deployment verification passes.

Jira delivery model recommendation (plan-only)

Recommended project type: Jira Software, company-managed, using the existing DEV project only after its settings, permissions, and issue-type scheme are verified. Do not create a second project for this program unless the existing project cannot provide the required controls. Recommended board: One Kanban board filtered to the GitLab delivery component and the label ncv-gitlab-deployment. Use columns Backlog, Ready, In Progress, Validation, Blocked, and Done. Kanban is preferred over Scrum because the program is dependency-led, has infrastructure gates, and does not require sprint commitments. Apply WIP limits to In Progress and Validation; keep blocked work visible rather than hiding it in a separate queue. Issue-type hierarchy: Jira Cloud’s standard hierarchy is Epic at level 1, standard issues at level 0, and Sub-task at level -1. Therefore:
Do not introduce a custom Workstream issue type for this plan. Represent each workstream as an Epic, use Tasks for executable delivery units, use Sub-tasks for bounded verification or configuration steps, and use Bugs for observed failures linked to the affected Task. Use issue links for dependencies such as blocks and is blocked by; do not encode dependency order only in text. Required fields on every Task: owner, reviewer, due date, dependency links, acceptance criteria, verification evidence, rollback procedure, affected GitLab group/project, and a single program label. Keep secrets as vault references only; Jira descriptions and comments must not contain secret values.

Proposed Workstream/Epic map

Each numbered implementation task becomes one Jira Task under exactly one Epic. Verification commands, evidence capture, and rollback checks become Sub-tasks only when they can be completed independently; the parent Task remains open until all required evidence is attached and the acceptance statement passes.

Task 1: Establish the implementation record and input inventory

Produces: A Jira implementation epic, owner matrix, access inventory, and rollback contacts.
  • Create a Jira epic titled Implement NCV GitLab Delivery Platform.
  • Create work items for infrastructure, security, backup, Jira integration, Azure inventory, repository migration, pipeline migration, runner setup, release management, endpoint updater POC, and cutover.
  • Assign a named owner and reviewer to every work item.
  • Record the supplied Hetzner Auction ID 3051857, monthly price, datacenter, IPv4 address, rescue access owner, and support phone password location.
  • Record DNS administrator, GitLab administrator, Azure DevOps organization administrator, Jira Cloud administrator, SMTP administrator, backup-storage administrator, Windows signing owner, and Apple account owner.
  • Create an encrypted secret inventory containing only locations/references, never secret values in Jira.
Acceptance: Every later task has an owner and the required administration access is confirmed. Rollback: None; this task changes no infrastructure.

Task 2: Prepare DNS before installation

Produces: Valid public DNS for GitLab and registry.
  • Create A record code.naudecoreventures.com pointing to the server IPv4.
  • Create A record registry.code.naudecoreventures.com pointing to the same IPv4.
  • Create AAAA records only if IPv6 is deliberately configured and firewalled.
  • Set TTL to 300 seconds during implementation; raise it after go-live.
  • Verify authoritative resolution from at least two external resolvers.
  • Confirm ports 80 and 443 can reach the server for Let’s Encrypt HTTP validation.
Verification commands:
Acceptance: Both names resolve to the intended server IP from outside NCV. Rollback: Remove or restore the previous DNS records.

Task 3: Install Ubuntu with RAID 1

Produces: A clean, mirrored Ubuntu 24.04 LTS host.
  • Activate Hetzner Rescue System and reboot.
  • Connect as root using the one-time rescue password.
  • Record detected NVMe device names and SMART status.
  • Run Hetzner installimage.
  • Select Ubuntu 24.04 LTS minimal/base image.
  • Set hostname code.naudecoreventures.com.
  • Enable software RAID and set RAID level 1 across both drives.
  • Use ext4 and leave enough free workspace for GitLab backup generation.
  • Complete installation, reboot, and verify the server boots from either mirror member.
  • Verify /proc/mdstat, mdadm --detail, filesystem capacity, and both NVMe devices.
Acceptance: RAID is healthy, the filesystem is mounted from RAID, and the effective capacity is approximately one 512 GB drive. Rollback: Re-enter Rescue System and rerun installimage; no production data exists yet.

Task 4: Apply the operating-system security baseline

Produces: Key-only administrative access, firewalls, patching, time synchronization, and hardware monitoring.
  • Create a named sudo administrator account.
  • Install the administrator’s Ed25519 public key.
  • Open a second SSH session and verify sudo before changing root/password settings.
  • Disable SSH password authentication and direct root login.
  • Configure UFW for SSH, HTTP, and HTTPS; do not expose PostgreSQL, Redis, Gitaly, or Registry internal ports.
  • Mirror equivalent allow rules in Hetzner Robot Firewall without locking out the active SSH source.
  • Install and enable fail2ban, unattended-upgrades, smartmontools, mdadm, jq, curl, ca-certificates, and chrony or systemd time synchronization.
  • Configure mdadm and SMART failure notifications.
  • Configure Hetzner SysMon for HTTPS and optionally SSH.
  • Keep the OS clock in UTC; configure GitLab’s display timezone as Africa/Lusaka.
Acceptance: Password SSH fails, key SSH succeeds, only approved public ports listen, automatic security updates are active, and RAID/SMART checks report healthy. Rollback: Use Hetzner Rescue System or KVM to restore SSH/firewall configuration.

Task 5: Install and configure GitLab

Files used: config/gitlab.rb.example, config/sshd-hardening.conf, scripts/gitlab-health-check.sh. Produces: A healthy GitLab instance at the required URL.
  • Install GitLab EE from the official Linux package repository without activating a paid license.
  • Set EXTERNAL_URL=https://code.naudecoreventures.com.
  • Apply the supplied gitlab.rb baseline.
  • Enable automatic Let’s Encrypt certificates for GitLab and the registry.
  • Disable public sign-up and require administrator-created accounts.
  • Configure outbound SMTP and verify invitation/password-reset delivery.
  • Enable the Container Registry at registry.code.naudecoreventures.com.
  • Run gitlab-ctl reconfigure and gitlab-ctl status.
  • Run gitlab-rake gitlab:check SANITIZE=true.
  • Change the initial root password and place it in NCV’s approved password vault.
  • Create a second administrator account and verify recovery access.
Acceptance: HTTPS is valid, login works, email works, the health check passes, and docker login registry.code.naudecoreventures.com reaches the registry authentication flow. Rollback: Restore /etc/gitlab configuration backup or reinstall the same GitLab version before production data is added.

Task 6: Configure identity, permissions, and project structure

Produces: Standardized NCV groups and least-privilege access.
  • Enforce 2FA for all human users.
  • Disable user-created top-level groups and unrestricted project creation.
  • Create private top-level group ncv.
  • Create subgroups ncv/platform, ncv/infrastructure, and ncv/products.
  • Create a release-only top-level group ncv-releases; make only its intended release projects public.
  • Create ncv/platform/ci-templates and tag the first template version v1.0.0.
  • Create ncv/infrastructure/gitlab-platform and commit this handbook after removing any secrets.
  • Define roles: Owner for a very small administrative set, Maintainer for technical leads, Developer for engineers, Reporter for read-only management/audit users.
  • Protect main, release/*, v*, and candidate-* refs.
  • Prevent direct pushes to main; merge through merge requests only.
Acceptance: A Developer cannot push to main, create a stable tag, or alter group settings. Rollback: Group owners can correct memberships and branch/tag rules; no code migration is required yet.

Task 7: Implement offsite backup before migration

Files used: scripts/backup-gitlab.sh, config/gitlab-backup.service, config/gitlab-backup.timer. Produces: Encrypted offsite backups and a proven restore process.
  • Provision two encrypted offsite Restic repositories: one for application backups and one, with separate access/password, for GitLab configuration and decryption secrets.
  • Store both repository locations and separate credentials in /root/.config/ncv-gitlab-backup.env with mode 0600; escrow recovery credentials outside the GitLab host.
  • Install Restic.
  • Install the supplied backup script and systemd service/timer.
  • Run one manual GitLab application backup.
  • Run gitlab-ctl backup-etc and include /etc/gitlab/config_backup, TLS material, and SSH host keys in encrypted offsite backup.
  • Verify both remote Restic repositories and perform ncv-gitlab-backup-check --check.
  • Build a temporary restore host or VM with the exact same GitLab version and edition.
  • Restore application backup and GitLab secrets/configuration.
  • Clone a test repository, inspect a release/package, and run a login test.
  • Record restore evidence in Jira.
Acceptance: An offsite snapshot exists and an exact-version restore has been demonstrated. Rollback: Disable the timer and remove credentials if backup configuration is incorrect; do not proceed with migration.

Task 8: Install the constrained Linux runner

Files used: config/gitlab-runner-config.toml.example. Produces: One trusted Docker runner for Java, Flutter checks, Linux builds, and deployment scripts.
  • Install Docker Engine and GitLab Runner.
  • Register an instance/group runner tagged linux-small and trusted.
  • Use the Docker executor with unprivileged containers.
  • Set global concurrent = 1 and runner limit = 1.
  • Limit each job to two CPUs and 8 GB memory.
  • Enable per-build Docker networks.
  • Persist only the runner cache directory.
  • Do not mount /var/run/docker.sock into general jobs.
  • Do not allow forks or untrusted public projects to use this runner.
  • Run a CPU/memory test job while observing GitLab response time.
Acceptance: One job runs; a second queues; GitLab remains responsive; jobs cannot read host GitLab directories. Rollback: Pause or unregister the runner. GitLab remains available independently.

Task 9: Connect Jira Cloud

Produces: Jira development-panel visibility for GitLab activity.
  • Confirm GitLab is public over HTTPS with a complete trusted certificate chain.
  • Install/configure the GitLab for Jira Cloud app using an approved Jira administrator.
  • Connect the intended GitLab groups/projects.
  • Configure project-level Jira issue integration where required.
  • Create a test branch containing a real Jira key.
  • Commit, push, open an MR, run a pipeline, and create a test deployment.
  • Confirm Jira shows branch, commit, MR, pipeline, and deployment data.
  • Publish the mandatory naming convention to developers.
Acceptance: A Jira issue displays live development information from the self-managed GitLab instance. Rollback: Remove the GitLab/Jira app connection; repositories and pipelines remain unaffected.

Task 10: Create CI templates and the release model

Files used: ci/templates/*, scripts/publish-generic-package.sh, scripts/promote-channel.sh, templates/channel-manifest.schema.json, and templates/release-notes.md. Produces: Reusable CI and an immutable release lifecycle.
  • Commit the templates, helper scripts, and ci/images/release-tools/Dockerfile to ncv/platform/ci-templates.
  • Build, scan, verify, and publish release-tools:v1.0.0 to the NCV Container Registry; record and later pin its digest.
  • Protect template tags; projects include templates by immutable tag, not main.
  • Create one ncv-releases/<pilot-product> project containing only release material.
  • Configure its Package Registry for the intended anonymous/authenticated download policy.
  • Protect the release project default main branch; store manifests beneath channels/.
  • Configure cross-project publish credentials using a job token allowlist or narrowly scoped project access token.
  • Publish a test immutable package.
  • Create a GitLab Release with release notes and package asset links.
  • Publish Experimental manifest pointing to the package.
  • Promote the exact SHA to Beta, then Stable, without rebuilding.
  • Verify each manifest, URL, package checksum, and GitLab release entry.
Acceptance: Experimental, Beta, and Stable point to the same artifact URL and SHA-256. Rollback: Revert a channel-manifest commit to the prior known-good package; never delete the known-good package during rollback.

Task 11: Inventory Azure DevOps

Produces: Complete migration inventory and wave plan.
  • List organizations, projects, repositories, default branches, branch policies, tags, Git LFS usage, submodules, and repository sizes.
  • List YAML and classic pipelines, schedules, triggers, environments, approvals, agent pools, artifacts, caches, variable groups, secure files, and service connections.
  • List Azure Artifacts feeds and package consumers/producers.
  • List webhooks, status checks, deployment scripts, and external integrations.
  • Classify repositories into pilot, standard, high-risk, archived, and retained-read-only.
  • Record every production deployment pipeline and its rollback method.
  • Choose migration waves and freeze owners.
Acceptance: No repository or operational dependency is unassigned. Rollback: None; inventory is read-only.

Task 12: Migrate the pilot repository

Files used: scripts/migrate-azure-repo.sh, scripts/verify-repo-migration.sh. Produces: A GitLab repository with all Git refs and LFS objects.
  • Create empty destination project under ncv/products.
  • Mirror-clone the Azure repository.
  • Fetch all Git LFS objects.
  • Push all refs to GitLab.
  • Push all LFS objects to GitLab.
  • Compare branch and tag sets.
  • Compare the default branch tip SHA.
  • Clone the GitLab repository into a clean directory and build locally.
  • Keep Azure PR/build history read-only; do not pretend Git mirror migration moved it.
Acceptance: Refs, tags, LFS objects, submodules, and a clean build match the Azure source. Rollback: Continue using Azure as canonical; delete or archive the incomplete GitLab project.

Task 13: Convert and validate the pilot pipeline

Produces: GitLab CI parity with the Azure pipeline.
  • Map Azure stages/jobs/steps to GitLab stages/jobs/scripts.
  • Map Azure agent pools to GitLab runner tags.
  • Recreate variables in GitLab group/project CI/CD settings; never copy secrets into YAML.
  • Recreate secure files in GitLab Secure Files or approved vault delivery.
  • Replace Azure service connections with narrowly scoped GitLab/deployment credentials.
  • Recreate schedules and manual inputs.
  • Produce JUnit XML and coverage artifacts where supported.
  • Build and deploy to a non-production test environment.
  • Compare outputs with Azure by file name, size, version metadata, and checksum where deterministic.
  • Run smoke and integration tests.
  • Run the release pipeline through Experimental and Beta; do not deploy Stable yet.
Acceptance: The GitLab pipeline produces functionally equivalent tested artifacts and deployment results. Rollback: Keep Azure production pipeline active; disable the GitLab production job.

Task 14: Pilot cutover

Produces: One product operating fully from GitLab.
  • Announce a repository freeze window in Jira and team communication.
  • Disable merges in Azure for the pilot repository.
  • Fetch and push the final delta to GitLab.
  • Run migration verification again.
  • Change developer remotes to GitLab.
  • Enable GitLab as the only production pipeline.
  • Run a tagged release and promote the exact artifact to Stable.
  • Confirm package download, release notes, deployment, monitoring, and rollback.
  • Mark Azure repository and pipeline read-only; retain history for the agreed retention period.
  • Record cutover evidence and owner sign-off in Jira.
Acceptance: All new commits and releases originate from GitLab and the known-good rollback remains available. Rollback: Re-enable Azure merge/deploy, restore developer remotes, and remove the GitLab Stable channel pointer created during the failed cutover.

Task 15: Migrate remaining repositories in waves

Produces: Azure DevOps retirement without a big-bang migration.
  • Repeat Tasks 12–14 by migration wave.
  • Do not mix unrelated high-risk production systems in one wave.
  • Validate backup growth and runner load after every wave.
  • Add a separate Linux runner before increasing concurrency on the GitLab host.
  • Archive rather than migrate dead repositories unless legal/business retention requires active hosting.
  • Revoke Azure service connections and tokens only after dependent systems are verified.
Acceptance: The inventory has no active unaccounted repository, pipeline, feed, or credential. Rollback: Per-project rollback, not whole-program rollback.

Task 16: Implement Windows and Apple runners

Produces: Native cross-platform builds.
  • Install a dedicated Windows runner tagged windows-x64.
  • Install Flutter, Visual Studio Build Tools/workloads, installer tooling, and code-signing access.
  • Run the runner under a restricted service account; protect the runner directory.
  • Store signing credentials only in protected variables/secure files and expose them only on protected refs.
  • Build, sign, package, and verify one Windows release.
  • Install a dedicated macOS runner tagged macos-arm64 when Apple delivery is required.
  • Configure Xcode, Flutter, Fastlane, keychain access, App Store Connect API key, TestFlight groups, and signing.
  • Verify macOS runner user-session and keychain behavior after reboot.
Acceptance: Native signed artifacts are produced only on their matching OS runners. Rollback: Pause the affected runner and continue non-native CI on Linux.

Task 17: Endpoint updater proof of concept

Produces: A go/no-go decision for automatic customer updates.
  • Implement the updater behind an application-owned interface so the package can be replaced later.
  • Validate channel selection from server policy rather than allowing ordinary customers to select Experimental.
  • Check current version and fetch release notes.
  • Download to a staging path and verify file length, SHA-256, and publisher signature.
  • Test optional update, mandatory update, restart-now, restart-later, silent idle update, and failed update recovery.
  • Test power/network interruption during download and install.
  • Test an application process holding files open.
  • Test downgrade/rollback policy.
  • Test standard-user installation permissions.
  • Test antivirus/EDR and Windows SmartScreen behavior with signed binaries.
  • Test Linux executable permissions and publisher authenticity.
  • Capture update success/failure telemetry without collecting customer secrets.
Acceptance: Security and recovery review approves the chosen updater and rollback path. Rollback: Disable auto-update via server policy and distribute the previous signed installer manually.

Task 18: Operational handover and go-live

Produces: A supportable platform rather than an undocumented installation.
  • Run all checks in docs/09-go-live-checklists.md.
  • Confirm backup and restore evidence is current.
  • Confirm administrator break-glass access.
  • Confirm monitoring and alert routing.
  • Confirm upgrade owner and maintenance policy.
  • Train developers using docs/04-developer-guide.md.
  • Train release managers using docs/07-release-package-update-guide.md.
  • Train infrastructure staff using the operations and DR runbooks.
  • Store this handbook in ncv/infrastructure/gitlab-platform and tag the approved baseline.
  • Close the Jira implementation epic only after owner and reviewer acceptance.
Acceptance: Operations, development, release management, and recovery each have a named trained owner and a verified runbook.