JOSSGAWIN 1ST TIME IN TOKYO: Infrastructure Deployment and Bootstrapping Manual

1.0 Scope and Prerequisites

This manual details the procedural framework for establishing the core technical infrastructure denoted by the project identifier "JOSSGAWIN 1ST TIME IN TOKYO." This operation is analogous to bootstrapping a new, scalable computing environment in a primary data center location. The primary objective is to achieve a fully operational, automated, and reproducible state with minimal manual intervention, maximizing long-term operational efficiency and return on infrastructure investment (ROI).

Applicability: This procedure is designed for system administrators, DevOps engineers, and infrastructure teams responsible for initial deployment in a new geographic region or data center pod.

Prerequisites & System Requirements:

• Hardware: At least one dedicated server with Intel VT-x/AMD-V support, a minimum of 16GB RAM, 500GB storage, and two network interfaces (NIC).
• Network: A dedicated /24 IPv4 subnet, configured DHCP scope (with options 66 and 67), and a router/firewall allowing TFTP (port 69), HTTP (port 80), and NFS (port 2049).
• Software: A base installation of a Linux distribution (e.g., Ubuntu Server 22.04 LTS or Rocky Linux 9) on the provisioning server, with root/sudo access.
• External Resources: Access to official distribution repositories and the project's internal configuration management repository (e.g., Git).

2.0 Procedure

Follow these steps sequentially to deploy the foundational environment.

1. Provisioning Server Configuration

   Install and configure the necessary services for network-based provisioning, specifically PXE (Preboot Execution Environment).

   # Update system and install packages
   sudo apt update && sudo apt install -y isc-dhcp-server tftpd-hpa nfs-kernel-server apache2 syslinux
   
   # Configure DHCP Server (/etc/dhcp/dhcpd.conf)
   subnet 192.168.1.0 netmask 255.255.255.0 {
     range 192.168.1.100 192.168.1.200;
     option routers 192.168.1.1;
     option subnet-mask 255.255.255.0;
     option domain-name-servers 8.8.8.8;
     # Critical PXE directives:
     filename "pxelinux.0";
     next-server 192.168.1.10; # IP of this provisioning server
   }

   Expected Result: A target machine connected to the same network segment will receive an IP address and the boot file pxelinux.0 from this server upon PXE boot.

2. Boot Image and Kernel Preparation

   Set up the TFTP root directory with the bootloader and kernel images.

   # Create TFTP directory structure
   sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg
   sudo cp /usr/lib/syslinux/modules/bios/*.c32 /var/lib/tftpboot/
   sudo cp /usr/lib/syslinux/modules/bios/pxelinux.0 /var/lib/tftpboot/
   
   # Download a minimal Linux kernel and initrd (e.g., for Ubuntu)
   cd /var/lib/tftpboot
   sudo wget https://archive.ubuntu.com/ubuntu/dists/jammy/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
   sudo wget https://archive.ubuntu.com/ubuntu/dists/jammy/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz
   
   # Create the default PXE configuration
   sudo nano /var/lib/tftpboot/pxelinux.cfg/default

   Insert the following configuration into the `default` file:

   DEFAULT linux
   LABEL linux
     KERNEL linux
     APPEND initrd=initrd.gz ip=dhcp url=http://192.168.1.10/ubuntu-preseed.cfg root=/dev/nfs nfsroot=192.168.1.10:/srv/nfs/rootfs --

   Expected Result: The TFTP server provides all necessary files for a client to load a kernel and begin an automated installation.

3. Automated Installer and Root Filesystem

   Create a preseed file for unattended OS installation and an NFS-exported root filesystem.

   # Configure Apache to serve the preseed file
   sudo cp /path/to/local/ubuntu-preseed.cfg /var/www/html/
   # Build or deploy a golden root filesystem image to /srv/nfs/rootfs
   sudo mkdir -p /srv/nfs/rootfs
   # Configure NFS export (/etc/exports)
   /srv/nfs/rootfs *(rw,sync,no_subtree_check,no_root_squash)

   Expected Result: The booting client retrieves the preseed configuration via HTTP, guiding an unattended installation that targets the NFS root, ensuring consistency.

4. Post-Provisioning Automation

   Integrate the newly provisioned system with configuration management (e.g., Ansible).

   # Example Ansible playbook snippet (post-boot.yml)
   - hosts: newly_provisioned
     become: yes
     tasks:
       - name: Register system with central inventory
         uri:
           url: "https://inventory.api.internal/register"
           method: POST
           body: "{{ ansible_facts }}"
       - name: Apply base security hardening
         include_role:
           name: base_hardening
       - name: Deploy JOSSGAWIN application stack
         include_role:
           name: jossgawin_stack

   Expected Result: The system transitions from a base OS to a fully configured, application-ready state without manual login, completing the "first time in Tokyo" bootstrap process.

3.0 Troubleshooting and Risk Mitigation

Potential failure points, their root causes, and corrective actions are outlined below. Proactive mitigation of these risks protects infrastructure investment and ensures deployment reliability.

• Issue: Target machine fails to PXE boot, showing "TFTP Open timeout."
  Root Cause & Solution: Network firewall blocking UDP port 69. Verify firewall rules on the provisioning server and intermediate switches. Test with `tcpdump -i eth0 port 69` on the server while initiating PXE boot.
• Issue: Client receives DHCP offer but fails to load `pxelinux.0`.
  Root Cause & Solution: Incorrect `next-server` directive in `dhcpd.conf` or missing/incorrectly permissioned files in `/var/lib/tftpboot/`. Confirm file permissions are world-readable (`chmod 644`).
• Issue: Installation hangs during preseed phase.
  Root Cause & Solution: Error in the preseed configuration file or unreachable HTTP URL. Access the installer's serial console (Alt+F4 in VGA) to view detailed error logs. Validate the preseed file syntax with `debconf-set-selections --check`.
• Issue: System boots but fails to apply Ansible playbook.
  Root Cause & Solution: Network connectivity issues or SSH host key mismatches. Ensure the base image has the correct SSH credentials pre-loaded and that the Ansible control node has network access to the new host's IP.

Investment Note: The initial complexity of this automated pipeline directly reduces long-term operational expenditure (OpEx) and systemic risk. It enables rapid, consistent scaling—a critical value driver for expansion into new regions like Tokyo. The primary risk is upfront configuration error, which is mitigated by rigorous testing in a staging environment that mirrors production.