Development Instances Overview

Infrastructure Tooling for P2R Developers

The dev-instances project provides modern, single-user development instances with Golden AMI deployment, API-based management, and hybrid storage architecture. Each instance is dedicated to exactly one developer, with persistent EBS home volumes that survive AMI upgrades.

Purpose

Development Instances serves as P2R’s modernized development environment infrastructure, providing:

Performance - EBS /home provides 10x faster I/O than the original EFS-based shared environment
Persistence - Separate EBS home volumes enable AMI upgrades without data loss
Usability - API and Slack integration for practical instance management (Phase 2+)
Security - EC2 Instance Connect with AWS SSO, no SSH keys or open ports
Maintainability - Golden AMI strategy for consistent, reproducible environments

Architecture

Core Components

                      ┌─────────────────────────────────────────────────────────┐
                      │              Golden AMI Build Pipeline                   │
                      │                                                          │
                      │   EC2 Image Builder → Golden AMI → SSM Parameter        │
                      │                                                          │
                      │   Monthly Schedule | Pre/Post-Upgrade Components        │
                      │   Ubuntu 24.04 LTS | Data Science Stack                 │
                      └─────────────────────────┬───────────────────────────────┘
                                                │
                                                │ AMI ID
                                                ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                        Instance Deployment (CloudFormation)                      │
│                                                                                  │
│   Template URL: s3://p2r-dev-instances-templates/instance-deployment-template   │
│                                                                                  │
│   ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────────────────┐  │
│   │  EC2 Instance    │  │  EBS Home Volume │  │  EFS Mounts (7 filesystems)  │  │
│   │  Single Owner    │  │  Persistent      │  │  Shared Data Access          │  │
│   └──────────────────┘  └──────────────────┘  └──────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────┘

Technology Stack

Component	Technology
Infrastructure	AWS CDK 2.233, TypeScript, CloudFormation
AMI Pipeline	EC2 Image Builder, Lambda, EventBridge
CI/CD	AWS CodePipeline, CodeBuild, Bitbucket
Base OS	Ubuntu 24.04 LTS
Instance Auth	EC2 Instance Connect, AWS SSO
Storage	EBS (home), EFS (shared data)
Configuration	SSM Parameter Store
Testing	Jest, ts-jest
Runtime	Node.js 20/22/24, npm 10/11

Repository Structure

dev-instances/
├── cdk/                           # CDK infrastructure code
│   ├── bin/
│   │   ├── dev-instances.ts       # Main CDK app (branch-aware deployment)
│   │   └── synthesize-instance-template.ts  # Template generator
│   ├── lib/
│   │   ├── config.ts              # DevInstancesConfig class + constants
│   │   ├── git-util.ts            # Git branch detection
│   │   ├── image-builder-stack.ts # Golden AMI pipeline infrastructure
│   │   ├── image-builder-pipeline-stack.ts  # CI/CD pipeline + S3 template bucket
│   │   ├── instance-deployment-stack.ts     # Instance deployment template
│   │   └── imagebuilder/
│   │       └── components/        # Image Builder components (TypeScript → YAML)
│   │           ├── build/         # Build phase components
│   │           │   ├── pre-upgrade/   # Before update-linux
│   │           │   └── post-upgrade/  # After update-linux
│   │           └── test/          # Test phase components
│   └── test/                      # Jest test files
├── docs/                          # Documentation
│   ├── ARCHITECTURE.md            # Complete system architecture
│   ├── INSTANCE_DEPLOYMENT_GUIDE.md   # User deployment guide
│   ├── TEMPLATE_MAINTENANCE.md    # DevOps maintenance guide
│   └── INTEGRATION_TESTING.md     # Testing procedures
├── lambda/                        # API Lambda functions (Phase 2)
├── cli/                           # Command-line interface (Phase 2)
├── slack-app/                     # Slack integration (Phase 3)
├── Makefile                       # Build automation
├── package.json                   # npm configuration
└── README.md                      # Project overview

Golden AMI Pipeline

What Gets Built

The Golden AMI pipeline creates Ubuntu 24.04 LTS AMIs on a monthly schedule with:

System Users (6 accounts)

Username	UID	GID	Purpose
datascience	8003	8003	Data science workloads
datatools	8018	8018	Data tools operations
orders	8014	9006	Order processing
convert	8002	8002	Data conversion
preselect	8015	8015	Preselection processing
households	8005	8005	Household operations

Development Tools

Git, Docker Engine, AWS CLI v2, Deno
Python 3.12.8 (Miniconda3), Poetry
Scala 2.12.20, Hadoop 3.4.0, Spark 3.5.2
EC2 Instance Connect package

System Configuration

Sophos gateway routing (Path2Response account)
EFS mounts for 7 filesystems
Passwordless sudo for sudo group
User package persistence system

Build Component Sequence

Pre-Upgrade Components (before update-linux):
  00-networking     → Sophos gateway configuration
  01-hosts-config   → /etc/hosts with P2R infrastructure
  02-system-config  → Sudoers, EFS mount points

System Update:
  update-linux      → AWS managed security updates

Post-Upgrade Components (after update-linux):
  00-system-users   → 6 system user accounts
  01-system-libs    → Build tools, network tools, DB clients
  02-dev-tools      → Git, Docker, AWS CLI, Deno
  03-additional-packages → Configurable APT packages
  04-snap-packages  → Configurable Snap packages
  05-ec2-instance-connect → SSH via AuthorizedKeysCommand
  06-user-package-persistence → APT hook + boot service
  07-data-science-stack → Python, Scala, Spark

Test Phase:
  01-gateway-test   → Internet connectivity validation
  02-service-validation → Docker daemon running
  04-dev-tools-integration → Tool functionality tests
  05-network-connectivity → DNS, HTTPS, AWS, EFS tests

Pipeline Triggers

Trigger	Condition
Scheduled	Monthly (1st at 2 AM UTC)
Base Image Update	Ubuntu 24.04 LTS security patches
Manual	AWS CLI or Console

Instance Deployment

Single-User Model

Each development instance is dedicated to exactly ONE developer:

Instance owned by a single P2R developer
Operates with owner’s AWS permissions
Contains only owner’s home directory on EBS
Owner can SSH to system user accounts for shared workloads

Deployment Methods

Via AWS Console:

Navigate to CloudFormation Console
Create stack with S3 URL: https://p2r-dev-instances-templates.s3.amazonaws.com/instance-deployment-template.json
Specify parameters (InstanceOwner required)
Deploy

Via AWS CLI:

aws cloudformation create-stack \
  --stack-name dev-instance-whofmann-01 \
  --template-url https://p2r-dev-instances-templates.s3.amazonaws.com/instance-deployment-template.json \
  --parameters \
    ParameterKey=InstanceOwner,ParameterValue=whofmann \
    ParameterKey=InstanceType,ParameterValue=t3.xlarge \
  --capabilities CAPABILITY_IAM \
  --profile p2r_root

CloudFormation Parameters

Parameter	Required	Default	Description
`InstanceOwner`	Yes	-	P2R username (e.g., “whofmann”)
`HomeVolumeId`	No	(new)	Existing EBS volume to attach
`InstanceType`	No	t3.xlarge	EC2 instance type
`HomeVolumeSize`	No	100 GB	Size for new home volumes
`RootVolumeSize`	No	150 GB	Root volume size

P2R Users (10 developers)

chouk, dfuller, eyang, ioliynyk, jmalone, jsmith, mford, mpelikan, pmartin, whofmann

Hybrid Storage Architecture

Storage Layout

/                        # Root EBS volume (OS, applications)
├── home/                # Dedicated EBS volume (persistent user data)
│   ├── datascience/     # System user (from AMI)
│   ├── datatools/       # System user (from AMI)
│   ├── orders/          # System user (from AMI)
│   ├── convert/         # System user (from AMI)
│   ├── preselect/       # System user (from AMI)
│   ├── households/      # System user (from AMI)
│   └── <owner>/         # P2R user (instance owner only)
│       ├── .dev-instance/    # Package persistence (bind-mounted)
│       ├── .linuxbrew/       # Homebrew installation
│       ├── .local/           # pip --user packages
│       └── code/             # User projects
├── var/
│   └── dev-instance/    # Bind mount from ~/.dev-instance
├── mnt/
│   ├── data/            # Symlink to owner's preferred EFS
│   ├── shared-home/     # EFS shared collaboration space
│   ├── etl.production/  # EFS (read-only)
│   ├── etl.development/ # EFS (read-write)
│   ├── orders.production/   # EFS (read-only)
│   ├── orders.rc/           # EFS (read-only)
│   ├── orders.staging/      # EFS (read-write)
│   └── orders.development/  # EFS (read-write)

EFS Filesystems (7 total)

Mount Point	Access	Description
`/mnt/shared-home`	Read-Write	Shared user collaboration space
`/mnt/etl.production`	Read-Only	ETL production data
`/mnt/etl.development`	Read-Write	ETL development data
`/mnt/orders.production`	Read-Only	Orders production data
`/mnt/orders.rc`	Read-Only	Orders RC data
`/mnt/orders.staging`	Read-Write	Orders staging data
`/mnt/orders.development`	Read-Write	Orders development data

Performance Rationale

Why EBS for /home?

Original implementation used EFS for /home
EFS was too slow for development work (IDE indexing, compilation, etc.)
EBS provides 10x faster I/O performance
Separate EBS volume persists across AMI upgrades

User Package Persistence

Problem

Users need to install additional software that persists across AMI upgrades.

Solution

Automatic package tracking and reinstallation system:

User installs package → APT hook tracks → ~/.dev-instance/apt-packages.txt
                                                    ↓
AMI upgrade → Boot service reads list → Reinstalls packages

Three Installation Options

Method	Command	Persistence	Best For
APT	`sudo apt-get install`	Auto-reinstall via boot service	System libraries, services
Homebrew	`brew install`	Direct on EBS	Dev tools, modern versions
Language Managers	`pip install --user`	Direct on EBS	Language ecosystems

Branch-Based Deployment

Main Branch (Automated Pipeline)

git checkout main
AWS_PROFILE=p2r_root npx cdk deploy
# Creates: DevInstancesImageBuilderPipeline
# Pipeline deploys:
#   - DevInstancesImageBuilderPipeline-ImageBuilderStack
#   - Uploads instance-deployment-template.json to S3

Features:

Self-mutating CodePipeline monitors Bitbucket main branch
Automatic builds on git push
Slack notifications for pipeline events
Writes SSM parameters for service discovery

Feature Branch (Manual Deployment)

git checkout feature/my-feature
AWS_PROFILE=p2r_root npx cdk deploy
# Creates: DevInstancesImageBuilder-feature-my-feature

Features:

Direct deployment (no pipeline)
Isolated testing environment
Branch-specific SSM namespace
Does not affect main deployment

Resource Isolation

Resource	Main Branch	Feature Branch
S3 Bucket	`p2r-devinst-imgbldr-main-root-us-east-1`	`p2r-devinst-imgbldr-{branch}-root-us-east-1`
SSM Namespace	`/config/dev-instances/`	`/config/dev-instances-{branch}/`
Stack Name	`DevInstancesImageBuilderPipeline`	`DevInstancesImageBuilder-{branch}`

Configuration Management

DevInstancesConfig Class

The project uses a class-based configuration approach that encapsulates branch context:

const config = new DevInstancesConfig(isMainBranch, branchName);

// SSM parameter paths - clean API
config.output.goldenAmiLatestId(region)      // Latest AMI ID
config.output.imageBuilderPipelineArn(region) // Pipeline ARN
config.input.vpcId(region)                    // VPC from Terraform

Key Configuration Constants

Constant	Location	Purpose
`RECIPE_VERSION`	config.ts	Manual versioning for Image Builder recipes
`SYSTEM_USERS`	config.ts	6 system user accounts
`EFS_CONFIG`	config.ts	7 EFS filesystems with mount options
`IMAGE_BUILDER_DEFAULTS`	config.ts	Build instance type, schedule, retention

SSM Parameter Structure

Infrastructure Configuration (Input)

/config/p2r-tf-infra-network/{region}/Vpc4General/id
/config/p2r-tf-infra-network/{region}/Vpc4General/private-subnet-ids
/config/p2r-tf-infra-network/{region}/Vpc4General/public-subnet-ids

User Configuration (50 parameters)

/config/dev-instances/global/users/{username}/uid
/config/dev-instances/global/users/{username}/shell
/config/dev-instances/global/users/{username}/groups
/config/dev-instances/global/users/{username}/data-mount-preference
/config/dev-instances/global/users/{username}/ssh-public-keys

Image Builder Outputs

/config/dev-instances/{region}/golden-ami/latest-id
/config/dev-instances/{region}/image-builder/pipeline-arn
/config/dev-instances/{region}/image-builder/recipe-arn
/config/dev-instances/{region}/image-builder/artifact-bucket-name

Instance Metadata (per instance)

/config/dev-instances/{region}/instances/by-id/{instance-id}/owner
/config/dev-instances/{region}/instances/by-id/{instance-id}/home-volume-id
/config/dev-instances/{region}/instances/by-id/{instance-id}/created-at
/config/dev-instances/{region}/instances/by-owner/{owner}/{instance-id}

Development Workflow

Build and Deploy

# Install dependencies
npm install

# Build TypeScript
npm run build

# Run tests
npm test

# Deploy (branch-aware)
AWS_PROFILE=p2r_root npx cdk deploy

# Generate instance template locally (for testing)
npm run template:generate

Trigger Manual AMI Build

# Get pipeline ARN
PIPELINE_ARN=$(aws ssm get-parameter \
  --name /config/dev-instances/us-east-1/image-builder/pipeline-arn \
  --query 'Parameter.Value' --output text --profile p2r_root)

# Start build
aws imagebuilder start-image-pipeline-execution \
  --image-pipeline-arn $PIPELINE_ARN \
  --profile p2r_root

Build time: 20-30 minutes.

View Build Logs

# CloudWatch Logs
aws logs tail /aws/imagebuilder/DevInstancesImageBuilder-pipeline \
  --profile p2r_root --follow

# S3 Logs
aws s3 ls s3://p2r-devinst-imgbldr-main-root-us-east-1/image-builder-logs/ \
  --profile p2r_root --recursive

Implementation Status

Phase 1: Golden AMI Pipeline - COMPLETE

Ticket	Description	Status
PATH-25858	EC2 Image Builder in CDK	Complete
PATH-25859	Core Software Setup	Complete
PATH-25860	Data Science Stack	Complete
PATH-25861	Instance Deployment Templates	Complete

Phase 2: Instance Management - PLANNED

Ticket	Description	Status
PATH-27115	Core Management API	Planned
PATH-27116	API Control Operations	Planned
PATH-27119	CLI Frontend	Planned
PATH-25864	Documentation & Migration	Planned

Phase 3: Advanced Features - PLANNED

Ticket	Description	Status
PATH-27117	Slack Integration	Planned
PATH-27118	AMI Upgrade Functionality	Planned

Success Metrics

Metric	Target
Performance	10x I/O improvement over EFS-only
Deployment Speed	< 5 minutes from API call to running
User Adoption	80% using API/Slack within 1 month
Support Reduction	50% fewer access-related tickets
Availability	99.5% uptime during business hours

Cost Estimates

Running Instance (t3.xlarge)

Resource	Monthly Cost
Instance (730 hours)	~$121
Root volume (150 GB)	~$15
Home volume (100 GB)	~$10
Total	~$146/month

Stopped Instance

Resource	Monthly Cost
Root volume (150 GB)	~$15
Home volume (100 GB)	~$10
Total	~$25/month

BERT Overview - Unified backend platform
Data Tools Overview - Legacy data processing
Project Inventory - Complete P2R codebase inventory
Development Process - Engineering team processes

Important Notes

Single-user model - Each instance is dedicated to exactly one developer
AMI upgrades preserve data - Home EBS volume persists across upgrades
Target account - Path2Response root account (448838825215) only
VPC peering - Required for EFS access to production/RC environments
Recipe versioning - Manual increment required when components change
Component lifecycle - Components are development artifacts, not production resources

Source: dev-instances repository (README.md, docs/ARCHITECTURE.md, cdk/lib/) Documentation created: 2026-01-24

Keyboard shortcuts

Path2Response Product Management Knowledge Base