Operations Overview

Automated system maintenance, health monitoring, and data synchronization tools for Path2Response back-office infrastructure.

Purpose

The Operations repository provides day-to-day operational tooling for the Infrastructure team. It performs automated system checks, data synchronization, and maintenance across staging, release-candidate (RC), and production environments. These tools run continuously via cron jobs on cc.path2response.com, the command-and-control server.

Primary Users: Infrastructure Team (Jason Smith, Wes Hofmann)

Key Responsibilities:

• System health monitoring at multiple intervals (10-minute, hourly, daily, weekly, monthly)
• Data synchronization from production to staging/development environments
• AWS resource management (EC2 instances, S3 storage, EFS volumes)
• Automated cleanup of temporary files and old data
• RC environment management (start/stop servers, pause/resume MongoDB Atlas)

Architecture

Directory Structure

operations/
├── bin/                    # Bash cron wrapper scripts
├── book/                   # Operations Reference Manual (mdbook)
│   └── src/               # Documentation source files
├── deno/                   # Deno utilities
│   ├── app/               # Deno applications (audit, costs, network)
│   ├── book-of-ops/       # AWS infrastructure documentation
│   ├── common/            # Shared Deno modules
│   ├── lib/               # Deno libraries (up, gateway)
│   └── utilities/         # Deno utility scripts
├── src/                    # TypeScript source code
│   ├── checks/            # Health check definitions
│   ├── rcman/             # RC management modules
│   ├── sync/              # Data synchronization modules
│   └── util/              # Shared utilities (AWS, functional, MongoDB)
├── tests/                  # Unit tests
├── crontab.txt            # Reference copy of production crontab
├── deno.jsonc             # Deno configuration
├── package.json           # Node.js dependencies
└── tsconfig.json          # TypeScript configuration

Technology Stack

Key Dependencies

{
  "@aws-sdk/client-ec2": "3.958.0",
  "@aws-sdk/client-s3": "3.958.0",
  "mongodb": "7.0.0",
  "node-slack": "0.0.7",
  "commander": "14.0.2"
}

Core Functionality

Utilities (Command-Line Tools)

Health Checks (system-health)

The system-health utility runs checks at different intervals:

EC2 Instance Management:

• Instances tagged with p2r=permanent are exempt from stale checks
• Instances can have custom timeouts (e.g., p2r=1.12:00 for 1 day 12 hours)
• Auto-kill option: p2r=1:00,kill terminates instance after timeout

Data Synchronization (sync-monkey)

Synchronizes data from production to staging/development:

Critical Note - MongoDB Filtered Sync: The operations.datafile collection sync uses a filter ({done: true, pid: {$exists: true}}) to prevent race conditions where staging processes could intercept production work queue items. See PATH-25842 for full context.

RC Environment Management (rc-man)

Manages Release Candidate infrastructure for cost optimization:

Warning: Paused MongoDB Atlas clusters auto-resume after 30 days.

Cleanup Operations (clean, archive)

Integrations

AWS Services

Internal Systems

External Services

Development

Prerequisites

• Node.js (project-local installation)
• Deno (for newer utilities)
• AWS CLI (apt install awscli)
• Rust + mdbook (for documentation)
• graphviz (for mdbook diagrams)

Build Commands

Deployment

1. Make changes in TypeScript source
2. Run npm run build to compile
3. Run npm install -g to install globally (live deployment)
4. Changes take effect immediately, including for cron jobs

Testing

# Run unit tests
npm run test

# Test specific utility without Slack
node ./dist/system-health.js --short --no-slacking

# Test RC management (dry run)
./dist/rc-man.js servers --start --no-slacking

Documentation (mdbook)

# Install dependencies
sudo apt install graphviz
cargo install mdbook-graphviz

# Serve documentation
cd book && mdbook serve -p 8089

The documentation auto-refreshes on save.

Cron Schedule Reference

Environment Configuration

All authentication uses environment variables or IAM roles:

Note: Command-line utilities require the secrets environment on cc.path2response.com.

Related Documentation

• Operations Reference Manual - book/src/SUMMARY.md
• [AWS SDK v3 Migration](operations repo: AWS_SDK_V3_MIGRATION.md) - Completed migration notes
• [Deno Book of Ops](operations repo: deno/book-of-ops/) - AWS infrastructure documentation
• [S3 Replication](operations repo: book/src/s3-replication.md) - Data sync topology
• [MongoDB Filtered Sync](operations repo: book/src/sync-monkey.md) - PATH-25842 race condition fix

Key Jira References

Source: README.md, book/src/SUMMARY.md, book/src/intro.md, book/src/utilities.md, book/src/system-health.md, book/src/sync-monkey.md, book/src/clean.md, book/src/archive.md, book/src/rc-man.md, book/src/s3-replication.md, book/src/aws-s3-audit.md, book/src/aws-efs-audit.md, package.json, crontab.txt, deno/README.md, AWS_SDK_V3_MIGRATION.md

Documentation created: 2026-01-24