Dev-Ops Repository Overview

Ansible Infrastructure Automation for Path2Response

The dev-ops repository contains Ansible playbooks, roles, and an API for automating server provisioning, application deployment, and infrastructure configuration across Path2Response environments.

Purpose

This repository serves as the single source of truth for all infrastructure automation at Path2Response:

• Server Provisioning - Configure new servers with required users, packages, and settings
• Application Deployment - Deploy and update P2R applications (Data Tools, Dashboards, Order App, etc.)
• Sprint Releases - Orchestrate coordinated releases across multiple applications
• Configuration Management - Manage NGINX, Node.js, Scala, and other dependencies
• Self-Service Deployments - Enable developers to trigger deployments via API

Repository Structure

/dev-ops/
├── ansible.cfg              # Ansible configuration (vault, logging, callbacks)
├── update-all.sh            # Deno utility update script
├── inventory/               # Environment-specific host definitions
│   ├── global/              # Shared server definitions
│   ├── production/          # Production environment
│   ├── staging/             # Staging/Development environment
│   └── releasecandidate/    # Release Candidate environment
├── playbooks/               # Ansible playbooks organized by purpose
│   ├── application/         # Application-specific deployments (59 playbooks)
│   ├── dependencies/        # Infrastructure setup (users, nginx, firewall)
│   ├── release/             # Sprint release orchestration
│   ├── server/              # Server provisioning playbooks
│   └── test/                # Test playbooks
├── roles/                   # Reusable Ansible roles (72 roles)
├── ansible-deployment-api/  # REST API for self-service deployments
├── operations-book/         # mdBook documentation (AWS alarms, Qualys scans)
├── scripts/                 # Helper scripts (encryption, JVM upgrades)
└── log/                     # Sprint release logs

Environments

Inventory Structure

Each environment has three inventory files:

Server Groups

Host Groups

Key Playbooks

Sprint Release Playbooks (playbooks/release/)

Sprint Release Workflow:

1. Stop Services (Queue + Data Tools)
2. System Packages (conditional)
3. Deploy coop-scala
4. Deploy cass-ws
5. Deploy client
6. Deploy all applications
7. Upgrade servers (conditional)
8. Start Services (Data Tools + Queue)

Application Playbooks (playbooks/application/)

Server Playbooks (playbooks/server/)

Dependencies Playbooks (playbooks/dependencies/)

Key Roles

Infrastructure Roles

Application Roles

Utility Roles

Ansible Deployment API

Location: /ansible-deployment-api/

A Node.js Express application that enables self-service Ansible deployments via REST API.

Purpose

Allows developers to trigger deployments without direct server access or Ansible knowledge.

Endpoints

API Authentication

• Basic authentication required for all endpoints
• Credentials configured in config/config.js (not in git)

Example Usage

# List available inventories
curl -u service-user:password \
  'https://ansible-deployment-api.path2response.com/services/inventories/list?devOpsBranch=production'

# Run a playbook
curl -u service-user:password \
  'https://ansible-deployment-api.path2response.com/services/playbooks/run' \
  -d 'inventory=production&playbook=application/datatools.yml&releaseTag=production&devOpsBranch=production'

Features

• FIFO Queue - Deployments execute sequentially to prevent conflicts
• Slack Notifications - Start, success, and failure notifications
• Logging - Full deployment logs saved with timestamps
• Branch-aware - Can deploy from any dev-ops branch

Project Structure

/ansible-deployment-api/
├── app.js                 # Express app configuration
├── bin/www                # Server entry point
├── config/                # Environment-specific configuration
├── lib/                   # Core libraries (devOps, slack, validation)
├── routes/                # Express route definitions
├── services/              # Business logic
│   ├── inventories/       # Inventory listing
│   └── playbooks/         # Playbook listing and execution
├── pm2-config.json        # PM2 process management
└── tracking/              # Deployment tracking

Configuration

ansible.cfg

[defaults]
remote_user = root
vault_password_file = ~/.ansible/.vault_pass
roles_path = roles/
log_path = log/ansible.log
any_errors_fatal = true
callback_whitelist = profile_tasks  # Task timing

[ssh_connection]
ssh_args = -o ForwardAgent=yes

[p2r]
version = 335.0.0-SNAPSHOT

Environment Variables

Vault Encryption

Sensitive files (SSH keys, credentials) are encrypted with Ansible Vault:

# Encrypt a file
ansible-vault encrypt <file>

# List all encrypted files
./scripts/listAllEncryptedFiles.sh

# Grep encrypted files
./scripts/grepAllEncryptedFiles.sh <pattern>

Slack Integration

Playbooks send notifications to Slack channels:

Notification Content:

• Playbook name and version
• Start/complete/failure status
• Environment context

Operations Book

Location: /operations-book/

An mdBook containing operational runbooks and documentation.

Contents

• AWS Alarms configuration
• Failed Console Login Attempts handling
• Qualys Vulnerability Scans procedures

Building

cd operations-book
mdbook build --open

Usage Examples

Deploy Data Tools to Production

ansible-playbook -i inventory/production playbooks/application/datatools.yml

Full Sprint Release

ansible-playbook -i inventory/production playbooks/release/sprint-release.yml

Provision a New Server

ansible-playbook -i inventory/production playbooks/server/edge01.yml

Update NGINX Configuration

ansible-playbook -i inventory/production playbooks/dependencies/nginx.yml

Upgrade Node.js on All Hosts

ansible-playbook -i inventory/production playbooks/dependencies/node-all.yml

Pause Order Queue (for maintenance)

ansible-playbook -i inventory/production playbooks/application/pause-orders-queue.yml

Deployment Patterns

Clone-Install Pattern

Used by most application roles to deploy code:

1. Clone - Git checkout to /tmp/ directory with unique SSH key
2. Build - Maven/npm build on local machine
3. Distribute - Sync artifacts to target servers
4. Cleanup - Remove temporary SSH keys

- role: clone-install
  branch: "{{ coop_scala_branch }}"
  checkout_path: "{{ coop_scala_tmp }}"
  git_repo: "git@bitbucket.org:path2response/coop-scala.git"
  java_home: "/usr/lib/jvm/java-17-openjdk-amd64/"

Service Account Pattern

Applications run under dedicated service accounts:

Start/Stop Pattern

Services are managed via process managers:

Related Documentation

• BERT Overview - Target platform for legacy migrations
• Dashboards Overview - Dashboards application details
• Data Tools Overview - Data Tools application details
• Project Inventory - Complete P2R codebase listing
• Tools and Systems - Infrastructure overview

Important Notes

• Vault Password - Required for encrypted files. Store at ~/.ansible/.vault_pass
• SSH Agent - ForwardAgent enabled for git operations
• Error Handling - any_errors_fatal=true stops playbook on first error
• Logging - All playbook runs logged to log/ansible.log
• Branch Alignment - Deployment branch should match environment (production, staging, etc.)
• FIFO Queue - API deployments queue to prevent race conditions
• Timing - profile_tasks callback provides execution timing for optimization

Source: dev-ops repository (README.md, ansible.cfg, playbooks/, roles/, inventory/, ansible-deployment-api/) Documentation created: 2026-01-24