Order Processing Overview

Legacy web application for managing Path2Response order workflows, Jira synchronization, and campaign fulfillment operations.

Status: Legacy System - Order App functionality has been migrated to BERT, with Shipment functionality migration in progress.

Purpose

The Order Processing system (internally called “Order App” or “Legacy”) automates and simplifies the tedious parts of order processing for Path2Response campaigns. It provides:

• A web-based interface for managing orders and models from Jira
• Automated job processing for P2A models, fulfillment, and shipment
• Jira synchronization to keep order data current
• A scheduler for batch processing and queue management
• Integration with BERT for transitioning functionality to the new platform

Primary Users:

• Data Science team (model runs, preselect configuration)
• Data Engineering team (fulfillment, shipment, job management)
• Operations team (order tracking, queue monitoring)

Architecture

Directory Structure

order-processing/
├── backend/                 # Sails.js API server
│   ├── api/
│   │   ├── controllers/     # API endpoints (44 controllers)
│   │   ├── models/          # MongoDB data models (37 models)
│   │   ├── services/        # Business logic services (50+ services)
│   │   ├── policies/        # Authentication/authorization
│   │   └── hooks/           # Sails lifecycle hooks
│   ├── config/              # Sails configuration
│   └── lib/                 # Utilities (logger, queue management)
├── frontend/                # AngularJS SPA
│   ├── src/app/
│   │   ├── orders/          # Main order management views
│   │   │   ├── preselect/   # Model configuration
│   │   │   ├── fulfillment/ # Fulfillment processing
│   │   │   ├── shipment/    # Shipment management
│   │   │   ├── scheduler/   # Job scheduling
│   │   │   └── queue/       # Queue monitoring
│   │   ├── core/            # Shared components
│   │   └── admin/           # Admin functions
│   └── config/              # Frontend configuration
├── edge-scripts/            # Server-side processing scripts
│   └── scripts/
│       ├── tasks/           # Job execution scripts
│       │   ├── p2a2/        # P2A2 model tasks
│       │   ├── p2a3xgb/     # XGBoost model tasks
│       │   ├── fia/         # Fulfillment Input Analysis
│       │   └── extract/     # Data extraction
│       ├── shipment/        # Shipment processing
│       └── util/            # Shared utilities
├── scripts/                 # Deployment and maintenance scripts
├── nginx.conf               # Production nginx configuration
└── pm2-process.json         # PM2 process management

Technology Stack

Key Dependencies

• javascript-shared - Internal shared library (error handling, utilities)
• sails-mongo - Custom MongoDB adapter (Atlas 6 compatible fork)
• jira-connector - Jira API integration
• winston - Logging with daily rotation

Core Functionality

Order Management

The system synchronizes with Jira to track orders and models:

• Orders - Campaign requests with client, title, dates, and fulfillment details
• Models - P2A model configurations (P2A2, P2A3/XGB, Hotline, DTC, etc.)
• Preselect - Model configuration and parameter setup
• Fulfillment - Processing and delivering audience files

Job Processing

The Order App manages several job types:

Feature Flags

The system uses feature flags to control functionality during BERT migration:

LEGACY_P2A3XGB_FULL_RUN_DISABLED      - Disable XGB runs in Legacy
LEGACY_SHIPMENT_RUN_DISABLED          - Disable shipment runs
LEGACY_FULFILLMENT_RUN_DISABLED       - Disable fulfillment runs
LEGACY_FEATURE_FLAG_DEBUG_MODE        - Enable debug logging

Feature flags are stored in MongoDB and keyed by:

• name - Flag identifier
• branchName - Git branch (staging, production, feature branch)
• deploymentEnvironment - DEV, STAGING, PRODUCTION

Jira Synchronization

The Order App maintains a local MongoDB cache of Jira issues:

• Background jobs sync data from Jira
• Order model extracts ~100+ fields from Jira custom fields
• Real-time updates via Jira webhooks
• Handles both PATH project orders and model subtasks

Integrations

BERT Integration

BERT (Base Environment for Re-tooled Technology) is the successor platform:

• BertIntegrationController - Receives commands from BERT
• serverlessToLegacyJobLauncher - Launches Legacy jobs from BERT
• Feature Flags - Control which functionality runs in Legacy vs BERT

Flow: BERT can trigger Legacy actions via the /bert/runActionForLegacy endpoint, allowing gradual migration of functionality.

Jira

• Custom authentication using Jira credentials
• Order/Model data synchronized from Jira
• Comments and attachments managed through Jira API
• Issue status transitions triggered from UI

Salesforce

• Order creation from Salesforce opportunities
• Middleware integration for data synchronization

Databricks

• Model execution on Databricks compute
• Configuration management for ML jobs

Slack

• Notifications for job completion and errors
• Alert integration for monitoring

Development

Prerequisites

# Required global packages
npm install -g bower gulp sails

# Node.js version
node --version  # Should be ~22 or ~24

Installation

# Clone and install
git clone git@bitbucket.org:path2response/order-processing.git
cd order-processing
npm install

# Install backend and frontend separately
cd backend && npm install
cd ../frontend && npm install && bower install

Configuration

1. Backend: Copy /backend/config/local_example.js to /backend/config/local.js
2. Frontend: Copy /frontend/config/config_example.json to /frontend/config/config.json

Running Locally

# Start backend
cd backend
sails lift
# API available at http://localhost:1337

# Start frontend (separate terminal)
cd frontend
gulp dist
gulp production
# UI available at http://localhost:3000

Production Deployment

The application runs on an EC2 instance with nginx:

Process management via Forever:

cd backend
npm run start          # Production
npm run start-staging  # Staging

Testing

cd backend
npm test  # Runs Mocha tests

Note: Tests are acknowledged as outdated and may not pass (see README TODO).

Data Models

Key Models

Order Model Fields

The Order model contains extensive fields including:

• Core Jira fields: id, key, fields, status
• Client/Title info: client, title, titleKey, titleInfo
• Dates: dueDate, mergeCutoff, mailDate, resultsDue
• Model configuration: modelType, buyerType, variableSources
• Fulfillment: fulfillmentFile1-5, fulfillmentComplete1-5
• Workflow state: workflowState, workflowStateKey

Related Documentation

• Path2Acquisition Flow - Overall data flow diagram
• BERT Overview - Successor platform documentation
• Development Process - Agile/Scrum process

Migration Status

The Order Processing system is being deprecated in favor of BERT. Current migration status:

Source: README.md, backend/README.md, frontend/README.md, TODO_PATH-25869.md, package.json, backend/package.json, Order.js, Featureflag.js, BertIntegrationController.js, nginx.conf

Documentation created: 2026-01-24