Path2Response Product Management Knowledge Base

Welcome to the comprehensive knowledge base for product management at Path2Response.

About This Book

This knowledge base serves as the central resource for:

1. New PM onboarding - Everything a new Product Manager needs to understand Path2Response
2. CTO’s interim PM responsibilities - Reference for current product management work
3. Claude Code as PM assistant - Context for AI-assisted product development
4. Institutional knowledge preservation - Documentation that persists beyond individuals

About Path2Response

Path2Response is a data cooperative specializing in direct mail marketing with growing digital audience capabilities. Founded ~2015 and headquartered in Broomfield, CO.

2025 Theme: “Performance, Performance, Performance”

Key Strategic Shift: Moving toward digital-first products — onboarding proprietary audiences into ad tech ecosystems like LiveRamp and The Trade Desk.

Core Products

Leadership

• Brian Rainey - CEO
• Karin Eisenmenger - COO (Product Owner for PROD)
• Michael Pelikan - CTO (Engineering leadership)
• Phil Hoey - CFO

How to Use This Book

Getting Started

If you’re new to Path2Response, start with:

1. Solution Overview - Core concepts and company identity
2. Path2Acquisition Flow - How our primary product works
3. Glossary - 100+ term definitions

For Product Managers

Key processes to understand:

1. PROD-PATH Process - How Product and Development work together
2. Stage-Gate Process - Initiative lifecycle (6 stages, 5 gates)
3. Work Classification - Initiative vs Non-Initiative vs Operational

For Technical Understanding

System documentation in priority order:

1. BERT Platform - Our unified backend (migration target)
2. Project Inventory - Complete inventory of 177 repositories
3. coop-scala - Core Spark data processing

For Industry Context

Understanding direct mail marketing:

1. Direct Mail Fundamentals - Value chain and economics
2. Data Cooperatives - How co-ops work
3. Competitive Landscape - Epsilon, Wiland, etc.

Key Concepts

PROD vs PATH

Work flows from PROD → PATH at Gate 2 when requirements are complete.

Work Classification

This knowledge base is maintained by the CTO with Claude Code assistance. Last updated: January 2026.

Path2Response Solution Overview

Source: New Employee Back-Office Orientation (2025-08)

Company Identity

Path2Response is a data cooperative specializing in customer acquisition through direct mail marketing, with expanding digital capabilities.

• Parent Company: Caveance (holding company)
• Taglines: Marketing updates taglines regularly; recent examples include “Marketing, Reimagined” and “Innovation and Data Drive Results”
• Mission: Foster a transparent culture of relentless innovation and excellence that empowers employees to flourish.
• Goal: Create great products that perform for our clients.

What We Do

“We are a data driven team that engages with brands who strive to provide relevant marketing to consumers. Path2Response is comprised of data visionaries and cutting edge technologists. Together, we are blazing a path to help our clients acquire new customers. We actively partner with our clients to provide innovative, data based solutions that allow them to create and maintain profitable relationships with their customers.”

Core Concepts

What is a Data Cooperative?

A data co-op is a group organized for sharing pooled data from consumers between two or more companies. Members contribute their transaction data and gain access to the cooperative’s combined data assets for prospecting.

What is Direct Mail?

Direct mail marketing is any physical correspondence sent to consumers in the hopes of getting them to patronize a client. This includes catalogs, postcards, letters, and other physical marketing materials.

RFM - The Foundation of Response Modeling

RFM analysis is fundamental to predicting consumer response behavior.

Core Product: Path2Acquisition

While Path2Response has various solution offerings, we focus on a single product: Path2Acquisition.

The other solutions are all variations on this product, focusing on particular needs and verticals:

• Path2Ignite - Performance-driven direct mail
• Digital Audiences - 300+ segments for digital channels
• Path2Contact - Reverse email append
• Path2Optimize - Data enrichment
• Path2Linkage - Data partnerships

Verticals We Serve

Important: Terms used throughout the system are Catalog-centric. Other verticals use equivalent terminology (e.g., “donors” instead of “customers” for Nonprofit).

The Path2Acquisition Flow

See Path2Acquisition Data Flow for the complete solution architecture diagram and detailed explanation.

Campaign Timeline

This extended timeline is critical for understanding why response analysis and match-back processes span long periods.

Related Documentation

• Path2Acquisition Data Flow - Detailed solution architecture
• Glossary - Terms and definitions
• Products & Services - Complete product portfolio

Path2Acquisition Data Flow

Source: New Employee Back-Office Orientation (2025-08), Page 2 Diagram

This document explains the complete Path2Acquisition solution flow, from client onboarding through response analysis.

Flow Diagram Overview

The Path2Acquisition system processes data through several major stages:

┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│   CLIENT    │ → │    DATA     │ → │  MODELING   │ → │  DELIVERY   │
│   INTAKE    │   │  PROCESSING │   │  & SCORING  │   │  & RESULTS  │
└─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘

Stage 1: Client Intake & Order Entry

Client Hierarchy

Broker
  └── Client
        └── Brand
              ├── Parent
              │     └── Title ←── House File (via SFTP)
              └── Sibling

Order Process

• Order initiates the campaign
• Order App manages order entry and configuration
• Order Processing validates and routes the order

Transaction Types

Stage 2: Data Ingestion & Processing

House File Processing

House File → CASS → Convert → Transactions → Data Attributes
                ↓
           DPV, DPBC

Transaction Storage

• Transactions stored in JSON format
• Title Key links transactions to specific titles
• Data Tools provide processing utilities

Data Enrichment

• BIGDBM - External data provider for additional attributes
• Data Attributes - Enhanced consumer data points

Stage 3: Digital Data Integration

First-Party Data Sources

Site Tag ─┬─ Cookie
          └─ Pixel
              ↓
         Site Visitors → Extract → Individuals → Identities
              ↑
    Hashed Email (HEM)
    ├── MD5
    ├── SHA-1
    └── SHA-2

Identity Resolution

• Extract process with CPRA compliance blocks
• Links digital Individuals to postal Identities
• Bridges online behavior to offline marketing

Stage 4: Householding

Process Flow

Individuals → Householding → Households File → Preselect
                  ↓
        ┌────────┴────────┐
        ↓                 ↓
   Addresses           Names
        ↓                 ↓
      DMA            Deceased
                      Pander

Output Branches

• Digital Solutions - For digital channel delivery
• Other Products - Path2Contact, Path2Optimize, etc.
• License Files - Data licensing arrangements

Stage 5: Title Responders & BERT

Title Responders

Identifies individuals who have responded to specific titles (catalogs/mailings).

BERT System

Order → BERT → Variables → Modeling
         ↑
    Order App
    Order Processing

BERT (Base Environment for Re-tooled Technology) is the unified backend application platform that consolidates internal tools (Order App, Data Tools, Dashboards, Response Analysis) into a single application. Within the Path2Acquisition flow, BERT:

• Manages order entry and processing (via Order App module)
• Generates modeling variables
• Feeds the scoring engine

Stage 6: Preselect & Universe Building

Preselect Process

Households File → Preselect → Universe
                      ↓
              ┌───────┴───────┐
              ↓               ↓
          Responder       Prospect

Synergistic Titles

Identifies related titles whose responders might respond well to the target title.

Example:

• Title: McGuckin Hardware
• Synergistic Titles: Home Depot, Lowes
• Question: What other titles would buyers of McGuckin also purchase from?

Suppressions

Stage 7: Modeling & Scoring

Model Types

Variables → Modeling → Score File
               ↓
        ┌──────┴──────┐
        ↓             ↓
    Affinity     Propensity

Score File Output

• Segments - Grouped by score ranges (deciles, quintiles)
• Used to prioritize mail recipients by predicted response

Stage 8: Fulfillment

Process Flow

Score File → Fulfillment → Fulfillment File → Shipment
                 ↓
              Nth-ing

Stage 9: Shipment & Merge/Purge

Shipment to Service Bureau

Fulfillment File → Shipment → Service Bureau → Mail File
                                    ↓
                              Merge / Purge
                              ┌─────┴─────┐
                              ↓           ↓
                            Hits        Nets

Merge Cutoff

The deadline for including names in the mailing. After this date, the file is locked for production.

Mail File Components

Stage 10: Mailing

Output

Mail File → In-House / In-Home → Mailing
                                    ↓
                               Mail Date

Stage 11: Response Analysis

Tracking Response

Mailing → Transaction → Response Analysis → Results
              ↓                                 ↓
          Response                    ┌────────┴────────┐
              ↓                       ↓                 ↓
         Match Back            Response Rate      Client Results
                               & RR Index

Match Back Process

Links responses (purchases) back to the original mailing to measure effectiveness.

Key Metrics

Final Output

• Client Results - Performance report for the client
• Recommendation - Guidance for future mailings

Campaign Timeline Summary

Order → [Weeks] → Merge Cutoff → [Weeks] → Mail Date → [Months] → Results
         ↑                                                           ↓
         └───────────────── Campaign Cycle ──────────────────────────┘
                              (Many Months)

System Components Reference

Related Documentation

• Solution Overview - High-level company and product overview
• Glossary - Complete term definitions
• Products & Services - Product portfolio details

Path2Response Glossary

Source: New Employee Back-Office Orientation (2025-08) and internal documentation

This glossary defines key terms used throughout Path2Response systems and processes. For industry context, see the Industry Knowledge Base.

Contributing

When researching projects, watch for undefined terms:

1. During research - Note unfamiliar acronyms or domain terms
2. Check glossary - See if term is already defined
3. Add if missing - Include definition with source context
4. Format - Bold term, followed by definition paragraph

Example addition:

**PBT (Probability by Tier)**
A model quality metric showing the probability distribution across decile segments. Better models show deeper differentiation between high and low segments.

Recent additions: XGBoost, P2A3XGB, FTB, MPT, Future Projections, Hotline Scoring, ARA, ERR, MGR, Encoded Windowed Variables, Responder Interval (Jan 2026, from DS documentation); DRAX, Orders Legacy, Shiny Reports (Jan 2026, from BERT documentation); PBT (Jan 2026, from MCMC project); Convert Type, Data Bearing, Globalblock, Disabled Title, Parent Title, File Type Tag, Transaction Type, Data Call (Jan 2026, from Data Acquisition SOP)

A

Affinity A model type measuring likelihood of interest in a product category based on behavioral patterns.

Anonymous Visitor A website visitor who is not logged in or otherwise identified by the brand. PDDM solutions like Path2Ignite can match anonymous visitors to postal addresses through co-op data matching, enabling direct mail retargeting.

AOV (Average Order Value) The average dollar amount per order, used to measure campaign profitability.

ARA (Automated Response Analysis) Data Science tool that measures campaign performance by matching responses (purchases/donations) back to mailed households. Provides reports via a Dash app.

AUC (Area Under Curve) Model quality metric measuring the area under the ROC curve. High AUC (top 90s) with low responder count indicates overfitting.

AUCPR (Area Under Precision-Recall Curve) Model quality metric, alternative to AUC, particularly useful for imbalanced datasets like response prediction.

Automation Discount USPS postage discount for mail pieces that can be processed by automated equipment. Requires readable barcodes (IMb) and standard dimensions.

B

BERT (Base Environment for Re-tooled Technology) Path2Response’s unified backend application platform that consolidates internal tools into a single application with a common codebase. BERT incorporates Order App, Data Tools, Dashboards, Response Analysis, and other internal applications. Named in August 2023 (submitted by Wells) after Albert Einstein’s pursuit of a unified field theory, with the Sesame Street character enabling a naming convention for related tools (Ernie, Big Bird, Oscar, etc.).

BIGDBM External data provider used for data enrichment and additional consumer attributes.

Best Name In fulfillment processing, the optimal individual/name associated with a household for direct response marketing. Path2Response outputs the “best name” for each household, which may differ from the original name on a client-provided file. This ensures the most likely responder in the household receives the mailing. Used in standard P2A fulfillments and Path2Optimize. See also: Path2Optimize, Householding.

Brand A specific brand within a client’s portfolio. Multiple brands may share a parent client.

Balance Model A post-merge solution that identifies names representing an incremental direct mail audience after the client’s merge/purge process. Provides additional mailing volume for clients with strong pre-merge results.

Three Balance Model solutions:

Criteria: Strong pre-merge results required (not for mediocre performers), recommended minimum 10M order qty, no significant backend fulfillment processing. Great option for tagged clients.

Turnaround: Same day if file arrives by 9 AM MST (delivery EOD MST); optimal 24 hours.

Positioning:

• Optimal: P2R provides names on frontend (pre-merge) AND backend (post-merge)
• Less than ideal: Backend only (may cause revenue cannibalism)
• Catalog: Lead with Site Visitors, consider higher price point
• Nonprofit: Lead with Path2Acquisition, Site Visitors as options

Revenue: >$550K since 2019 without promotion; 2024 had 23 titles (22 Nonprofit), >$92K revenue, median qty 21,000.

See also: Path2Advantage, Post-Merge, Merge/Purge.

Breakeven The response rate needed for a campaign to cover its costs. Calculated based on campaign costs, average order value, and margin.

Broker See List Broker.

C

Commingling Combining mail from multiple mailers to achieve higher presort densities and better postal discounts. Enables smaller mailers to access discounts normally requiring higher volumes.

Compiled List A mailing list assembled from public and commercial sources (directories, registrations, surveys) rather than actual purchase behavior. Lower cost but typically lower response rates than response lists.

CPM (Cost Per Mille) Cost per thousand - the standard pricing unit for mailing lists. A list priced at $100 CPM costs $100 per 1,000 names.

Campaign A complete direct mail initiative from order through response analysis. Campaigns typically span many months.

CASS (Coding Accuracy Support System) USPS system that standardizes addresses and adds delivery point codes. Required for postal discounts.

Caveance The holding company that owns Path2Response, LLC.

CCPA (California Consumer Privacy Act) California privacy law governing consumer data rights. See also CPRA.

Convert Type In Data Acquisition, the fundamental classification of incoming data files:

• House: Records part of the target title’s house file; suppressed for prospect fulfillments; does not support transactional fields
• Transaction: Records with transactional data that link to house file records via customerId
• Master: Records not associated to a specific target title; affect overall database or modeling (e.g., DMA, select, omit)

See also: File Type Tag, Data Acquisition Overview.

Client A company purchasing prospecting or data services from Path2Response.

Complete/Replacement A transaction file type that fully replaces previous data (as opposed to incremental updates).

Computer Validation Final address verification performed by the service bureau before mailing.

Convert The process of transforming client data into Path2Response’s standard format and layout.

Cookie Browser-based identifier used for tracking website visitors.

CPRA (California Privacy Rights Act) Updated California privacy regulation that supersedes CCPA. Path2Response is compliant.

D

Data Attributes Enhanced consumer data points derived from transaction history and external sources.

Data Bearing (Title) A title that contributes data to the Path2Response cooperative database. Controlled by the “disabled” checkbox in Data Tools. Non-data-bearing titles (disabled) have data that cannot be used in model runs. See also: Disabled Title, Globalblock.

Data Call A client onboarding meeting where Data Acquisition discusses data requirements, file formats, and transfer processes with a new client. New client file analysis SLA is 3 business days with a data call vs 5 business days without one.

Data Broker A business that aggregates consumer information from multiple sources, analyzes/enriches it, and sells or licenses it to third parties. Distinct from list brokers (who facilitate rentals) and data cooperatives (member-based).

Data Cooperative (Data Co-op) A group organized for sharing pooled consumer data between member companies. Members contribute transaction data and gain access to the cooperative’s combined assets. Examples: Path2Response, Epsilon/Abacus, Wiland.

DDU (Destination Delivery Unit) The local post office that handles final mail delivery. Entering mail at DDU level provides the highest postal discount but is logistically complex.

Data Tools Path2Response utilities for data processing and manipulation.

Deceased Suppression process that removes deceased individuals from mail files.

Direct Mail Physical correspondence sent to consumers to encourage patronage of a client.

Disabled (Title) A title status flag in Data Tools indicating the title’s data cannot be used in model runs. Disabled titles will not be selected in preselect. Non-contributing titles should be marked as disabled but NOT marked as DNR/DNM. See also: Data Bearing, Globalblock.

DMA (Direct Marketing Association) Industry organization; also refers to preference processing for consumers who opt out of marketing.

DNM (Do Not Mail) Suppression flag indicating a consumer should not receive mail from any source.

DNR (Do Not Rent) Suppression flag indicating a name cannot be rented to other mailers (but owner can still mail).

DPV (Delivery Point Validation) USPS verification that confirms an address is deliverable.

DPBC (Delivery Point Bar Code) Postal barcode added to mail pieces for sorting efficiency.

Deciles Division of a scored audience into 10 equal groups ranked by predicted response. Decile 1 is highest predicted responders; Decile 10 is lowest. See also Quintiles, Segments.

Direct Mail Agency A full-service marketing firm that manages direct mail campaigns for brands, handling strategy, creative, list selection, production, and analytics. Example: Belardi Wong.

DTC (Direct-To-Consumer) Business model where brands sell directly to consumers rather than through retailers.

Dummy Fulfill / Dummy Ship Internal operational process where a fulfillment is generated but shipped only to Path2Response (orders@path2response.com) rather than the client. Used in Path2Expand workflow to capture the P2A names that need to be excluded from the P2EX universe. After dummy fulfillment, the ImportID is added to Prospect/Score Excludes for the subsequent model. See also: Path2Expand, ImportID.

DRAX Path2Response’s reporting system that replaced the legacy Shiny Reports application. Part of the Sesame Street naming convention for tools related to BERT. Contains two main dashboards:

• Guidance for Preselect Criteria - Summary of client titles, household versions, database growth, housefile reports
• Title Benchmark Report - Summary statistics for variables across titles

E

Encoded Windowed Variables Data Science innovation that compresses multiple time-windowed RFM variables into a single dense variable per title. Solves the problem of windowed variables not making it into the final 300-variable model due to low density. Uses cumulative encoding across windows (30, 60, 90, 180, 365, 720, 9999 days). Resulted in 13-92% increase in unique titles used in final models.

Important: “Encoded Variables” is an internal term only — do not use with clients. This is NOT a new product, new model, or new data source. It’s an enhancement that helps the existing model make better use of existing data. When discussing with clients, frame as improved leverage of recency and synergy signals, which are universally understood terms. See Client Messaging section in data-science-overview.md.

Epsilon/Abacus Major data cooperative competitor (part of Publicis Groupe). Epsilon acquired Abacus Alliance and operates one of the largest marketing data platforms globally.

ERR (Estimated Response Rate) Model metric showing the estimated likelihood a responder will purchase, based on Future Projection and Model Projection. Used to compare different model configurations on the Model Grade Report.

Extract Process that pulls individual-level data from site visitors while applying privacy compliance blocks.

F

First-Party Data Data collected directly by the client from their own customers and website visitors.

Format The structure/layout specification for data files.

Frequency In RFM analysis, how often a consumer makes purchases.

Fulfillment The process of preparing the final mail file for production.

Fulfillment File The completed file ready for shipment to the service bureau.

File Type Tag In Data Acquisition, a classification tag applied to incoming data files that determines how they are processed and converted. Common tags include:

• suppression (house): DNR/DNM housefile names
• transaction (transaction): Housefile transactions only
• complete (transaction): Combined names + transactions
• house (house): Housefile names only
• inquiry (transaction): Catalog requesters
• shipto (house): Ship-to addresses/giftees
• select/omit (master): Files for modeling selection/omission
• mailfile (master): Consumers mailed or to be mailed
• response (master): People who responded to a mailing

See Data Acquisition Overview for complete list.

FTB (First Time Buyer) Model configuration option that limits responders to only those making a purchase for the first time (according to co-op data). Default buyer type for tagged titles.

Future Projections Data Science process to determine the optimal responder interval for model training. Evaluates different responder interval scenarios and delivers comparative correlations between responder intervals and the measurement interval. Key insight: models predict past response, but campaigns need to predict future response.

G-H

Globalblock A client status flag indicating the client’s data is not contributed to the Path2Response cooperative database. Unlike disabled titles (which are loaded but not used in modeling), globalblocked clients have data that is completely excluded from the database. See also: Disabled Title, Data Bearing.

Gross to Net (Gross → Net) The reduction from raw record count to usable record count during file processing. In Path2Optimize, the “gross” count of records on a client-provided file reduces to a “net” count of records that can be matched to qualified individuals/households in the P2R database. Match rates vary based on file quality, recency, accuracy, and duplication. See also: Path2Optimize.

Hashed Email (HEM) Privacy-safe email identifier created through one-way hashing algorithms (MD5, SHA-1, SHA-2).

Holdout Panel A control group that does not receive a mailing, used to measure the incremental impact of a campaign. Results are compared between mailed and holdout groups to calculate true lift.

Hits In merge/purge, records that matched across multiple lists (duplicates found).

House File A client’s existing customer database, typically provided via SFTP.

Householding Process of grouping individuals into household units to prevent duplicate mailings to the same address.

Households File Master file of valid, deduplicated households with unique IDs.

Hotline Scoring Simple regression model that scores prospects with browse data at the target title. Used for programmatic direct mail (PDDM, Swift) and to incorporate browsers for titles whose tag is too new for XGBoost.

I

ID Unique identifier assigned to households or individuals in the system.

Identities Resolved identity records linking digital behavior to postal addresses.

In-Home When a mailing has been delivered and the consumer has seen/opened it.

In-House When a mailing has arrived at the household.

Incremental A transaction file type that adds new records to existing data (as opposed to complete replacement).

Individuals Person-level records extracted from site visitor data.

Intent Signal Behavioral data indicating purchase intent, such as cart abandonment, pages visited, time on site, or order placed. Used in PDDM to identify and prioritize high-value prospects.

IMb (Intelligent Mail Barcode) USPS barcode standard required for automation discounts. Contains routing information and unique piece identifiers for tracking.

ImportID Unique identifier for a fulfillment file. Used in Path2Expand workflow to exclude P2A fulfillment names from the P2EX universe by adding the ImportID to the Prospect/Score Excludes field. See also: Path2Expand, Dummy Fulfill.

Incoming Routes SFTP paths configured in Data Tools for receiving client data files. Routes should be set under the parent title even if the parent::child duplicate is not data bearing. For example, “Vionic” incoming routes are set under “Caleres” (the parent). The parent can be identified from the <parent::child> syntax in dropdown searches.

J-K

JSON Data format used for storing transaction records.

Keycode Unique identifier printed on mail pieces to track response source.

L

Layout The field structure and format specification for data files.

Lettershop A facility specializing in mail piece assembly and preparation, including inserting, tabbing, labeling, and sorting. Often part of a service bureau.

License Files Data provided through licensing arrangements rather than cooperative membership.

Lift The improvement in response rate achieved through modeling or optimization. Example: “Co-op optimization delivered 25% lift over unscored names.”

LTV (Lifetime Value) The total revenue a customer is expected to generate over their entire relationship with a brand. PDDM campaigns aim to acquire customers with higher LTV and increase LTV of existing customers through retention.

List Broker A specialist who helps mailers find and rent appropriate mailing lists. Earns commission (~20%) from list managers. No additional cost to mailer for using a broker.

List Exchange A barter arrangement where two organizations trade one-time access to their mailing lists, often at no cost or minimal processing fees.

List Manager A company that represents list owners, handling rentals, maintenance, promotion, and compliance monitoring (including seed name tracking).

List Owner An organization that owns customer/subscriber data available for rental (e.g., magazines, catalogs, nonprofits).

List Rental One-time use of a mailing list owned by another organization. Names cannot be reused without paying again. Lists typically include seed names to verify compliance.

Lists of Lists Attribution showing which source list(s) each name originated from.

M

Marketing Mail USPS mail class (formerly Standard Mail) used for advertising, catalogs, and promotions. Lower cost than First Class but slower delivery (3-10 days).

Mail Date The date a mailing enters the postal system.

Mail File The final, production-ready file sent for printing and mailing.

Mailing The actual direct mail campaign sent to consumers.

Match Back Process of linking responses (purchases) back to the original mailing to measure effectiveness.

MD5 Hashing algorithm used for creating hashed email identifiers.

Merge Cutoff Deadline for including names in a mailing. After this date, the file is locked for production.

Merge/Purge Process of combining multiple lists and removing duplicates.

Modeling Statistical analysis to predict consumer response behavior.

Monetary In RFM analysis, how much a consumer spends.

MGR (Model Grade Report) A Data Science report containing key model quality metrics: AUC, AUCPR, and ERR. Used to evaluate whether a model is performing well before deployment. High AUC values (90s) with few responders may indicate overfitting. See also: AUC, AUCPR, ERR, PBT.

MPT (Model Parameter Tuning) Process of determining the best preselect criteria and model settings to achieve the optimal estimated response rate (ERR). Steps include: setting circulation size, running Future Projections, determining responder interval, comparing prospecting types via 5% models. Future Projections must complete before 5% models are run. See also: Future Projections, ERR.

Multi-Buyer See Multis.

Multis Names appearing on multiple source lists, identified during merge/purge. Multi-buyers typically show higher response rates and are often treated as a premium segment.

N

Names Consumer name data within the householding process.

NCOA (National Change of Address) USPS database used to update addresses for consumers who have moved. Typically covers 48 months of address changes.

NDC (Network Distribution Center) Large regional USPS facilities that serve as hubs for mail processing. NDC destination entry discounts were eliminated in July 2025.

Nets Unique records remaining after merge/purge deduplication.

Nonprofit Vertical serving charitable organizations. Uses terminology like “donors,” “donations,” “gifts.”

Nth-ing Statistical sampling technique selecting every Nth record, typically for testing.

O

Omits Specific records to exclude from a mailing.

Order The initial request that initiates a campaign.

Order App System for order entry and campaign management. Now a module within BERT. The legacy deployment (“Orders Legacy”) remains operational while final functionality (Shipment) is ported to BERT.

Order Processing Validation and routing of orders through the system.

Orders Legacy The original standalone Order App deployment on EC2. Being phased out as functionality is migrated to BERT. Shipment is the last remaining functionality to port; once complete, Orders Legacy will be decommissioned.

P

Pander Suppression list of consumers who have requested not to be contacted.

Parent (Title) A primary title that has related sibling/child titles. Clients with multiple titles should have a parent title identical to the client name. Data contacts, incoming routes, and fulfillment routes should be set under the parent title. Child titles use the syntax <parent::child> in Data Tools dropdowns. Example: “Vionic” is a child under “Caleres” (parent). See also: Sibling, Incoming Routes.

Path2Acquisition Path2Response’s core product for customer acquisition through prospecting. Uses XGBoost classification models (P2A3XGB) to predict response likelihood. Can be used in Balance Model position (post-merge) to provide incremental prospects. See also: P2A3XGB, Balance Model.

Path2Advantage Path2Response’s housefile site visitor product. Identifies existing customers who have recently visited the client’s website, enabling re-engagement campaigns. Used in Balance Model position (post-merge) to provide incremental housefile names. Default window is last 30 days. Potential limitation: may generate small quantities for some titles. Requires client to have a tracking tag on their website.

Path2Expand (P2EX) Product that provides incremental prospect names beyond the primary Path2Acquisition universe. Cannot be run standalone — Path2Acquisition must always run first in the order.

How it works:

1. Run Path2Acquisition model normally (MPT, 5% counts, front-end omits)
2. P2A names are excluded from P2EX universe (via ImportID in Prospect/Score Excludes)
3. P2EX uses same model parameters as P2A (cloned model)
4. When fulfilling both models, P2A is Fulfillment Priority #1

Example: Client orders 250K from P2A model with segment size 100K. Dummy fulfill/ship 300K to omit on frontend of P2EX model, then P2EX provides additional names beyond P2A.

Key constraint: Model parameters must NOT be changed between P2A and P2EX — only the Prospect/Score Excludes field is modified.

See also: Path2Acquisition, Balance Model.

Path2Ignite Path2Response’s Performance-Driven Direct Mail (PDDM) product. Combines site visitor identification with co-op transaction data for triggered/programmatic direct mail with 24-48 hour turnaround. Can be white-labeled by agencies. See also Path2Ignite Boost, Path2Ignite Ultimate.

Path2Ignite Boost Path2Ignite variant that delivers top Path2Acquisition modeled prospects daily. Model is rebuilt weekly with fresh data. Audiences are incremental to (not duplicates of) site visitor-based Path2Ignite audiences.

Path2Ignite Ultimate Path2Ignite variant combining both site visitor-based targeting (Path2Ignite) and model-based targeting (Boost) for maximum daily growth potential.

Path2Optimize (P2O) Data enrichment product that scores and optimizes client-provided files with minimal transactional data. Available for Contributing and non-Participating brands.

How it works:

1. Client provides file (leads, compiled lists, retail data, lapsed customers, etc.)
2. P2R matches records to qualified individuals/households in database (“Gross → Net”)
3. Net identified records become the scoring universe
4. P2A3XGB model scores records using client’s own responders (if available) or proxy responders from similar category titles
5. Output: Optimized name/address, original client ID, segment/tier indicator

Common file types optimized:

• Sweeps/giveaway entries
• Top-of-funnel leads (limited data elements)
• Compiled/rental lists purchased from other brands
• Retail store location data
• Lapsed customer files
• Cross-sell files from other titles within parent company
• Housefile portions needing optimization

Key concepts:

• Gross → Net: Raw file count reduces to subset that matches P2R database; varies by file quality
• Best Name logic: Outputs optimal individual for direct response in household (may differ from original name)
• June 2025 enhancement: Can now return client-provided ID (CustID, DonorID) with fulfillments

See also: P2A3XGB, Best Name, Proxy Responders.

PBT (Probability by Tier) A model quality visualization showing the predicted probability of response across 10 equal-sized tiers (segments). The graph uses box plots to display the distribution of probabilities within each tier. A good PBT graph shows a smooth decline from tier 1 (highest probability) to tier 10 (lowest probability). Key elements:

• X-axis: Tiers 1-10 (segments ranked by predicted response probability)
• Y-axis: Probability of response
• Blue box: Middle 50% of probabilities (interquartile range)
• Whiskers: Top and bottom 25% of probabilities
• Orange line: Median probability for that tier
• “Depth”: The vertical spread between tier 1 and tier 10 - greater depth indicates stronger model discrimination A flat or erratic PBT graph indicates a model problem requiring Data Science review. See also: Model Grade Report, IRG.

P2A3XGB Path2Response’s production XGBoost classification model for Path2Acquisition. The pipeline consists of six steps: Train Select, Train, Score Select, Score, Reports Select, Reports. Initial training uses 50,000+ variables with up to 110,000 households; the final scoring model uses only the top 300 most important variables to efficiently score 35+ million households. See also: XGBoost, Encoded Windowed Variables.

PDDM (Performance-Driven Direct Mail) Category of direct mail products that use real-time behavioral triggers (site visits, cart abandonment) to send personalized mail quickly. Path2Response’s product in this category is Path2Ignite.

PII (Personally Identifiable Information) Data that can identify an individual. Limited usage at Path2Response.

Pixel Image-based tracking mechanism for email and web.

Preselect Process defining the available universe of names for selection.

Presort Sorting mail by ZIP code/destination before presenting to USPS. Deeper presort levels (5-digit, carrier route) earn higher discounts.

Priority (List) In merge/purge, the ranking that determines which source “owns” a duplicate name. House file typically gets highest priority, followed by best-performing rented lists.

Programmatic Direct Mail Automated, trigger-based direct mail that responds to real-time events (site visits, cart abandonment). Also called triggered mail. See also PDDM, Path2Ignite.

Propensity A model type measuring likelihood to respond to a specific offer.

Post-Merge The phase after a client’s merge/purge process is complete. Balance Models operate in the post-merge position, providing incremental names after duplicates have been removed. Path2Response optimally provides names both pre-merge (standard Path2Acquisition) and post-merge (Balance Model). See also: Balance Model, Pre-Merge, Merge/Purge.

Pre-Merge The phase before a client’s merge/purge process, when list sources are assembled but not yet deduplicated. Standard Path2Acquisition orders are delivered pre-merge. Clients with strong pre-merge results are candidates for post-merge Balance Models. See also: Post-Merge, Balance Model, Merge/Purge.

Proxy Responders In modeling, a pool of responders from a cohort of similar titles used when the client’s own customer data is unavailable or insufficient for model training. Used in Path2Optimize when scoring client-provided files that lack transactional history. The proxy responders are drawn from titles in the same category as the client. See also: Path2Optimize, Responder.

Prospect A consumer who has not yet responded to a specific title.

Q

Quintiles Division of a scored audience into 5 equal groups ranked by predicted response. Similar to deciles but with 5 groups instead of 10. See also Deciles, Segments.

R

Recency In RFM analysis, how recently a consumer made a purchase.

Recommendation Guidance provided to clients for future mailings based on results analysis.

Responder A consumer who has previously responded to a specific title.

Responder Interval The time period during which responder data is collected for model training. Two types: Recency (most recent months with data) and Seasonal (same months from prior year, for seasonal products). Future Projections helps determine the optimal responder interval. See also: Future Projections, Prospect Range, windowedEndDate.

Response A consumer action (typically a purchase) resulting from a mailing.

Response Analysis Process of measuring and evaluating campaign performance.

Response List A mailing list of people who have taken an action (purchased, donated, subscribed). Higher quality and cost than compiled lists because it includes proven responders.

Response Rate

Retargeting Marketing to people who have previously interacted with a brand (visited website, abandoned cart, etc.) but did not convert. Direct mail retargeting sends physical mail to these visitors. See also PDDM, Triggered Mail.

Remarketing Often used interchangeably with retargeting. Refers to re-engaging consumers across channels (online and offline) after initial interaction. Percentage of mail recipients who responded (purchased).

Results Campaign performance data including response rates and financial metrics.

RFM (Recency, Frequency, Monetary) Fundamental framework for analyzing and predicting consumer behavior.

RR Index Response rate indexed against a baseline for comparison.

S

SCF (Sectional Center Facility) USPS facilities serving specific 3-digit ZIP code areas. SCF destination entry provides postal discounts (reduced in July 2025 but still significant).

Score File Output of modeling containing consumer scores and segments.

Seed Names Decoy addresses inserted into rented lists by list owners to verify one-time use compliance and monitor mail content/timing. Unauthorized re-use is detected when seeds receive duplicate mailings.

Segments Groups of consumers categorized by score ranges (deciles, quintiles, etc.).

Selects Additional targeting criteria applied to a mailing list (e.g., recency, geography, dollar amount). Each select typically adds $5-$25/M to base list price.

Service Bureau Third-party facility that produces and mails direct mail pieces. Services include merge/purge, printing, lettershop, and postal optimization.

SFTP Secure File Transfer Protocol - method for transferring house files and data.

SHA-1, SHA-2 Secure hashing algorithms used for creating hashed email identifiers.

Shiny Reports Legacy R-based reporting application. Fully replaced by DRAX. See DRAX.

Shipment Delivery of fulfillment file to the service bureau.

Sibling A related title within the same brand family.

Site Tag JavaScript tracking code placed on client websites to capture visitor behavior.

Site Visitors Aggregated data on website visitor behavior.

SPII (Sensitive PII) Highly sensitive personal data (SSN, credit card, passwords). NOT allowed at Path2Response.

Suppressions Categorical exclusions from mailings.

Swift Belardi Wong’s programmatic postcard marketing product. Uses SmartMatch (visitor identification) and SmartScore (predictive scoring). Path2Response provides the data layer for Swift.

Synergistic Titles Related titles whose responders are likely to respond well to a target title.

T

Triggered Mail Direct mail automatically sent based on a behavioral trigger (site visit, cart abandonment, inactivity). See also Programmatic Direct Mail, PDDM.

Title A specific catalog or mailing program (e.g., “McGuckin Hardware”).

Title Key Unique identifier linking transactions to specific titles.

Title Responders Consumers who have responded to a specific title.

Transaction A purchase record from a consumer.

Transaction Type In Data Acquisition, a classification applied to transactional records based on the nature of the action:

• purchase - Amount > 0; maps to PURCHASE for modeling
• donation - Charitable gift; maps to PURCHASE
• subscription - Advance payment for recurring services; maps to PURCHASE
• return - Amount < 0
• inquiry - Catalog request; amount = 0
• cancellation - Cancelled order
• pledge - Commitment to future donation
• shipment - Where items shipped (replaces deprecated “giftee”)
• unknown - Catch-all; should not be modeled

“The Subscription Line”: Netflix (monthly billing) is NOT a subscription; National Geographic (advance payment for recurring delivery) IS a subscription.

Transactions The collection of purchase records stored in the system.

U

Universe The total pool of potential mail recipients available for selection.

V

Variables Data points generated by BERT and used for modeling.

Verticals Industry categories served: Catalog, Nonprofit, DTC, Non Participating.

W

Windowed Variables (Simulation Variables) Variables based on windowedEndDate that capture purchasing behavior within specific time windows: 30, 60, 90, 180, 365, 720, or 9999 days. Challenge: rarely make it into the final 300 model variables due to low density in smaller titles or time windows. Solution: Encoded Windowed Variables compress multiple windows into single dense variables. See also: Encoded Windowed Variables, windowedEndDate.

windowedEndDate The date that defines “current time” for model training, set to a date in the past. All windowed variables are calculated relative to this date. Critical rules: (1) must be the same for both prospects and responders; (2) Prospect End Date = windowedEndDate; (3) Prospect Start Date = 1 year before windowedEndDate; (4) Responder Start Date = 1 day after windowedEndDate. See also: Windowed Variables, Responder Interval, Prospect Range.

White Label A product or service produced by one company that other companies rebrand and sell as their own. Path2Ignite can be white-labeled by agencies, who manage the client relationship while Path2Response provides the data/audience layer invisibly.

Wiland Major data cooperative competitor headquartered in Niwot, CO. Known for 160B+ transactions, heavy ML/AI investment, and privacy-conscious approach (doesn’t sell raw consumer data).

Worksharing USPS term for mailers doing work (sorting, transporting, preparing) that USPS would otherwise perform. Worksharing earns postal discounts.

X

XGBoost (eXtreme Gradient Boosting) A gradient boosting framework using decision trees, and the core technology behind Path2Response’s P2A3XGB model for Path2Acquisition. Well-suited for: large datasets with many variables, binary classification (responder vs non-responder), handling missing values, and feature importance ranking. See also: P2A3XGB, Modeling, Variables.

$ Metrics

$ Per Book Revenue generated per catalog mailed.

$ / Book Index Revenue per book indexed against a baseline for comparison.

Related Documentation

• Solution Overview - Company and product overview
• Path2Acquisition Flow - Complete solution architecture
• Products & Services - Product portfolio details
• Data Acquisition Overview - Data Ops processes and SLAs
• Production Team Overview - Client Support Services processes
• Data Science Overview - Modeling and DS processes
• Industry Knowledge Base - Direct mail industry context
  • Direct Mail Fundamentals
  • Data Cooperatives
  • List Rental & Exchange
  • Service Bureau Operations
  • Postal Operations
  • Competitive Landscape

Path2Response Company Overview

Related Documentation:

• Solution Overview - Mission, core concepts, verticals
• Path2Acquisition Flow - How the core product works
• Glossary - Term definitions

Company Identity

Official Name: Path2Response, LLC Parent Company: Caveance (holding company) Taglines: Marketing updates taglines regularly; recent examples include “Marketing, Reimagined” and “Innovation and Data Drive Results” Industry: Marketing Data Cooperative / Data Broker Founded: ~2015 (celebrating 10-year anniversary in 2025)

Mission & Vision

Path2Response was founded on the principle that “There has to be a better way” for how data cooperatives operate. The company focuses on creating marketing audiences using multichannel datasets combined with advanced machine learning.

Core Mission: Help clients reach their ideal campaign audiences using real-world consumer spending behavior data.

Location

Path2Response is a fully remote company with employees distributed across ~15 US states.

Corporate Address (legal/mail only): 390 Interlocken Crescent, Suite 350 Broomfield, CO 80021

Contact:

• Phone: (303) 927-7733
• Email: inquiries@path2response.com
• LinkedIn: https://www.linkedin.com/company/path2response-llc/

Key Differentiators

Multichannel Dataset

Integrates online and offline transaction and behavioral data to identify signals competitors miss.

Machine Learning Approach

Uses a single configurable model with thousands of iterations based on billions of data points, emphasizing automated data processing rather than manual methods.

Performance Focus

97% client retention rate with 20-60% performance lift for campaigns using their digital data.

Data Freshness

Daily updates to their database, with weekly updates to individual-level purchase transaction records.

Company Scale

• Data Volume: Billions of individual-level purchase transaction records
• Digital Signals: 5B+ unique site visits daily
• Coverage: Nationwide consumer spend behavior across 40+ product categories
• Update Frequency: Data updated weekly; digital behavior signals daily

Regulatory Status

Path2Response is registered as a data broker under Texas law with the Texas Secretary of State.

Industry Recognition

• 97% client retention rate
• 30+ years of collective leadership experience in cooperative database marketing
• Industry pioneers who “launched the marketing cooperative data industry”

Path2Response Products & Services

Product Portfolio Overview

Path2Response offers a suite of data-driven marketing solutions built on their cooperative database of billions of consumer transaction records.

For technical understanding: See Path2Acquisition Flow for how the core product works, and Glossary for term definitions.

Core Products

Path2Acquisition

What It Is: Custom audience creation solution for new customer acquisition.

How It Works:

• Leverages billions of individual-level consumer purchase behaviors
• Creates thousands of model iterations to identify optimal performance and maximum scale
• Delivers custom audiences within hours (not days or weeks)
• Combines online and offline spending patterns

Key Features:

• Custom audiences tailored to client goals and online signals
• Machine learning reduces human bias
• Multichannel data at scale
• Results transparency before launch

Use Cases: Direct mail campaigns, display advertising, multichannel marketing

Path2Ignite (Performance Driven Direct Mail)

What It Is: Performance-driven direct mail (PDDM) solution that sends postal mail to recent website visitors—both known customers/donors and anonymous visitors—combining first-party co-op data with site visitor intent signals.

Core Concept: PDDM is automated “trigger” or “programmatic” mail deployed as close to real-time as possible. Daily mailings maximize recency as a performance driver. Typically uses cost-efficient formats (postcards, trifolds) that are quick to print.

Product Variants

How It Works

Path2Ignite (Site Visitor-Based):

1. Day Zero: Client’s site visitors captured via tags; data feed received by Path2Response
2. Next Day: Site visitors matched and optimized with proprietary scoring algorithm
3. Fulfillment: Audience fulfilled to printer/mailer
4. Mail: Deployed into USPS mail stream

Path2Ignite Boost (Model-Based):

1. Client’s Path2Acquisition model custom-built with most recent data (rebuilt weekly)
2. Top modeled prospect audiences fulfilled to printer daily
3. Audiences are unique from/incremental to site visitor-based audiences

Intent Signals Used

• Cart abandonment
• Pages visited
• Order placed/amount
• Session-specific behavior
• Co-op transaction data matching

Use Cases

New Customer/Donor Acquisition:

• Reach anonymous site visitors through a unique multichannel touchpoint
• PDDM as part of integrated acquisition strategy

Existing Customer/Donor Engagement:

• Active customers: Increase retention, drive repeat transactions, boost LTV
• Lapsed customers: Reactivate—a recent website visit means they’re “raising their hands”

Proven Results (vs. Holdout Panels)

Go-to-Market Models

White-Label (for Agencies):

• Agency manages order process, creative, printer selection
• Path2Response provides data/audiences invisibly
• Agency owns client relationship

Data Provider (for Agency Partners like Belardi Wong):

• Path2Response provides data layer for agency’s branded product (e.g., Swift)
• Path2Response not visible to end client

Implementation Process

1. Agency Setup: White-labels Path2Ignite; manages order process, sends customer file, owns creative and printer selection
2. Privacy Review: Path2Response reviews privacy policy, sends tag pixel credentials, assists with pixel setup
3. Data Onboarding: Client provides customer housefile and sends weekly customer/transaction/suppression files
4. Audience Build: Path2Response builds, fulfills, and ships Path2Ignite and Boost names to printer
5. Production: Printer prints and mails postcards within 48 hours
6. Analytics: Path2Response runs Response Analysis and provides regular updates

Path2Response Responsibilities

• Tag and site visitor identification
• Model, score, rank
• List ops: test & holdout panels, fulfillment
• Audience strategy, optimization, best practices
• Analytics and results reporting

Agency Partner Responsibilities

• Creative for PDDM piece
• PDDM-capable printer selection
• Strategy/support for client
• Sell Path2Ignite to clients/prospects

Co-op Participation Impact

• Co-op participants: Signature scoring algorithm + custom acquisition model
• Non-participants: Signature scoring algorithm only

Market Context

• PDDM market spend: $317MM (2022), growing at 27.4% CAGR
• Projected: $523MM by 2025
• Represents ~1.3% of total direct mail spend and rising

Digital Audiences

What It Is: 300+ unique audience segments powered by first-party transaction data for digital marketing campaigns.

Audience Categories:

Data Foundation:

• Billions of individual purchase transaction records
• Daily updates tracking digital behavior signals
• 5+ billion unique site visits
• Nationwide spending data across 40+ product categories

Path2Contact (Reverse Email Append)

What It Is: Transforms anonymous email addresses into actionable customer data with postal addresses and responsiveness rankings.

Process:

1. Upload: Submit email addresses with missing contact details
2. Enrichment: Match against consumer database to add names and postal addresses
3. Ranking: Proprietary response rankings based on actual purchase behavior

Key Capabilities:

• Converts “mystery emails” into complete records
• Behavioral scoring based on real-world purchase patterns
• Enables multichannel engagement (email + direct mail)

Use Cases:

• Nonprofits: Connect donors across multiple channels
• Retailers: Engage online shoppers with offline campaigns
• Agencies: Enhance client campaign effectiveness

Path2Optimize (Data Optimization)

What It Is: Data enrichment service that scores and optimizes client-provided files with minimal transactional data using P2A3XGB modeling. Available for Contributing and non-Participating brands.

Accepts:

• External compiled files and third-party lists
• Lead generation data (contests, sweeps, info requests)
• Consumer contact information from retail locations
• Lapsed customer files
• Cross-sell files from other titles within parent company
• Housefile portions needing optimization

How It Works:

1. P2R receives client-provided file
2. Records matched to P2R database (“Gross → Net” — match rate varies by file quality)
3. Net records become scoring universe
4. P2A3XGB model scores using client responders (if available) or proxy responders from similar category
5. Output: Optimized name/address, original client ID (June 2025+), segment/tier indicator

Delivers:

• Complete “best name” and postal data (optimal individual for direct response in household)
• P2A3XGB model scoring ranking consumers by response likelihood
• Tiered rankings for targeting optimization
• Client-provided ID returned with fulfillment (CustID, DonorID)

Proven Results:

• 100% performance improvement for outdoor apparel retailer
• 60% response rate lift over other providers for home furnishings
• 127% average gift increase for education nonprofit

Path2Expand (P2EX)

What It Is: Product providing incremental prospect names beyond the primary Path2Acquisition universe. Extends reach when clients need more names than their P2A segment delivers.

Key Constraint: Cannot run standalone — Path2Acquisition must always run first in the order.

How It Works:

1. Run Path2Acquisition model normally (MPT, 5% counts, front-end omits)
2. Dummy fulfill/ship P2A names to exclude from P2EX universe
3. Clone P2A model for P2EX (same parameters, only add ImportID to Prospect/Score Excludes)
4. P2EX delivers additional names beyond P2A segment
5. When fulfilling both, P2A is Fulfillment Priority #1

Use Case: Client orders 250K from P2A model, but segment size is only 100K. Path2Expand provides additional incremental names beyond the P2A universe.

Comparison with Balance Model:

• Balance Model: Post-merge solution (after client’s merge/purge)
• Path2Expand: Pre-merge extension (before client’s merge/purge, same order as P2A)

Path2Linkage (Data Partnerships)

What It Is: “The ultimate source of truth for transaction-based US consumer IDs” - enabling data partnership arrangements.

Coverage & Scale:

• Nearly every U.S. household
• Hundreds of millions of individual consumer records
• Updated 5 times weekly

Data Characteristics:

• Built on first-party (Name & Address) data linked to transaction history
• Excludes unverified third-party data sources
• Compliant with federal and state privacy legislation

Companion Product - Path2InMarket:

• Individual-level transaction data with recency indicators
• Target “in-market active spenders” in 10+ categories
• Transaction recency flags and spend indices

Reactivation Services

What It Is: Reactivates lapsed buyers and donors through targeted outreach.

Capabilities:

• Identifies previously engaged customers for re-engagement
• Targets multichannel donor behavior
• Leverages actual giving data updated weekly

Data Foundation

All products are built on Path2Response’s cooperative database:

Delivery & Integration

• Turnaround: Custom audiences delivered within hours
• Channels: Direct mail, digital display, mobile, Connected TV
• Formats: Multiple delivery formats for campaign deployment

Path2Response Technology & Data

Data Cooperative Model

What is a Data Cooperative?

Path2Response operates as a marketing data cooperative where member organizations (retailers, nonprofits, catalogs) contribute customer data in exchange for access to other consumers’ information. This creates a shared pool of transaction data that benefits all members.

How It Works

1. Members contribute their customer transaction data to the cooperative
2. Path2Response aggregates data across all members
3. Machine learning models identify patterns and build audiences
4. Members access audiences of consumers who have demonstrated relevant behaviors

Data Assets

Transaction Data

Digital Behavior Signals

Consumer Records

Data Sources

Primary Sources

• Cooperative Database: Member-contributed customer transaction data
• Website Tracking: Cookies and similar technologies capturing browsing behavior
• Digital Activity: Online behavior signals and site visitation data

Secondary Sources

• Public records
• Census data
• Telephone directories
• Licensed third-party data
• Online activity providers

Technology Platform

Machine Learning Approach

Path2Response emphasizes automated, ML-driven data processing:

• Single Configurable Model: One model architecture with thousands of iterations
• Scale: Processes billions of data points per model run
• Speed: “Thousands of iterations in a single day”
• Automation: Reduces manual methods and human bias
• Optimization: Identifies optimal performance and maximum scale

Key Technical Capabilities

Data Processing Pipeline

1. Ingest: Daily digital signals, weekly transaction data
2. Match: Link records across sources using first-party identifiers
3. Model: Run ML iterations to build optimized audiences
4. Score: Rank consumers by response likelihood / direct response-readiness
5. Deliver: Provide audiences for campaign deployment

Data Quality & Differentiation

First-Party Focus

Path2Response emphasizes using verified, transaction-based data:

“Actual, verified buying behavior—not ‘modeled’ intent or stagnant surveys”

“Proven shoppers, not ‘ghost’ leads”

Data Freshness

• Transaction records updated weekly
• Digital behavior signals updated daily
• Site visitor identification within 24-48 hours

Exclusions

Path2Linkage explicitly excludes unverified third-party data sources to maintain data quality.

Privacy & Compliance

Regulatory Status

• Registered as a data broker under Texas law with the Texas Secretary of State
• SOC-2 certified (mentioned in nonprofit marketing materials)

Consumer Privacy Rights

Data Retention

“We will retain personal information for as long as necessary to fulfill the purpose of collection” including legal requirements and fraud prevention.

Compliance Framework

Path2Linkage is described as “compliant with federal and state privacy legislation, including disclosure, consent, and opt-in requirements.”

Integration Channels

Audiences can be deployed across:

• Direct mail
• Digital display advertising
• Mobile advertising
• Connected TV (CTV)
• Multi-channel campaigns

Path2Response Team & Leadership

Source: 2025-Q4 EOQ Organization Chart (page 51) Last Updated: January 2026

Executive Team

Organization by Department

Finance & Administration (Phil Hoey, CFO)

Sales & Client Development (Chris McDonald, CRO/President)

Client Partners

Client Development

Marketing

Technology (Michael Pelikan, CTO)

Engineering (John Malone, Director of Engineering)

Data Science (Igor Oliynyk, Director of Data Science)

Operations (Karin Eisenmenger, COO)

Data Acquisition

Client Support

Performance & Results

Product Management

Note: Andrea Gioia held this role briefly (~50 days) after Wells Spence departed. Position currently unfilled (as of January 2026).

New Initiatives

Digital Strategy (Curt Blattner, VP Digital Strategy)

Key Contacts by Function

Team Size Summary

Location

Path2Response is a fully remote company. Employees are distributed across ~15 US states.

Corporate Address (legal/mail only): 390 Interlocken Crescent, Suite 350 Broomfield, CO 80021

Related Documentation

• CLAUDE.md - Development team details
• Data Science Overview - DS team and processes
• PROD-PATH Process - How PROD and PATH teams interact

Path2Response Markets & Clients

Target Market Segments

Primary Customer Types

Industries Served

Retail & E-Commerce

Nonprofit Sector

Professional Services & Financial

Agencies

• Full-service direct marketing agencies
• Digital marketing agencies
• Nonprofit-focused agencies
• Performance marketing agencies

Case Study Results

Nonprofit Results

Brand Results

Client Testimonials

“From a performance perspective, the Path2Response audiences are consistently among the top performers.” — Director, Full Service Direct Marketing Agency

“The site tag worked! Great results from our tag test for our client in July.” — Senior Acquisition Planner

“Got our clients returns this morning. Average Gift outstanding. That ROI was the tops by 22%.” — Account Manager of Acquisition Services

“[Path2Response delivers] double response rates compared to other sources.” — Direct Marketing Manager

Client Retention

97% client retention rate — indicating high satisfaction and ongoing value delivery.

Solutions by Market

For Nonprofits

• Donor acquisition modeling
• Multichannel donor targeting
• Lapsed donor reactivation
• Giving behavior analysis
• 19 specialized donor categories

For Retail/DTC Brands

• Path2Acquisition (custom intent audiences)
• Path2Ignite (performance-driven direct mail)
• Digital audience targeting
• Site visitor reactivation
• Lapsed buyer recovery

For Agencies

• White-label audience solutions
• Client campaign enhancement
• Multi-channel deployment
• Performance tracking and reporting
• Daily campaign capabilities

Path2Response Project Inventory

Complete inventory of P2R codebases organized by technology stack. Use this as a reference for documentation efforts and understanding the technical landscape.

Base path: /Users/mpelikan/Documents/code/p2r/

Node.js Projects (package.json)

Core Business Systems

Infrastructure & CDK

Operations & DevTools

Business Services & Integrations

Client Tools

Java/Scala Projects (pom.xml)

Core Processing (coop-scala)

The coop-scala repository is the heart of P2R’s data processing - Spark jobs running on EMR for audience building, householding, and fulfillment.

Key Spark Jobs (coop-scala/spark-parent-pom/)

Core Libraries (coop-scala/scala-parent-pom/)

Client Libraries

CASS Processing

Python Projects

Ansible Projects

SBT Projects (Scala)

Documentation Priority

Tier 1: Core Business (High Impact)

These systems are critical for understanding P2R operations:

1. bert - Target platform for all migrations
2. order-processing - How orders flow through the system
3. coop-scala (overview) - Core data processing engine
4. response-analysis - ARA functionality
5. operations - Day-to-day operational tooling

Tier 2: Supporting Infrastructure

6. cdk-backend - AWS infrastructure patterns
7. infrastructure - Core AWS setup
8. devtools - Development workflow automation
9. biz-services - Business logic layer
10. client - Client-facing tools

Tier 3: Integrations & Utilities

11. 4cite-api - Partner integration
12. shopify - Shopify integration
13. sovrn - Sovrn integration
14. javascript-shared - Shared code
15. ds-modeling - Data Science tools

Legacy Systems (BERT Migration Candidates)

Statistics

Source: /Users/mpelikan/Desktop/projects/.txt* Created: 2026-01-24

BERT Application Overview

Base Environment for Re-tooled Technology

BERT is Path2Response’s unified backend and frontend application platform. It’s a TypeScript monorepo containing AWS CDK infrastructure, a React frontend, and an API generation layer that maintains synchronized contracts between frontend and backend.

Origin & Naming

Named: August 2023 Submitted by: Wells Selection: Executive team vote following company-wide naming contest

Name Inspiration

1. Albert Einstein — Championed the pursuit of a unified field theory, merging “seemingly un-mergable knowledge of the universe into a new paradigm, facilitating all kinds of good new pursuits/iterations and breakthroughs.”

2. Sesame Street — Bert (short for Albert) enables a naming convention for related tools: Ernie, Big Bird, Oscar, etc.

Purpose

BERT serves as the consolidation target for legacy systems and provides:

• Job Management - 25+ job types for data processing workflows
• Model Runs - Data science model execution and monitoring
• Order Fulfillment - End-to-end order processing workflows
• Jira Integration - Bidirectional sync with Jira for operations
• Reporting & Dashboards - Business intelligence and metrics
• Data Management - Client, title, and audience management

Target Users: Data Scientists, Operations, Developers, Stakeholders

Migration Status

BERT incorporates and will incorporate the following internal applications:

Order App Migration

Order App functionality has been migrated to BERT. The legacy orders deployment remains operational while final functionality is ported:

Architecture

Monorepo Structure

/bert/
├── api-generation/        # API contract generation engine (TypeScript)
├── cdk/                   # AWS CDK infrastructure & Lambda functions
├── react-frontend/        # React SPA frontend
├── scripts/               # Deployment & utility scripts
├── common-shared/         # Shared types (symlinked to react-frontend)
└── package.json           # Root workspace orchestration

Version: 335.0.0-SNAPSHOT Node Requirements: Node ~24, npm ~11

Technology Stack

Core Components

1. CDK Backend (/cdk/)

AWS CDK infrastructure-as-code containing Lambda functions and serverless orchestration.

Entry Point: bin/bert-app.ts

Two CDK Stacks:

• BertAppStack - Application infrastructure
• BertAppCodePipelineStack - CI/CD pipeline

Primary Constructs:

API Endpoints: 38+ namespaces in /cdk/api/proxy-mongo-backed-readonly/endpoints/:

2. React Frontend (/react-frontend/)

Single Page Application built with React and Material-UI.

Entry Point: src/index.tsx

Application Providers:

• Redux store
• AWS configuration
• Cognito authentication
• User preferences
• Alerts & notifications
• Dark mode theming

44 Page Routes including:

• Dashboards, Queue, Scheduler
• Orders, Fulfillments, Shipments
• Digital Audiences, Title Management
• Recency Analysis, Model Runs
• Admin, Preferences, Alerts
• API Debug (developers)
• Jira Sync Status

Component Organization:

3. API Generation (/api-generation/)

Maintains single source of truth for API contracts, preventing frontend/backend type mismatches.

Workflow:

1. Define - API contract in react-frontend/src/common-shared/api/definitions/
2. Generate - Run npm run generate from root
3. Generated Artifacts:
   • common-shared/api/gen/ - Shared types & validation
   • react-frontend/src/lib/api/gen/ - Frontend API client
   • cdk/api/proxy-mongo-backed-readonly/gen/ - Backend routing
4. Implement - Lambda handler at cdk/api/.../endpoints/{namespace}/{function}/handler.ts

Infrastructure

Authentication

• Cognito User Pool with Google OIDC social login
• JWT tokens for API authorization
• Cognito App Client tied to CloudFront

Networking

• CloudFront - CDN for frontend (OAI-protected S3)
• API Gateway - REST API with Cognito authorizer
• WAF - Web Application Firewall (VPN CIDR restricted)
• VPC - Private Lambda subnets whitelisted on MongoDB Atlas

Data Storage

• MongoDB Atlas - Primary database (external)
• EFS - Elastic File System for Lambda file processing
• S3 - Frontend hosting + caching

Async Processing

• SQS FIFO Queues - Background job processing
• EventBridge - Scheduled tasks (cron)
• Step Functions - Workflow orchestration

Monitoring

• CloudWatch Logs - API and Lambda execution
• CloudWatch Alarms - Operational alerts
• Slack Integration - Pipeline notifications

Server Deployments

Development Workflows

Build & Deploy (Full)

npm run reinstall-full    # Full reinstall + lint + build
npm run build             # Build all layers
npm run test              # Run all tests
npm run synth             # Generate CloudFormation
npm run deploy            # Build and deploy to AWS

Local Frontend Development

npm run deploy-local-dev  # Gets aws_config.json from deployed stack
cd react-frontend/
npm start                 # Dev server on localhost:3000

Frontend connects to deployed backend - fast UI iteration without CDK deploy.

Adding a New API

1. Define contract in react-frontend/src/common-shared/api/definitions/{namespace}/{fn}.ts
2. Run npm run generate from root
3. Implement handler at cdk/api/.../endpoints/{namespace}/{fn}/handler.ts
4. Frontend uses: import { fn } from "@lib/api/gen/..."

Branch-Based Deployment

Each Git branch deploys as a separate CloudFormation stack:

• Push to feature-xyz → deploys BertAppStack-feature-xyz
• Separate URL, separate resources
• Clean up via cdk destroy when done

Configuration

AWS Secrets (Secrets Manager)

Frontend Config (aws_config.json)

Injected at deployment with stack outputs:

{
  "DEPLOY_STACK_ID": "...",
  "MAINTENANCE_MODE": false,
  "COGNITO_AUTH_BASE_URL": "...",
  "API_BASE_URL": "https://...execute-api.../prod/",
  "BRANCH_NAME": "...",
  "CLOUDFRONT_URL": "..."
}

Key Integrations

Related Documentation

• Dashboards Overview - Legacy system with migration plan
• Data Tools Overview - Legacy data processing
• Project Inventory - Complete P2R codebase inventory
• Tools and Systems - Overview of all P2R tools
• Path2Acquisition Flow - How BERT modules fit in the data flow

Important Notes

• Type Safety - Full end-to-end TypeScript from API definitions through Lambda implementations
• Branch Isolation - Each branch deploys as separate stack (clean up unused branches to control costs)
• MongoDB Atlas - Requires VPN/whitelisted IPs for Lambda access
• Memory Requirements - TypeScript compilation needs 4GB allocation
• Test Coverage - 90% threshold enforced
• Shiny → DRAX - Legacy Shiny Reports have been ported to DRAX reports

Source: /Users/mpelikan/Documents/code/p2r/bert (README.md, cdk/README.md, react-frontend/README.md, api-generation/README.md, .amazonq/rules/memory-bank/) Documentation created: 2026-01-24

Order Processing Overview

Legacy web application for managing Path2Response order workflows, Jira synchronization, and campaign fulfillment operations.

Status: Legacy System - Order App functionality has been migrated to BERT, with Shipment functionality migration in progress.

Purpose

The Order Processing system (internally called “Order App” or “Legacy”) automates and simplifies the tedious parts of order processing for Path2Response campaigns. It provides:

• A web-based interface for managing orders and models from Jira
• Automated job processing for P2A models, fulfillment, and shipment
• Jira synchronization to keep order data current
• A scheduler for batch processing and queue management
• Integration with BERT for transitioning functionality to the new platform

Primary Users:

• Data Science team (model runs, preselect configuration)
• Data Engineering team (fulfillment, shipment, job management)
• Operations team (order tracking, queue monitoring)

Architecture

Directory Structure

order-processing/
├── backend/                 # Sails.js API server
│   ├── api/
│   │   ├── controllers/     # API endpoints (44 controllers)
│   │   ├── models/          # MongoDB data models (37 models)
│   │   ├── services/        # Business logic services (50+ services)
│   │   ├── policies/        # Authentication/authorization
│   │   └── hooks/           # Sails lifecycle hooks
│   ├── config/              # Sails configuration
│   └── lib/                 # Utilities (logger, queue management)
├── frontend/                # AngularJS SPA
│   ├── src/app/
│   │   ├── orders/          # Main order management views
│   │   │   ├── preselect/   # Model configuration
│   │   │   ├── fulfillment/ # Fulfillment processing
│   │   │   ├── shipment/    # Shipment management
│   │   │   ├── scheduler/   # Job scheduling
│   │   │   └── queue/       # Queue monitoring
│   │   ├── core/            # Shared components
│   │   └── admin/           # Admin functions
│   └── config/              # Frontend configuration
├── edge-scripts/            # Server-side processing scripts
│   └── scripts/
│       ├── tasks/           # Job execution scripts
│       │   ├── p2a2/        # P2A2 model tasks
│       │   ├── p2a3xgb/     # XGBoost model tasks
│       │   ├── fia/         # Fulfillment Input Analysis
│       │   └── extract/     # Data extraction
│       ├── shipment/        # Shipment processing
│       └── util/            # Shared utilities
├── scripts/                 # Deployment and maintenance scripts
├── nginx.conf               # Production nginx configuration
└── pm2-process.json         # PM2 process management

Technology Stack

Key Dependencies

• javascript-shared - Internal shared library (error handling, utilities)
• sails-mongo - Custom MongoDB adapter (Atlas 6 compatible fork)
• jira-connector - Jira API integration
• winston - Logging with daily rotation

Core Functionality

Order Management

The system synchronizes with Jira to track orders and models:

• Orders - Campaign requests with client, title, dates, and fulfillment details
• Models - P2A model configurations (P2A2, P2A3/XGB, Hotline, DTC, etc.)
• Preselect - Model configuration and parameter setup
• Fulfillment - Processing and delivering audience files

Job Processing

The Order App manages several job types:

Feature Flags

The system uses feature flags to control functionality during BERT migration:

LEGACY_P2A3XGB_FULL_RUN_DISABLED      - Disable XGB runs in Legacy
LEGACY_SHIPMENT_RUN_DISABLED          - Disable shipment runs
LEGACY_FULFILLMENT_RUN_DISABLED       - Disable fulfillment runs
LEGACY_FEATURE_FLAG_DEBUG_MODE        - Enable debug logging

Feature flags are stored in MongoDB and keyed by:

• name - Flag identifier
• branchName - Git branch (staging, production, feature branch)
• deploymentEnvironment - DEV, STAGING, PRODUCTION

Jira Synchronization

The Order App maintains a local MongoDB cache of Jira issues:

• Background jobs sync data from Jira
• Order model extracts ~100+ fields from Jira custom fields
• Real-time updates via Jira webhooks
• Handles both PATH project orders and model subtasks

Integrations

BERT Integration

BERT (Base Environment for Re-tooled Technology) is the successor platform:

• BertIntegrationController - Receives commands from BERT
• serverlessToLegacyJobLauncher - Launches Legacy jobs from BERT
• Feature Flags - Control which functionality runs in Legacy vs BERT

Flow: BERT can trigger Legacy actions via the /bert/runActionForLegacy endpoint, allowing gradual migration of functionality.

Jira

• Custom authentication using Jira credentials
• Order/Model data synchronized from Jira
• Comments and attachments managed through Jira API
• Issue status transitions triggered from UI

Salesforce

• Order creation from Salesforce opportunities
• Middleware integration for data synchronization

Databricks

• Model execution on Databricks compute
• Configuration management for ML jobs

Slack

• Notifications for job completion and errors
• Alert integration for monitoring

Development

Prerequisites

# Required global packages
npm install -g bower gulp sails

# Node.js version
node --version  # Should be ~22 or ~24

Installation

# Clone and install
git clone git@bitbucket.org:path2response/order-processing.git
cd order-processing
npm install

# Install backend and frontend separately
cd backend && npm install
cd ../frontend && npm install && bower install

Configuration

1. Backend: Copy /backend/config/local_example.js to /backend/config/local.js
2. Frontend: Copy /frontend/config/config_example.json to /frontend/config/config.json

Running Locally

# Start backend
cd backend
sails lift
# API available at http://localhost:1337

# Start frontend (separate terminal)
cd frontend
gulp dist
gulp production
# UI available at http://localhost:3000

Production Deployment

The application runs on an EC2 instance with nginx:

Process management via Forever:

cd backend
npm run start          # Production
npm run start-staging  # Staging

Testing

cd backend
npm test  # Runs Mocha tests

Note: Tests are acknowledged as outdated and may not pass (see README TODO).

Data Models

Key Models

Order Model Fields

The Order model contains extensive fields including:

• Core Jira fields: id, key, fields, status
• Client/Title info: client, title, titleKey, titleInfo
• Dates: dueDate, mergeCutoff, mailDate, resultsDue
• Model configuration: modelType, buyerType, variableSources
• Fulfillment: fulfillmentFile1-5, fulfillmentComplete1-5
• Workflow state: workflowState, workflowStateKey

Related Documentation

• Path2Acquisition Flow - Overall data flow diagram
• BERT Overview - Successor platform documentation
• Development Process - Agile/Scrum process

Migration Status

The Order Processing system is being deprecated in favor of BERT. Current migration status:

Source: README.md, backend/README.md, frontend/README.md, TODO_PATH-25869.md, package.json, backend/package.json, Order.js, Featureflag.js, BertIntegrationController.js, nginx.conf

Documentation created: 2026-01-24

coop-scala Overview

Scala/Spark mono-repository for Path2Response’s core data processing infrastructure, including audience selection (preselect), householding, fulfillment, and analytics.

Purpose

coop-scala is the heart of P2R’s data processing. It contains all the Scala/Spark applications that run on AWS EMR (Elastic MapReduce) to:

• Build audiences (Preselect) - Score and select prospects from the cooperative data using machine learning models
• Process households (Households) - Match, deduplicate, and manage household data across the cooperative
• Fulfill orders (Fulfillment) - Process keycoding and list preparation for direct mail campaigns
• Generate statistics (Generate-Stats) - Create analytics and reports from customer transaction data
• Handle third-party data (Third-Party, 4Cite) - Process external data sources including BigDBM and browse behavior data
• Extract data (Extract) - Export addresses and identities for CASS/NCOA processing

Architecture

Repository Structure

coop-scala/                           # Root POM - Version 335.0.0-SNAPSHOT
├── scala-parent-pom/                 # Core libraries (non-Spark)
│   ├── common/                       # Shared utilities, collections
│   ├── cli/                          # Command-line interface utilities
│   ├── emr-core/                     # EMR launcher framework
│   │   └── emr-api/                  # EMR API, configuration, logging
│   ├── json/                         # JSON serialization
│   │   ├── json-definitions/         # Core JSON types
│   │   ├── json-definitions-4js/     # JSON for JavaScript compatibility
│   │   └── json-preselect/           # Preselect-specific JSON
│   ├── helpers/                      # Utility scripts (copy-mongo-to-s3)
│   ├── another-name-parser/          # Name parsing library
│   ├── modeling-support/             # ML/modeling utilities
│   ├── operations/                   # Ops automation tools
│   ├── spark-core/                   # Spark API wrappers
│   └── databricks-core/              # Databricks API (legacy)
│
├── spark-parent-pom/                 # Spark applications
│   ├── spark-common/                 # Shared Spark utilities
│   ├── spark-common-test/            # Spark testing framework
│   ├── spark-launcher/               # EMR job submission framework
│   │   ├── emr-launcher/             # Core EMR launcher
│   │   ├── emr-launcher-ext/         # Extended EMR functionality
│   │   ├── emr-revive/               # Retry/recovery (EmrReanimator)
│   │   └── spark-driver/             # Driver utilities
│   │
│   ├── preselect/                    # Audience selection (core logic)
│   │   ├── preselect-app/            # Main application
│   │   ├── preselect-core/           # Core algorithms
│   │   ├── preselect-classifiers/    # Household classification
│   │   ├── preselect-population/     # Population handling
│   │   ├── preselect-variables/      # Variable generation
│   │   ├── preselect-configuration/  # Configuration management
│   │   ├── preselect-callbacks/      # Monitoring hooks
│   │   ├── preselect-collections/    # Data structures
│   │   └── preselect-dates/          # Date handling
│   ├── preselect-emr/                # EMR deployment (main CLI)
│   ├── preselect-cli/                # Cloudera deployment (legacy)
│   ├── preselect-databricks/         # Databricks deployment (legacy)
│   │
│   ├── households/                   # Household processing (core)
│   │   ├── households-capture/       # Data capture processing
│   │   ├── households-common/        # Shared utilities
│   │   ├── households-reports/       # Reporting
│   │   └── households-stages-emr/    # EMR stages
│   ├── households-emr/               # EMR deployment
│   │
│   ├── fulfillment/                  # Order fulfillment (core)
│   ├── fulfillment-emr/              # EMR deployment
│   │
│   ├── generate-stats/               # Statistics generation
│   │   ├── generate-stats-core/      # Core logic
│   │   ├── generate-stats-emr/       # EMR deployment
│   │   └── docs/                     # Documentation
│   │
│   ├── 4cite/                        # Browse data (4Cite partner)
│   ├── 4cite-emr/                    # EMR deployment
│   │
│   ├── third-party/                  # Third-party integrations
│   ├── third-party-emr/              # EMR deployment (BigDBM, etc.)
│   │
│   ├── extract-data/                 # Data extraction core
│   ├── extract-emr/                  # EMR deployment
│   ├── extract-ncoa/                 # CASS/NCOA extraction
│   │
│   ├── convert-emr/                  # Data conversion
│   ├── digital-audience/             # Digital audience building
│   ├── digital-audience-emr/         # EMR deployment
│   ├── reseller-append/              # Reseller data processing
│   ├── reseller-emr/                 # EMR deployment
│   │
│   ├── databricks/                   # Databricks integration (legacy)
│   ├── coop-spark-automation/        # Automated Spark jobs
│   ├── integration-tests/            # Integration test suite
│   └── ops/                          # Operational automation
│
├── docu/                             # Legacy documentation
├── preselect-book/                   # Preselect mdbook docs
└── examples/                         # Example code

Technology Stack

Build Tools:

• Maven with custom parent POMs
• Scalastyle for code quality
• Scalariform for code formatting
• Maven Shade Plugin for fat JARs (EMR deployment)

EMR Launcher Architecture

The EMR launcher system provides a sophisticated framework for submitting and monitoring Spark jobs:

Local Machine                          S3                          EMR Cluster
─────────────                          ──                          ───────────
Script (.sc)
    │
Launcher4Emr
    │ upload files
    ├─────────────────────────────→ input files
    │ serialize options
    ├─────────────────────────────→ options.json
    │
EmrSimpleLauncher
    │
EmrLauncher
    │ upload JAR
    ├─────────────────────────────→ job JAR
    │ submit cluster
    ├──────────────────────────────────────────→ Cluster Created
    │                                                    │
EmrProvider.waitForJob()                                │
    │ poll status (30s)                                 │
    ├────────────────────────────────────────→ get status
    │ write log                                         │
    ├─────────────────────────────→ job log             │
    │                                                    │ download JAR
    │                                                Driver4Emr runs
    │                                                    │
    │                                                Execute Spark Job
    │                                                    │
    │                              results ←────────────┤
    │ detect completion                                 │
    ├────────────────────────────────────────→ Cluster Terminated
    │
onAfterJobComplete()
    │ download results

Key Components:

• Script Layer (*.sc) - Entry point bash/scala scripts
• Launcher Layer (*Launcher4Emr) - Parse options, upload files, submit jobs
• Simple Launcher (EmrSimpleLauncher) - Configure cluster specs
• EMR Launcher Core (EmrLauncher) - Builder pattern for configuration
• EMR Provider (EmrProvider) - Polling, monitoring, status logging
• Driver Layer (*Driver4Emr) - Executes on EMR cluster

Spark Jobs

Preselect (Audience Selection)

Purpose: Score and select prospects from cooperative data for direct mail campaigns.

Key Scripts:

Drivers:

• PreselectDriver4Emr - Main preselect execution
• VariablesDriver - Variable computation
• EmailMatchDriver4Emr - Email matching
• PDPDriver4Emr - Data protection
• PickBrowseDriver4Emr - Browse data selection
• SelectAttributesDriver4Emr - Attribute selection
• TitleTransactionCountsDriver4Emr - Transaction counts
• DumpTransactionsDriver4Emr - Transaction export

Buyer Types (Responder Logic):

Training Data Process:

1. Step 1: Generate large sample with binary variables (1M prospects, 100K responders)
2. Step 2: Select optimal households (Python/external)
3. Step 3: Extract full training data for optimal households using training-ids/ folder

Households

Purpose: Match, deduplicate, and manage household data across the cooperative.

Modules:

• households-capture/ - Data capture processing
• households-common/ - Shared utilities
• households-reports/ - Reporting
• households-stages-emr/ - EMR stage processing

Integration:

• Households processing runs automatically
• Triggers generate-stats workflow upon completion
• Data stored in dated folders: extract/YYYY-MM-DD/clientData

Fulfillment

Purpose: Process keycoding and list preparation for direct mail campaigns.

Key Drivers:

• FulfillmentInputAnalysisDriver4Emr - Analyze input data, build match files
• KeycodingDriver4Emr - Main fulfillment/keycoding processing

Matching Types:

Workflow:

1. JIRA Attachment - State/zipcode files attached to ORDER tickets
2. Match File Processing - Convert to importId format, upload to S3
3. Fulfillment Input Analysis - Pre-compute matches, output “line-by-line” files
4. Keycoding - Filter based on pre-computed matches

Generate-Stats

Purpose: Create customer statistics and reports from transaction data.

Tools:

Production Integration:

• Runs automatically after households processing completes
• Uses large, memory-optimized EMR clusters
• Standard statistics enabled by default (titleStats, transRecency, repeatPurchaseHistory, etc.)
• Results stored in MongoDB

4Cite (Browse Data)

Purpose: Process second-party browse/website behavior data from 4Cite partner.

Key Scripts:

• extractmd5-emr.sc / extractmd5-databricks.sc - MD5 extraction
• Monthly browse reporting

Integration:

• allowSecondParty flag controls 4Cite data inclusion in preselect
• --override-4cite-exclusion parameter to include blocked data

Third-Party (BigDBM)

Purpose: Process external data enrichment from BigDBM partner.

Drivers:

• ConvertBigDBMFullDriver4Emr - Convert BigDBM data
• ConvertBigDBMNameAddressDriver4Emr - Name/address conversion
• ExtractBigDBMAddressIdentityDriver4Emr - Extract identities
• GatherBigDBMFullDriver4Emr - Gather BigDBM data
• VerifyBigDBMFullDriver4Emr - Verify data integrity

Extract (CASS/NCOA)

Purpose: Export addresses and identities for CASS (Coding Accuracy Support System) and NCOA (National Change of Address) processing.

Drivers:

• ExtractNCOAAddressIdentityDriver4Emr - Extract for NCOA
• ProcessNcoaResultsDriver4Emr - Process NCOA results
• ReportNcoaResultsDriver4Emr - Report on NCOA
• MatchDataIdentityDriver4Emr - Match identities
• MatchPrisonIdentityDriver4Emr - Prison database matching

Core Libraries

emr-core (EMR Framework)

Location: scala-parent-pom/emr-core/emr-api/

Core EMR launcher infrastructure:

• EmrLauncher - Builder pattern for job configuration
• EmrProvider - Cluster monitoring and status
• EmrBehaviors - Launch/monitoring logic
• EmrAccessors - Configuration defaults
• EmrExclusionConfig - Instance/zone exclusion management
• EmrReanimator - Retry/recovery strategies

Retry Strategies:

• OPTIMISTIC - Cheapest, prefers ARM/spot
• CAREFUL - Balanced approach
• PESSIMISTIC - Faster failover to x86
• PARANOID - Quickest on-demand failover

common

Location: scala-parent-pom/common/

Shared utilities and collections:

• HhIdSet / LongSet - Memory-efficient household ID storage (~8 bytes per ID)
• General utilities for all modules

json

Location: scala-parent-pom/json/

JSON serialization:

• json-definitions - Core JSON types
• json-definitions-4js - JavaScript compatibility
• json-preselect - Preselect-specific types

spark-core

Location: scala-parent-pom/spark-core/spark-api/

Spark API wrappers and utilities.

another-name-parser

Location: scala-parent-pom/another-name-parser/

Name parsing library for identity matching.

helpers

Location: scala-parent-pom/helpers/

Utility scripts:

• copy-mongo-to-s3 - Sync MongoDB collections to S3 as JSON lines

Data Flow

Overall Data Pipeline

┌──────────────────┐
│   Raw Data       │  (Transaction data from cooperative members)
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   Households     │  (Match, dedupe, manage household data)
└────────┬─────────┘
         │
         ├─────────────────────────────────────────────┐
         │                                             │
         ▼                                             ▼
┌──────────────────┐                       ┌──────────────────┐
│  Generate-Stats  │ (Automatic)           │   Preselect      │ (On-demand)
│  (Analytics)     │                       │   (Audience)     │
└──────────────────┘                       └────────┬─────────┘
                                                    │
                                                    ▼
                                           ┌──────────────────┐
                                           │   Fulfillment    │
                                           │   (Keycoding)    │
                                           └────────┬─────────┘
                                                    │
                                                    ▼
                                           ┌──────────────────┐
                                           │   Extract        │
                                           │   (CASS/NCOA)    │
                                           └──────────────────┘

Preselect Data Flow

1. Input: Preselect configuration (JSON) + households data
2. Classification: Identify responders vs. prospects
3. Variable Generation: Compute features for scoring
4. Model Scoring: Apply ML models
5. Selection: Choose top prospects
6. Output: Selected audience files

Fulfillment Data Flow

1. Input: fulfillment_input2.tsv + match files (states, zips)
2. Match File Processing: Convert to importId format
3. Fulfillment Input Analysis: Pre-compute all matches
4. Output: Line-by-line files with matchingKeys
5. Keycoding: Filter based on match keys

Development

Building

# Full build with tests
mvn clean install

# Skip tests
mvn clean install -DskipTests

# Build specific module
mvn clean install -pl spark-parent-pom/preselect-emr -am

Testing

# Run all tests
mvn test

# Run specific test
mvn test -pl spark-parent-pom/preselect-emr -Dtest=MyTestClass

Code Quality

# Check style
mvn scalastyle:check

# Format code
./format.sh

Deployment

Each EMR module has a build.sh script that creates deployment packages:

# Build EMR deployment
cd spark-parent-pom/preselect-emr
./build.sh

# Install all EMR modules
./install-all.sh

Cluster Sizing

Default cluster sizes (compute nodes):

Sizing Theory:

• Partition count / 8 executors per node = ideal compute nodes
• Add 5-10% fudge factor for EMR partition variance
• Full households file: 2,880 partitions / 8 = 360 executors = 90 nodes + 5% = 95 nodes

Environment Variables

EMR Exclusions

Centralized instance exclusion via S3:

# Add exclusion
emr-exclusions --command add --zone us-east-1a --instance-type m7g.2xlarge --reason "Capacity issues"

# Remove exclusion
emr-exclusions --command remove --zone us-east-1a --instance-type m7g.2xlarge

# List all exclusions
emr-exclusions --command list

Monitoring Scripts

# Watch job progress
emr-watch.sc --cluster-id j-XXXXXXXXXXXXX

# Watch state transitions
emr-watch-states.sc --cluster-id j-XXXXXXXXXXXXX

# View EMR configuration
emr-config

Job Status Logs

Location: s3://bucket/job/[dev/]YYYY/MM/DD/{clusterId}.json

Updated every 30 seconds with:

• Cluster ID and state
• Step ID and state
• Worker counts and fleet type
• Timestamps
• Error information
• Configuration details

Related Documentation

Key Integration Points

BERT (Base Environment for Re-tooled Technology)

BERT orchestrates coop-scala jobs for:

• Campaign processing workflows
• Automated statistics generation
• Fulfillment operations

Order App

Triggers fulfillment jobs from campaign orders.

Data Tools

Coordinates data processing and extract operations.

MongoDB

• Statistics results storage
• Configuration sync via copy-mongo-to-s3

Source: README.md, .amazonq/docs/, spark-parent-pom//README.md, spark-parent-pom/preselect-emr/docs/, spark-parent-pom/generate-stats/docs/, spark-parent-pom/fulfillment-emr/FULFILLMENT_MATCHING_GUIDE.md, docu/, preselect-book/*

Documentation created: 2026-01-24

Dashboards Application Overview

Legacy System Notice: Dashboards is a standalone Node.js application that has not yet been incorporated into BERT. A serverless migration plan exists to retire this application by splitting it into a CDK-based Audit Repository and migrating all user-facing functionality to BERT.

What is Dashboards?

The Dashboards application provides every person at Path2Response with an individualized and actionable view into the metrics, opportunities, tasks, and data that need their attention. It serves as the central hub for business intelligence, financial operations, and data synchronization across the company.

Purpose

Dashboards serves three primary functions:

1. Business Intelligence - 30+ specialized reports covering financial, operational, development, and client management aspects
2. Data Synchronization - The Audit system synchronizes data between Salesforce, QuickBooks, MongoDB, Slack, and Jira
3. Financial Operations - Invoice generation, payment recording, broker commission processing, and revenue forecasting

Current Architecture

Technology Stack

Project Structure

The application consists of two main components:

/dashboards/
├── app.js                    # Main Express server
├── /dashboards/              # Dashboard modules (account, finance, engineering)
├── /routes/                  # Express route handlers
├── /views/                   # EJS templates
├── /public/                  # Static assets (CSS, client-side JS)
├── /users/                   # User data storage (JSON files)
├── /sessions/                # Session file storage
└── /reports/                 # Audit and reporting system
    ├── audit.js              # Main audit script
    ├── /audit/               # Audit data output (JSON files)
    ├── /cache/               # Report cache
    └── /lib/                 # Core business logic
        ├── /audit/           # Data synchronization engine
        ├── /forecast/        # Revenue prediction engine
        ├── /reports/         # 30+ report modules
        └── /common/          # Shared utilities

Core Systems

1. Audit System

The Audit System is the core data synchronization engine. It fetches data from multiple external sources, assembles and processes it, performs analysis, and saves structured data that powers all reports and dashboards.

Six-Stage Pipeline

Load → Fetch → Assemble → Process → Analyze → Save

1. Load - Load existing Google users/groups and configuration
2. Fetch - Pull raw data from external sources
3. Assemble - Combine and structure raw data
4. Process - Transform and enrich assembled data
5. Analyze - Identify issues, generate findings, calculate metrics
6. Save - Persist processed data to JSON files

Data Sources

Running Audits

# Incremental audit (only changed data since last run)
./audit.js -d

# Complete audit (fetch all data from scratch)
./audit.js -c -d

Standard Audit Flow (Production)

run_audit.sh
├── audit.js -d
├── clear_cache.sh
└── [Production Only]
    ├── generate_invoices.sh
    ├── record_payments.sh
    ├── record_mailfiles.sh
    ├── update_salesforce.sh
    └── sync_to_s3.sh

2. Reports System

The Reports System generates comprehensive business intelligence reports, transforming audit data into actionable insights. It consists of 30+ specialized reports.

Report Categories

Financial Reports

Client Management Reports

Operations Reports

Development Reports (Jira Integration)

Report Architecture

Reports follow a standard pattern:

1. Define audit data requirements (audit_parts)
2. Load audit data via loadAudit
3. Process data according to business rules
4. Generate HTML output via HtmlBuilder
5. Return HTML string or JSON data

Reports are cached in /reports/cache/ and invalidated when new audits complete.

3. Forecast System

The Forecast System is a sophisticated revenue prediction engine that analyzes historical ordering patterns to forecast future revenue up to 3 years out.

“The best predictor of future behavior is past behavior.”

Four-Stage Pipeline

Audit Data → Analyze → Predict → Forecast → Google Sheets

1. Analyze - Consolidate orders monthly, detect ordering frequency
2. Predict - Detect trends and statistical patterns
3. Forecast - Generate revenue projections
4. Google - Publish results to Google Sheets

Account Classifications

By Maturity:

• Mature Titles - Ordering with P2R for ~1 year or more (high confidence)
• Young Titles - New to P2R, limited history (uses vertical averages)
• Stubbed Titles - Placeholder accounts for projected BizDev pipeline

By Vertical: Catalog, DTC, Nonprofit, Non Participating, PreScreen, Publishing

By Classification: Class A (largest), Class B, Class C, Class D (smallest)

Key Features

• Custom year definition (365 days from first order, not calendar)
• Monthly order consolidation for consistent analysis
• CV (credit mailer) filtering from revenue trends
• Forecast locking after 3rd of each month
• New client acquisition modeling from BizDev pipeline

Business Process Scripts

Core Operations

Monthly Processes

Broker Commissions - Critical monthly finance process:

1. Finance validates broker invoices and credit memos
2. Engineering runs generateBrokerCommissions.js -m YYYY-MM
3. System creates Journal Entries, Bills, and Vendor Credits in QuickBooks
4. Finance reconciles and closes books

# Run for previous month (e.g., if current month is July 2026)
./generateBrokerCommissions.js -m 2026-06

Revenue Forecasting

# Generate algorithmic revenue forecast
./forecast_revenue.sh

Output written to /reports/cache/revenueForecast/ and published to Google Sheets.

Integration Points

Data Pipeline

MongoDB/Salesforce/QuickBooks → Audit → Cache → Reports → Dashboards

Authentication Flow

Google OAuth → Passport → Session → User JSON files

External URLs

Environment Configuration

Required Environment Variables

# Salesforce
SALESFORCE_ACCOUNT="account"
SALESFORCE_PASSWORD="password"
SALESFORCE_TOKEN="token"

# Slack
SLACK_TOKEN="token"

# MongoDB
MONGO_ORDERPROCESSING="connection_string"
MONGO_OPERATIONS="connection_string"
MONGO_STATISTICS="connection_string"

# AWS
P2R_BUCKET_URI="s3://bucket-name"
P2R_RUNTIME_ENVIRONMENT="production"  # Controls post-audit operations

Serverless Migration Plan

Executive Summary

The current Dashboards repository will be retired and split into two projects:

1. New Audit Repository (CDK Serverless) - Core data processing, audit, forecast, and post-audit tools
2. BERT Integration - All user-facing functionality including routes, views, and report generation

Migration Phases

Phase 1: Create Audit Repository (CDK)

• Extract /reports/lib/audit/ → Lambda functions
• Extract /reports/lib/forecast/ → Step Functions workflow
• Migrate post-audit scripts → EventBridge + Lambda
• Store data in S3 instead of local JSON files

Phase 2: Migrate User-Facing to BERT

• Convert Express routes → BERT React Router
• Convert EJS templates → React components
• Convert report modules → BERT report components
• Migrate authentication to BERT’s auth system

Phase 3: Integration and Testing

• Connect BERT to Audit Repository APIs
• Parallel operation with gradual user migration
• Performance validation

Phase 4: Retirement

• Complete migration of all users
• Decommission Dashboards repository

Target Architecture

┌─────────────────────────────────────────────────────────────┐
│                    BERT Application                         │
│              (All Dashboard/Report UI)                      │
│  • Routes (migrated from /routes)                           │
│  • Components (migrated from /views + /reports/lib/reports) │
│  • Dashboards (migrated from /dashboards)                   │
└──────────────────────┬──────────────────────────────────────┘
                       │ API Calls
                       ↓
                ┌──────────────────┐
                │ Audit Repository │
                │   (CDK Project)  │
                │    API Gateway   │
                └──────┬───────────┘
                       │
        ┌──────────────┼────────────┐
        ↓              ↓            ↓
   ┌──────────┐ ┌──────────┐ ┌──────────┐
   │  Lambda  │ │  Lambda  │ │   Step   │
   │ (Audit)  │ │(Forecast)│ │Functions │
   └─────┬────┘ └─────┬────┘ └─────┬────┘
         │            │            │
         └────────────┼────────────┘
                      ↓
        ┌────────────────────────────┐
        │        Data Storage        │
        │  S3 Buckets | EventBridge  │
        │  (Audit/Forecast/Cache)    │
        └────────────────────────────┘

AWS Services in Target Architecture

Key Benefits

1. Scalability - Automatic scaling based on demand
2. Availability - Multi-region deployment options
3. Maintenance - Reduced server management overhead
4. Cost - Pay-per-use model vs. always-on EC2

Risk Mitigations

Related Documentation

• Data Tools Overview - Another legacy system also targeted for BERT migration
• BERT Overview - Target platform for consolidated applications
• Production Team Overview - Production operations that use Dashboards
• Data Acquisition Overview - Data operations that trigger audits

Important Notes

• Audit freshness is critical - Most operations require recent audit data
• Never run multiple audits simultaneously
• Production safety - Many modules include testing flags to prevent accidental modifications
• Network access - MongoDB connections require VPN/developer network access
• Broker commissions - Requires Finance department coordination before running
• Data dependencies - Reports depend on audit system; forecast depends on audit

Source: Dashboards repository documentation (README.md, reports/README.md, lib/README.md, lib/audit/README.md, lib/forecast/README.md, lib/reports/README.md) Documentation created: 2026-01-24

Data Tools Services Overview

Legacy System Notice: Data Tools is a legacy application targeted for migration into BERT (Base Environment for Re-tooled Technology) in 2026. This documentation captures the current state for reference during the migration planning process.

What is Data Tools?

Data Tools is a collection of automated services that handle data processing workflows at Path2Response. These services manage file ingestion, conversion, validation, and various operational tasks that support the core data cooperative business.

Architecture

Data Tools consists of three categories of services:

1. Embedded Services - Run within the main app.js application
2. Micro Services - Standalone services for specific processing tasks
3. Deprecated Services - Legacy functionality no longer in active use

1. Embedded Services (app.js)

These services are embedded directly in the main Data Tools application:

2. Micro Services

Standalone services that handle specific processing workflows:

3. Deprecated Services

Services no longer in active use:

Key Integrations

Data Tools integrates with several internal and external systems:

• SmartFile - SFTP server for client file uploads
• MongoDB - Document storage for processing state
• Slack - Notifications (#sitetagging, #orders-data channels)
• Zendesk - Ticket management and titlekey organization
• Salesforce - Title synchronization via Dashboards
• OneTrust - Data Subject Request origination

Data Flow

Client Data (SmartFile)
        ↓
fileOperationsMaster (scan routes)
        ↓
fileOperationsChild (extract/clean)
        ↓
importRunner/convert (process)
        ↓
autoValidate (review)
        ↓
cassRunner (CASS append)
        ↓
Ready for downstream processing

Migration Considerations

When migrating to BERT, consider:

1. Service consolidation - Many embedded services may be candidates for consolidation
2. Non-functional services - oldAccount is disabled and may not need migration
3. Deprecated services - sameDayOmitRunner should be evaluated for retirement
4. Integration points - SmartFile, Zendesk, Salesforce integrations need BERT equivalents
5. Notification patterns - Slack channel notifications should map to BERT alerting

Source: Data Tools Services (Confluence) Last updated from source: 2019-07-25 Documentation created: 2026-01-24

Response Analysis (ARA) Overview

Automated batch processing system that measures direct mail campaign effectiveness by correlating promotional mailings with customer transaction data from the data cooperative.

LEGACY NOTICE: This system is a candidate for BERT migration. The current implementation uses Deno/TypeScript for orchestration with a Python processing engine (auto_ra). Future consolidation into the BERT platform is anticipated.

Purpose

Response Analysis (internally called “ARA” or simply “RA”) is Path2Response’s automated system for measuring the effectiveness of direct mail marketing campaigns. It answers the fundamental question: “Did the people we mailed to make purchases?”

The system:

• Processes thousands of promotions daily
• Correlates mailing records with post-mail transaction data
• Calculates response rates and campaign performance metrics
• Provides ongoing analysis until measurement windows close
• Generates reports consumed by the Dashboards application

Architecture

Directory Structure

cdk-backend/projects/response-analysis/
├── bin/                          # CLI entry points
│   └── ra                        # Main executable
├── docs/                         # mdbook documentation
│   └── src/
│       ├── architecture.md
│       ├── data-sources.md
│       ├── glossary.md
│       └── usage.md
├── lib/response-analysis/        # Source code
│   ├── response-analysis.main.ts # CLI entry point and orchestration
│   ├── run-response-analysis.ts  # Core processing runner
│   ├── assemble-campaigns.ts     # Campaign/promotion assembly logic
│   ├── title-transactions.ts     # Transaction data handling
│   ├── manifest-writer.ts        # Output metadata generation
│   ├── memo.ts                   # Households memo integration
│   ├── utils.ts                  # Utility functions
│   ├── audit-query.main.ts       # Audit Query (aq) CLI tool
│   └── types/
│       └── account.ts            # TypeScript type definitions
├── README.md                     # Primary documentation
├── compile.sh                    # Linting/formatting script
└── update.sh                     # Dependency update script

Technology Stack

System Components

                    ┌─────────────────────────────────────────────────────────┐
                    │                    Data Sources                         │
                    │                                                         │
                    │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
                    │  │ Dashboards  │  │ Households  │  │ Transaction │     │
                    │  │ Audit Data  │  │    Data     │  │    Data     │     │
                    │  │    (S3)     │  │    (S3)     │  │  (S3/EMR)   │     │
                    │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘     │
                    └─────────┼────────────────┼────────────────┼────────────┘
                              │                │                │
                              └────────────────┼────────────────┘
                                               │
                    ┌──────────────────────────▼──────────────────────────────┐
                    │           Response Analysis Controller (ra)             │
                    │                                                         │
                    │  • Orchestrates workflow        • Manages job queue     │
                    │  • Coordinates data sources     • Handles errors        │
                    │  • Generates reports            • Slack notifications   │
                    └──────────────────────────┬──────────────────────────────┘
                                               │
                    ┌──────────────────────────▼──────────────────────────────┐
                    │              auto_ra Processing (Python)                │
                    │                                                         │
                    │  • Statistical analysis         • Response calculation  │
                    │  • Polars data processing       • Result generation     │
                    │  • Serial execution (~3s/promotion)                     │
                    └──────────────────────────┬──────────────────────────────┘
                                               │
                    ┌──────────────────────────▼──────────────────────────────┐
                    │                 Results Storage (EFS)                   │
                    │                                                         │
                    │  /mnt/data/prod/<title>/ra/                             │
                    │    ├── campaigns/<campaign_id>/                         │
                    │    │     ├── .account.json.gz    (metadata)             │
                    │    │     ├── .log.txt.gz         (execution log)        │
                    │    │     └── .config-*.json      (job config)           │
                    │    └── promotions/<order_key>/                          │
                    └─────────────────────────────────────────────────────────┘

Core Functionality

Processing Workflow

1. Data Discovery

   • Locate current households file on S3
   • Download households metadata (title keys, transaction availability dates)
   • Fast-scan Dashboards audit for account files

2. Campaign Assembly

   • Parse Account documents to find campaigns and promotions
   • Filter by date ranges (mailed within last year, data available after mail date)
   • Group promotions into campaigns or mark as standalone
   • Determine which jobs need to run (incomplete, not yet finalized)

3. Transaction Preparation

   • Check S3 cache for title transaction data
   • If missing, trigger EMR job (title-transaction-counts) - adds 20+ minutes
   • Download and decompress transaction files

4. Response Analysis Execution

   • For each runnable campaign/promotion:
     • Generate configuration JSON (pythonic snake_case naming)
     • Call auto_ra Python script with configuration
     • Capture stdout/stderr and job status
   • For campaigns with multiple promotions:
     • Process each promotion individually
     • Run combination step to aggregate results

5. Results Storage

   • Write analysis results to EFS (/mnt/data/prod/<title>/ra/)
   • Generate metadata files (.account.json.gz, .log.txt.gz)
   • Prune old versions (retain 31 days)

Processing Logic

Key Terms

CLI Usage

# Standard run
ra

# Preview without executing
ra --dry-run

# Filter by title
ra --title burrow --title cosabella

# Filter by order
ra --order 50984

# Rerun completed analyses
ra --assume-incomplete

# Verbose logging
ra -v

# Custom slack channel
ra --slack-channel @jsmith

# Disable slack
ra --no-slacking

Performance Characteristics

Integrations

Upstream Systems

Downstream Systems

AWS Services

Data Model

Account Document Structure

Account {
  AccountId: string           // "001Du000003bxG7IAI"
  Name: string               // "Ada Health"
  TitleKey: string           // "adahealth"
  Vertical: "Catalog" | "Nonprofit" | "DTC" | ...
  Campaigns: {
    [campaignId]: Campaign {
      Name: string
      StartDate, EndDate: string
      Opportunities: Opportunity[]
      RaJob: { Status, Config, Log, Error }
    }
  }
  Opportunities: {
    [opportunityKey]: Opportunity {
      Name: string
      MailDate, MeasurementDate: string
      OrderKey: string
      Order: { Fulfillments, Shipments }
      RaJob: { Status, Config, Log, Error }
    }
  }
  RaJob: { Complete: boolean, Success: boolean }
}

Output Files

Development

Prerequisites

1. auto_ra Python script installed

   • From ds-modeling package
   • Symlinked to ~/.local/bin/auto_ra

2. title-transaction-counts EMR utility

   • Included in preselect-emr installer

3. AWS CLI v2 (not Python venv version 1)

4. Access permissions

   • S3: Read/write to households data
   • EFS: Read/write to /mnt/data/prod/*
   • Dashboards audit: Read access

Installation

# Standard installation
sudo cp ./bin/* /usr/local/bin/
sudo rsync -avL --delete ./lib/ra/ /usr/local/lib/ra/

# User installation
cp ./bin/* ~/.local/bin/
rsync -avL --delete ./lib/ra/ ~/.local/lib/ra/

# Verify
ra --help

Maintenance

# Format, lint, and check
./compile.sh

# Update dependencies
./update.sh

Environment Variables

Related Documentation

• Source repository: /Users/mpelikan/Documents/code/p2r/cdk-backend/projects/response-analysis/
• Full mdbook docs: projects/response-analysis/docs/
• auto_ra (Python): ds-modeling repository
• title-transaction-counts: preselect-emr installer
• Dashboards: Consumes RA results for client reporting

Audit Query Tool (aq)

The Response Analysis project also includes a utility for querying Dashboards audit data:

# Get suppression import-ids for PDDM/Swift hotline runs
aq pddm-suppress birkenstock

Returns import-ids of shipments that need suppression based on SalesForce-configured suppression periods.

Source files:

• /Users/mpelikan/Documents/code/p2r/cdk-backend/projects/response-analysis/README.md
• /Users/mpelikan/Documents/code/p2r/cdk-backend/projects/response-analysis/docs/src/*.md
• /Users/mpelikan/Documents/code/p2r/cdk-backend/projects/response-analysis/lib/response-analysis/*.ts

Documentation created: 2026-01-24

Operations Overview

Automated system maintenance, health monitoring, and data synchronization tools for Path2Response back-office infrastructure.

Purpose

The Operations repository provides day-to-day operational tooling for the Infrastructure team. It performs automated system checks, data synchronization, and maintenance across staging, release-candidate (RC), and production environments. These tools run continuously via cron jobs on cc.path2response.com, the command-and-control server.

Primary Users: Infrastructure Team (Jason Smith, Wes Hofmann)

Key Responsibilities:

• System health monitoring at multiple intervals (10-minute, hourly, daily, weekly, monthly)
• Data synchronization from production to staging/development environments
• AWS resource management (EC2 instances, S3 storage, EFS volumes)
• Automated cleanup of temporary files and old data
• RC environment management (start/stop servers, pause/resume MongoDB Atlas)

Architecture

Directory Structure

operations/
├── bin/                    # Bash cron wrapper scripts
├── book/                   # Operations Reference Manual (mdbook)
│   └── src/               # Documentation source files
├── deno/                   # Deno utilities
│   ├── app/               # Deno applications (audit, costs, network)
│   ├── book-of-ops/       # AWS infrastructure documentation
│   ├── common/            # Shared Deno modules
│   ├── lib/               # Deno libraries (up, gateway)
│   └── utilities/         # Deno utility scripts
├── src/                    # TypeScript source code
│   ├── checks/            # Health check definitions
│   ├── rcman/             # RC management modules
│   ├── sync/              # Data synchronization modules
│   └── util/              # Shared utilities (AWS, functional, MongoDB)
├── tests/                  # Unit tests
├── crontab.txt            # Reference copy of production crontab
├── deno.jsonc             # Deno configuration
├── package.json           # Node.js dependencies
└── tsconfig.json          # TypeScript configuration

Technology Stack

Key Dependencies

{
  "@aws-sdk/client-ec2": "3.958.0",
  "@aws-sdk/client-s3": "3.958.0",
  "mongodb": "7.0.0",
  "node-slack": "0.0.7",
  "commander": "14.0.2"
}

Core Functionality

Utilities (Command-Line Tools)

Health Checks (system-health)

The system-health utility runs checks at different intervals:

EC2 Instance Management:

• Instances tagged with p2r=permanent are exempt from stale checks
• Instances can have custom timeouts (e.g., p2r=1.12:00 for 1 day 12 hours)
• Auto-kill option: p2r=1:00,kill terminates instance after timeout

Data Synchronization (sync-monkey)

Synchronizes data from production to staging/development:

Critical Note - MongoDB Filtered Sync: The operations.datafile collection sync uses a filter ({done: true, pid: {$exists: true}}) to prevent race conditions where staging processes could intercept production work queue items. See PATH-25842 for full context.

RC Environment Management (rc-man)

Manages Release Candidate infrastructure for cost optimization:

Warning: Paused MongoDB Atlas clusters auto-resume after 30 days.

Cleanup Operations (clean, archive)

Integrations

AWS Services

Internal Systems

External Services

Development

Prerequisites

• Node.js (project-local installation)
• Deno (for newer utilities)
• AWS CLI (apt install awscli)
• Rust + mdbook (for documentation)
• graphviz (for mdbook diagrams)

Build Commands

Deployment

1. Make changes in TypeScript source
2. Run npm run build to compile
3. Run npm install -g to install globally (live deployment)
4. Changes take effect immediately, including for cron jobs

Testing

# Run unit tests
npm run test

# Test specific utility without Slack
node ./dist/system-health.js --short --no-slacking

# Test RC management (dry run)
./dist/rc-man.js servers --start --no-slacking

Documentation (mdbook)

# Install dependencies
sudo apt install graphviz
cargo install mdbook-graphviz

# Serve documentation
cd book && mdbook serve -p 8089

The documentation auto-refreshes on save.

Cron Schedule Reference

Environment Configuration

All authentication uses environment variables or IAM roles:

Note: Command-line utilities require the secrets environment on cc.path2response.com.

Related Documentation

• Operations Reference Manual - book/src/SUMMARY.md
• [AWS SDK v3 Migration](operations repo: AWS_SDK_V3_MIGRATION.md) - Completed migration notes
• [Deno Book of Ops](operations repo: deno/book-of-ops/) - AWS infrastructure documentation
• [S3 Replication](operations repo: book/src/s3-replication.md) - Data sync topology
• [MongoDB Filtered Sync](operations repo: book/src/sync-monkey.md) - PATH-25842 race condition fix

Key Jira References

Source: README.md, book/src/SUMMARY.md, book/src/intro.md, book/src/utilities.md, book/src/system-health.md, book/src/sync-monkey.md, book/src/clean.md, book/src/archive.md, book/src/rc-man.md, book/src/s3-replication.md, book/src/aws-s3-audit.md, book/src/aws-efs-audit.md, package.json, crontab.txt, deno/README.md, AWS_SDK_V3_MIGRATION.md

Documentation created: 2026-01-24

CDK Backend Overview

AWS CDK infrastructure for Path2Response’s Step Functions workflows and batch processing systems.

Purpose

The cdk-backend repository defines and deploys AWS infrastructure for Path2Response’s data processing pipelines. It manages:

• Step Functions Workflows - Orchestrated multi-step data processing pipelines
• AWS Batch Compute - Scalable compute resources for heavy processing jobs
• Lambda Functions - Lightweight serverless functions for order processing
• Docker Images - Container definitions for batch and Lambda workloads
• EFS Integration - Shared file system access for order processing data

This infrastructure supports the core Path2Acquisition product by enabling automated audience creation, model training, and data file generation workflows.

Architecture

Directory Structure

cdk-backend/
├── cdk-stepfunctions/       # CDK stack definitions for Step Functions
│   ├── bin/main.ts          # Main CDK app entry point
│   └── lib/
│       ├── sfn-stacks/      # Step Function workflow stacks (17 workflows)
│       └── util/            # Shared utilities for CDK constructs
├── step-scripts/            # Deno/TypeScript step implementations
│   └── src/bin.step/        # Step function step scripts
├── projects/                # Standalone utility projects
│   ├── athanor/             # Workflow runner for licensed files
│   ├── response-analysis/   # Automated response analysis (RA)
│   ├── digital-audience/    # Digital audience processing
│   ├── sumcats/             # Category summaries processing
│   └── experiments/         # Experimental features
├── cicd/                    # Build and deployment tools
│   ├── backend              # Deployment CLI
│   └── it                   # Integration testing CLI
├── docker/                  # Docker image definitions
│   ├── backend-batch-orders/    # EMR-compatible batch processing
│   └── backend-lambda-orders/   # Lambda function container
├── commons/                 # Shared code libraries
│   ├── deno/                # Deno-compatible utilities
│   ├── node/                # Node.js utilities
│   └── any/                 # Platform-agnostic code
├── infrastructure/          # VPC and network infrastructure CDK
│   ├── cdk-vpc4emr/         # VPC for EMR clusters
│   ├── cdk-vpc4general/     # General purpose VPC
│   └── cdk-vpc4melissa/     # MelissaData integration VPC
└── book/                    # Test report output (mdBook)

Technology Stack

Compute Instance Tiers

AWS Batch job queues are configured with different instance sizes for various workload requirements:

Step Functions Workflows

The system deploys 17 Step Function state machines for different processing workflows:

Core P2A3 Workflows

Licensed Files Workflows

Operational Workflows

P2A3XGB Workflow Steps

The primary P2A3XGB workflow consists of 8 steps:

1. Initialize (S) - Set up working directories and validate inputs
2. TrainSelect (S) - Select training data based on model parameters
3. Train (L) - Execute XGBoost model training
4. ScoreSelect (M) - Prepare scoring dataset
5. Score (XXL) - Score all households against trained model
6. ReportsSelect (S) - Prepare reporting data
7. Reports (XXL) - Generate model performance reports
8. Finalization (S) - Clean up and stage outputs

Projects

Athanor

Location: projects/athanor/

A workflow runner for multi-step data processing with resume capability. Named after the alchemical furnace that transforms base materials into valuable outputs.

Key Features:

• Multi-step workflows with automatic resume from failure
• File-based locking (SHARED/EXCLUSIVE) for concurrent operations
• Dry-run mode for previewing steps
• State filtering for processing subsets of data
• EMR integration for distributed processing
• Dual execution mode (CLI and Step Functions)

Workflows:

• SumCats - Category-based audience summaries for digital advertising
• Summary - Customer profiles with demographics and purchase history
• Linkage - Identity matching files for cross-system data linking
• P2A3XGB - Standard XGBoost model training
• P2A3XGB-MCMC - MCMC-optimized training variant

Usage:

# Run a summary workflow
./bin/ath create summary --hh-date 2025-11-04

# Run with state filtering
./bin/ath create summary --hh-date 2025-11-04 --states ca,tx,ny

# Preview without executing
./bin/ath create summary --hh-date 2025-11-04 --dry-run

Response Analysis

Location: projects/response-analysis/

Automated batch runner for response analysis (RA) jobs. Determines which promotions need analysis based on transaction data availability and runs them automatically.

Key Features:

• Processes promotions where transaction data is after mail date
• Batches campaigns together for combined analysis
• Integrates with Dashboards audit data
• Runs auto_ra Python script for each promotion
• Stores results to EFS at /mnt/data/prod/<title>/ra/*

Data Sources:

• Dashboards audit account collection
• Households-memo transaction dates mapping
• Title transactions from current households file (EMR)
• Production data in /mnt/data/prod/*

Digital Audience

Location: projects/digital-audience/

Processing scripts for digital audience file generation. Symlinks to step-scripts implementation.

CICD Tools

The cicd/ directory contains two command-line tools for deployment and testing:

backend

Deployment utility for building and deploying CDK stacks.

Key Commands:

# Check current checkout status
backend info

# Deploy all stacks
backend deploy

# Deploy with clean CDK (recommended regularly)
backend deploy --clean-cdk

# Deploy single stack
backend deploy --stack p2a3xgb2

# Checkout staging branches
backend checkout-staging

# Checkout release tags
backend checkout-tag

it (Integration Testing)

Test execution and reporting utility.

Key Commands:

# Batch testing (recommended)
it batch all              # Run all tests with dependency resolution
it batch core licensed    # Run specific test suites
it batch all --dry-run    # Preview execution order

# Individual testing
it start p2a3all          # Start specific test
it ls                     # Check test status

# Reporting
it capture                # Capture test output
it report                 # Generate test report

Test Suites:

• core - Essential tests (p2a3all, p2a3counts, hotline)
• licensed - Licensed data processing tests
• licensed-v2 - New V2 licensed data tests
• fulfillment - Order processing and validation
• quick - Fast smoke tests (p2a3counts, hotline)
• all - Every available test

Integrations

Related Repositories

These projects must be checked out at the same level as cdk-backend:

AWS Services

External Systems

Development

Prerequisites

• Deno 2.0+ - Runtime for step scripts and Athanor
• Node.js - CDK and build tooling
• AWS CLI - Account access
• Maven - Building coop-scala
• mdBook - Documentation and test reports
• Docker - Container builds

Setup

1. Clone required repositories to same workspace level:

   git clone git@bitbucket.org:path2response/cdk-backend.git
   git clone git@bitbucket.org:path2response/coop-scala.git
   git clone git@bitbucket.org:path2response/ds-modeling.git
   git clone git@bitbucket.org:path2response/order-processing.git
   git clone git@bitbucket.org:path2response/data-science.git
   

2. Build coop-scala:

   cd coop-scala
   mvn clean install
   

3. Build cicd tools:

   cd cdk-backend/cicd
   ./build.sh
   

4. Configure CDK target in ~/.cdk.json:

   {
     "context": {
       "target": "staging",
       "version": 22301
     }
   }
   

Deployment

# From workspace root (not inside project)
cd ~/workspace/

# Verify checkout status
backend info

# Deploy (full clean recommended)
backend deploy --clean-cdk

Testing

# Run all tests with batch command
it batch all

# View results
it capture
it report

# View report in browser
cd /mnt/data/it/<version>/report
mdbook serve

Deployment Environments

Related Documentation

• Path2Acquisition Flow - Complete data flow diagram
• Glossary - Term definitions
• Response Analysis - RA system documentation
• Athanor Documentation - Full workflow runner documentation

Source: README.md, INSTALL.md, cdk-stepfunctions/README.md, cicd/README.md, projects/athanor/README.md, projects/response-analysis/README.md, docker/README.md, commons/any/README.md, step-scripts/README.md

Documentation created: 2026-01-24

DevTools Overview

Development automation tools for sprint management, release processes, and backlog workflows at Path2Response.

Purpose

The devtools repository contains a collection of Node.js CLI tools used by the Engineering team to automate recurring sprint and development processes. These tools integrate with Jira, Bitbucket, Slack, and Google Workspace to streamline the two-week sprint cycle and maintain operational efficiency.

Architecture

Repository Structure

devtools/
├── backlogManagement/      # PROD <-> PATH workflow automation
├── createRecurringSprintStories/  # Automated sprint story creation
├── findExternalShareFiles/ # Google Drive security audit
├── notifySprint/           # Sprint announcement notifications
├── setOrdersSupport/       # Support rotation management
├── sprintRelease/          # Release automation (tags, branches, versions)
├── sprintReviewExport/     # Sprint review data export
├── sprintTestPlan/         # Test plan generation
├── sprintVerifyVersions/   # Version verification for releases
├── testPlansNagging/       # Test plan reminder notifications
└── package.json            # Root orchestration

Technology Stack

Common Patterns

All tools follow consistent patterns:

1. CLI with yargs - Standard argument parsing with --help support
2. async.auto() - Dependency-based task orchestration
3. Dry-run mode - --dry-run flag for safe testing
4. JSON output - --json-output for programmatic consumption
5. Environment variables - Credentials via env vars, not config files

Tools

backlogManagement

Purpose: Manage PROD-PATH workflow transitions, including triage tasks and epic creation.

Location: backlogManagement/

CLI Commands:

Usage:

cd backlogManagement
npx ts-node bin/createPRODToPathEpic.ts --prod-jira-key PROD-1234
npx ts-node bin/createPRODToPathEpic.ts --prod-jira-key PROD-1234 --skip-refinement-task

Templates Used:

• PATH-24850: Triage task template
• PATH-24853: PATH epic template
• PATH-24855: Refinement task template

createRecurringSprintStories

Purpose: Automate creation of recurring Jira stories for sprint ceremonies, maintenance tasks, and operational activities.

Location: createRecurringSprintStories/

Story Builders Include:

• Sprint release activities (tags, branches, versions)
• Dependency maintenance across all projects
• Branch cleanup
• SSL certificate updates
• CDK backend installations (Dev, RC, Production)
• NCOA runs
• AWS cost reviews
• Security Hub/GuardDuty findings review
• Melissa data artifact updates
• Browse transaction counts
• End of month close activities
• Domain transfer management

Usage:

cd createRecurringSprintStories

# List available story builders for a sprint
node lib/createRecurringSprintStories.js -s 250 -n

# Dry-run for a specific sprint
node lib/createRecurringSprintStories.js --target-sprint-number 250 --dry-run

# Create specific stories only
node lib/createRecurringSprintStories.js -s 250 -o releaseSprint,performDependencyMaintenanceOnAllProjects

Environment Variables:

• JIRA_SERVICE_USER_NAME - Atlassian email
• JIRA_SERVICE_USER_PASSWORD - Atlassian API token

sprintRelease

Purpose: Automate sprint release activities including branching, tagging, version management, and deployments.

Location: sprintRelease/

CLI Actions (in lib/actions/):

Usage:

cd sprintRelease

# Create staging branch from production tag
node lib/actions/createBranches.js -b 330.0.0 -s staging/sprint-331 --repos all

# Tag releases
node lib/actions/tagReleases.js -t 331.0.0 --repos all --dry-run

# Check versions
node lib/actions/printVersions.js --repos all

sprintTestPlan

Purpose: Generate test plan content for Google Sheets from Jira sprint data.

Location: sprintTestPlan/

Key Files:

• sprintTestPlanExport.js - Main export script
• baseTests.json - Standard test cases
• automatedTests.json - Automated test definitions

Author: Michael Pelikan

sprintReviewExport

Purpose: Generate sprint review sheet contents from Jira data for sprint retrospectives.

Location: sprintReviewExport/

Key File: sprintReviewExport.js - Exports to CSV format

Author: Originally in jmalone’s paraphernalia, moved to devtools

sprintVerifyVersions

Purpose: Validate version consistency after DevOps performs sprint release activities.

Location: sprintVerifyVersions/

Key Files:

• verify_versions.sh - Main verification script
• getRepositoryProjects.js - Generates project list for verification

notifySprint

Purpose: Send sprint notifications to company Slack channels announcing sprint completion and new sprint start.

Location: notifySprint/

Key Files:

• notifySprint.js - Main notification script
• sprint_announcement.txt - Announcement template with placeholders
• activities.json - Random activities for announcement variety
• codeFreezeLanguage.txt - Code freeze notification text

Environment Variables:

• SLACK_TOKEN - Slack API token
• JIRA_HOST - Jira instance (default: caveance.atlassian.net)
• JIRA_SERVICE_USER_NAME - Jira user email
• JIRA_SERVICE_USER_TOKEN - Jira API token

Slack Channel: Posts to #general (C024QPRA3)

Author: Michael Pelikan

setOrdersSupport

Purpose: Update the #orders Slack channel topic based on the Engineering/Data Science support rotation schedule from Google Sheets.

Location: setOrdersSupport/

Dependencies: @slack/web-api, moment-timezone

Author: Michael Pelikan

testPlansNagging

Purpose: Send reminder messages to Slack for incomplete pre-release and post-release test plan items.

Location: testPlansNagging/

Features:

• Queries Jira for incomplete test items
• Sends targeted reminders via Slack
• Supports pre-release and post-release phases

Author: John Malone

findExternalShareFiles

Purpose: Security audit tool to find Google Drive files shared externally (“Anyone with link”) that are owned internally.

Location: findExternalShareFiles/

Author: Michael Pelikan

Integrations

Jira (Atlassian)

All tools connect to the Caveance Atlassian instance:

• Host: caveance.atlassian.net
• Authentication: Basic auth with API token
• Projects: PATH (Engineering), PROD (Product)
• Board ID: 4 (PATH Sprint Board)

Bitbucket

Sprint release tools interact with Bitbucket repositories:

• Repository URL pattern: git@bitbucket.org:path2response/{repo}.git
• Operations: clone, branch, tag, merge, push

Slack

Multiple tools post to Slack channels:

• notifySprint - Sprint announcements to #general
• setOrdersSupport - Updates #orders channel topic
• testPlansNagging - Test plan reminders

Google Workspace

• setOrdersSupport - Reads support rotation from Google Sheets
• findExternalShareFiles - Audits Google Drive sharing permissions

Nexus Repository

• sprintRelease - Deploys releases to internal Nexus

Development

Prerequisites

# Node.js version
node --version  # Should be ~22 or ~24

# npm version
npm --version   # Should be ~10 or ~11

Installation

# Clone repository
git clone git@bitbucket.org:path2response/devtools.git
cd devtools

# Install all sub-projects
npm run reinstall

Per-Project Commands

Each sub-project supports:

Environment Variables

Versioning

All tools share a version number with the main sprint releases:

• Current: 335.0.0-SNAPSHOT
• Format: {sprint}.{patch}.{build}[-SNAPSHOT]

Related Documentation

Source Files Read:

• /Users/mpelikan/Documents/code/p2r/devtools/README.md
• /Users/mpelikan/Documents/code/p2r/devtools/package.json
• /Users/mpelikan/Documents/code/p2r/devtools/backlogManagement/README.md
• /Users/mpelikan/Documents/code/p2r/devtools/sprintRelease/README.md
• /Users/mpelikan/Documents/code/p2r/devtools/createRecurringSprintStories/README.md
• All tool package.json files
• Sample source files for architecture patterns

Documentation created: 2026-01-24

Process for Product Management and Development Issues

Source: Confluence - Process for Product Management and Development Issues

This document defines the interaction between Product Management (PROD) and Development (PATH) at Path2Response. It is the foundational process that governs how work flows between teams.

Process Flowchart

The following diagram shows the complete process flow for issues in both PROD and PATH projects:

Key decision points:

• Product Management (left): Triages based on significance, Epic structure, technical nature, and readiness
• Development (right): Triages based on bug classification and technical scope
• Conversion points: Issues flow between PROD and PATH based on triage decisions

Overview

The Fundamental Distinction

The process recognizes two distinct but interdependent functions:

Product Management (PROD):

• Owns the “what” and “why” of products
• Defines product vision, strategy, and market fit
• Conducts pre-development activities: ideation, business case, ROI analysis, requirements definition
• Handles post-development activities: go-to-market, training, commercialization, rollout
• Manages both Revenue Products (client-facing) and Platform Products (internal tools)
• Responsible for ensuring development efforts align with business goals and deliver market value

Development Team (PATH):

• Owns the “how” of technical implementation
• Creates, designs, deploys, and supports software
• Executes on well-defined requirements with clear acceptance criteria
• Manages technical debt, architecture decisions, and system quality
• Focuses on building solutions that are performant, scalable, and maintainable

Why This Separation Is Critical

Without clear separation, several failure modes emerge:

1. Scope Creep: Development teams spend time on discovery and requirements that Product Management should own
2. Weak Requirements: Issues enter development without proper analysis, leading to rework
3. Misaligned Priorities: Urgent requests override strategic initiatives; ROI-driven prioritization breaks down
4. Incomplete Delivery: Products get built but not launched properly—missing training, documentation, or go-to-market
5. Burnout and Friction: Teams struggle with unclear accountability

Work Classification

Not all work requires the same level of process. Work at Path2Response falls into three categories:

Key principle: This document (PROD-PATH Process) defines how work flows between teams. The Stage-Gate Process defines when gates and approvals occur for Initiatives.

For detailed classification criteria, see the Work Classification Guide.

How They Work Together

• Initiatives follow Stage-Gate for governance, but use this PROD-PATH process for the actual work flow
• Non-Initiatives use this PROD-PATH process directly (no Stage-Gate gates)
• Operational work bypasses PROD entirely and goes directly to PATH

Jira Projects

Process Flow

Entry Points

PROD Issue Created

When someone creates an issue in PROD, they’re saying: “I have a product need, opportunity, or problem that requires Product Management attention.”

Examples:

• New product idea from Product Management
• Feature request from Production
• Strategic initiative from executives
• Internal tool improvement needed by Data Acquisition
• Market opportunity identified by Business Development

PATH Issue Created

When someone creates an issue in PATH, they’re saying: “I have a technical issue that needs Development attention.”

Examples:

• Bug in existing functionality
• Clearly-defined technical task (e.g., “upgrade Node.js to version 24”)
• Technical debt that needs addressing
• Performance issue requiring investigation

Reality of Entry Points

People often don’t know which project to create issues in. A reported “bug” might actually be a misunderstanding of how the product should work (a Product issue). A “feature request” might be a simple configuration change (a Development issue).

The process accommodates this reality. Both teams perform triage at entry, with explicit expectation that issues will be converted between PROD and PATH as needed. This is intentional design, not failure.

Product Management Flow

Step 1: “Does it Require Significant Work?”

This is the first triage decision for any PROD issue (see flowchart).

“Significant work” means:

• Multiple people or teams involved
• Timeline spans multiple sprints
• Coordination and planning required
• Substantial dependencies
• Outcome materially impacts the business

If NO (not significant): Goes directly to Product Scrum Process (routine PM work—tasks like updating marketing copy, creating presentations, conducting competitive analysis, writing documentation).

If YES (significant work): Proceeds to Epic evaluation (Step 2).

Step 2: “Is Issue an Epic?”

For significant work, the next question is whether it’s already structured as an Epic.

Why Epics matter:

• Provide structure for complex work
• Enable progress tracking and visibility
• Allow prioritization within initiative
• Clarify handoffs between teams
• Make dependencies visible

If NO (not yet an Epic): Convert to Epic first, then proceed.

If YES (already an Epic): Create child issues under the Epic for PM activities (discovery, ideation, analysis, requirements, stakeholder alignment).

Step 3: “Is Issue Technical?”

After Epic structure is established, determine if the work requires Development.

What “technical” means:

• The issue is about how something should be implemented, not what should be built or why
• Writing code, changing infrastructure, or modifying system architecture is clearly required
• Scope is well-understood and doesn’t require significant product discovery

What “technical” does NOT mean:

• “We need some kind of technical solution” (still a product problem—what solution?)
• “This involves our software” (everything does)
• “Development will eventually work on it” (true, but not yet)

If NO (not technical): Returns to Product Scrum Process for PM-only work.

If YES (technical): Proceeds to readiness evaluation (Step 4).

Step 4: “Is Valid and Simple?”

For technical issues, determine if they can go directly to PATH or need pre-development work.

“Valid” means:

• Legitimate need (not based on misunderstanding)
• Aligns with product strategy and priorities
• Business case is sound
• Stakeholders agree this should be done
• Priority is established based on business value and ROI

“Simple” means:

• Scope is clear and constrained
• Requirements are well-understood (meets PATH Epic Requirements)
• No significant discovery or analysis needed
• No cross-team coordination or complex dependencies
• Acceptance criteria are obvious and testable
• Source of truth for new data/fields is identified
• Impact on other products has been analyzed
• Release parameters are defined (patch vs standard release)