Dashboards Application Overview

Legacy System Notice: Dashboards is a standalone Node.js application that has not yet been incorporated into BERT. A serverless migration plan exists to retire this application by splitting it into a CDK-based Audit Repository and migrating all user-facing functionality to BERT.

What is Dashboards?

The Dashboards application provides every person at Path2Response with an individualized and actionable view into the metrics, opportunities, tasks, and data that need their attention. It serves as the central hub for business intelligence, financial operations, and data synchronization across the company.

Purpose

Dashboards serves three primary functions:

1. Business Intelligence - 30+ specialized reports covering financial, operational, development, and client management aspects
2. Data Synchronization - The Audit system synchronizes data between Salesforce, QuickBooks, MongoDB, Slack, and Jira
3. Financial Operations - Invoice generation, payment recording, broker commission processing, and revenue forecasting

Current Architecture

Technology Stack

Project Structure

The application consists of two main components:

/dashboards/
├── app.js                    # Main Express server
├── /dashboards/              # Dashboard modules (account, finance, engineering)
├── /routes/                  # Express route handlers
├── /views/                   # EJS templates
├── /public/                  # Static assets (CSS, client-side JS)
├── /users/                   # User data storage (JSON files)
├── /sessions/                # Session file storage
└── /reports/                 # Audit and reporting system
    ├── audit.js              # Main audit script
    ├── /audit/               # Audit data output (JSON files)
    ├── /cache/               # Report cache
    └── /lib/                 # Core business logic
        ├── /audit/           # Data synchronization engine
        ├── /forecast/        # Revenue prediction engine
        ├── /reports/         # 30+ report modules
        └── /common/          # Shared utilities

Core Systems

1. Audit System

The Audit System is the core data synchronization engine. It fetches data from multiple external sources, assembles and processes it, performs analysis, and saves structured data that powers all reports and dashboards.

Six-Stage Pipeline

Load → Fetch → Assemble → Process → Analyze → Save

1. Load - Load existing Google users/groups and configuration
2. Fetch - Pull raw data from external sources
3. Assemble - Combine and structure raw data
4. Process - Transform and enrich assembled data
5. Analyze - Identify issues, generate findings, calculate metrics
6. Save - Persist processed data to JSON files

Data Sources

Running Audits

# Incremental audit (only changed data since last run)
./audit.js -d

# Complete audit (fetch all data from scratch)
./audit.js -c -d

Standard Audit Flow (Production)

run_audit.sh
├── audit.js -d
├── clear_cache.sh
└── [Production Only]
    ├── generate_invoices.sh
    ├── record_payments.sh
    ├── record_mailfiles.sh
    ├── update_salesforce.sh
    └── sync_to_s3.sh

2. Reports System

The Reports System generates comprehensive business intelligence reports, transforming audit data into actionable insights. It consists of 30+ specialized reports.

Report Categories

Financial Reports

Client Management Reports

Operations Reports

Development Reports (Jira Integration)

Report Architecture

Reports follow a standard pattern:

1. Define audit data requirements (audit_parts)
2. Load audit data via loadAudit
3. Process data according to business rules
4. Generate HTML output via HtmlBuilder
5. Return HTML string or JSON data

Reports are cached in /reports/cache/ and invalidated when new audits complete.

3. Forecast System

The Forecast System is a sophisticated revenue prediction engine that analyzes historical ordering patterns to forecast future revenue up to 3 years out.

“The best predictor of future behavior is past behavior.”

Four-Stage Pipeline

Audit Data → Analyze → Predict → Forecast → Google Sheets

1. Analyze - Consolidate orders monthly, detect ordering frequency
2. Predict - Detect trends and statistical patterns
3. Forecast - Generate revenue projections
4. Google - Publish results to Google Sheets

Account Classifications

By Maturity:

• Mature Titles - Ordering with P2R for ~1 year or more (high confidence)
• Young Titles - New to P2R, limited history (uses vertical averages)
• Stubbed Titles - Placeholder accounts for projected BizDev pipeline

By Vertical: Catalog, DTC, Nonprofit, Non Participating, PreScreen, Publishing

By Classification: Class A (largest), Class B, Class C, Class D (smallest)

Key Features

• Custom year definition (365 days from first order, not calendar)
• Monthly order consolidation for consistent analysis
• CV (credit mailer) filtering from revenue trends
• Forecast locking after 3rd of each month
• New client acquisition modeling from BizDev pipeline

Business Process Scripts

Core Operations

Monthly Processes

Broker Commissions - Critical monthly finance process:

1. Finance validates broker invoices and credit memos
2. Engineering runs generateBrokerCommissions.js -m YYYY-MM
3. System creates Journal Entries, Bills, and Vendor Credits in QuickBooks
4. Finance reconciles and closes books

# Run for previous month (e.g., if current month is July 2026)
./generateBrokerCommissions.js -m 2026-06

Revenue Forecasting

# Generate algorithmic revenue forecast
./forecast_revenue.sh

Output written to /reports/cache/revenueForecast/ and published to Google Sheets.

Integration Points

Data Pipeline

MongoDB/Salesforce/QuickBooks → Audit → Cache → Reports → Dashboards

Authentication Flow

Google OAuth → Passport → Session → User JSON files

External URLs

Environment Configuration

Required Environment Variables

# Salesforce
SALESFORCE_ACCOUNT="account"
SALESFORCE_PASSWORD="password"
SALESFORCE_TOKEN="token"

# Slack
SLACK_TOKEN="token"

# MongoDB
MONGO_ORDERPROCESSING="connection_string"
MONGO_OPERATIONS="connection_string"
MONGO_STATISTICS="connection_string"

# AWS
P2R_BUCKET_URI="s3://bucket-name"
P2R_RUNTIME_ENVIRONMENT="production"  # Controls post-audit operations

Serverless Migration Plan

Executive Summary

The current Dashboards repository will be retired and split into two projects:

1. New Audit Repository (CDK Serverless) - Core data processing, audit, forecast, and post-audit tools
2. BERT Integration - All user-facing functionality including routes, views, and report generation

Migration Phases

Phase 1: Create Audit Repository (CDK)

• Extract /reports/lib/audit/ → Lambda functions
• Extract /reports/lib/forecast/ → Step Functions workflow
• Migrate post-audit scripts → EventBridge + Lambda
• Store data in S3 instead of local JSON files

Phase 2: Migrate User-Facing to BERT

• Convert Express routes → BERT React Router
• Convert EJS templates → React components
• Convert report modules → BERT report components
• Migrate authentication to BERT’s auth system

Phase 3: Integration and Testing

• Connect BERT to Audit Repository APIs
• Parallel operation with gradual user migration
• Performance validation

Phase 4: Retirement

• Complete migration of all users
• Decommission Dashboards repository

Target Architecture

┌─────────────────────────────────────────────────────────────┐
│                    BERT Application                         │
│              (All Dashboard/Report UI)                      │
│  • Routes (migrated from /routes)                           │
│  • Components (migrated from /views + /reports/lib/reports) │
│  • Dashboards (migrated from /dashboards)                   │
└──────────────────────┬──────────────────────────────────────┘
                       │ API Calls
                       ↓
                ┌──────────────────┐
                │ Audit Repository │
                │   (CDK Project)  │
                │    API Gateway   │
                └──────┬───────────┘
                       │
        ┌──────────────┼────────────┐
        ↓              ↓            ↓
   ┌──────────┐ ┌──────────┐ ┌──────────┐
   │  Lambda  │ │  Lambda  │ │   Step   │
   │ (Audit)  │ │(Forecast)│ │Functions │
   └─────┬────┘ └─────┬────┘ └─────┬────┘
         │            │            │
         └────────────┼────────────┘
                      ↓
        ┌────────────────────────────┐
        │        Data Storage        │
        │  S3 Buckets | EventBridge  │
        │  (Audit/Forecast/Cache)    │
        └────────────────────────────┘

AWS Services in Target Architecture

Key Benefits

1. Scalability - Automatic scaling based on demand
2. Availability - Multi-region deployment options
3. Maintenance - Reduced server management overhead
4. Cost - Pay-per-use model vs. always-on EC2

Risk Mitigations

Related Documentation

• Data Tools Overview - Another legacy system also targeted for BERT migration
• BERT Overview - Target platform for consolidated applications
• Production Team Overview - Production operations that use Dashboards
• Data Acquisition Overview - Data operations that trigger audits

Important Notes

• Audit freshness is critical - Most operations require recent audit data
• Never run multiple audits simultaneously
• Production safety - Many modules include testing flags to prevent accidental modifications
• Network access - MongoDB connections require VPN/developer network access
• Broker commissions - Requires Finance department coordination before running
• Data dependencies - Reports depend on audit system; forecast depends on audit

Source: Dashboards repository documentation (README.md, reports/README.md, lib/README.md, lib/audit/README.md, lib/forecast/README.md, lib/reports/README.md) Documentation created: 2026-01-24