Data Tools Services Overview
Legacy System Notice: Data Tools is a legacy application targeted for migration into BERT (Base Environment for Re-tooled Technology) in 2026. This documentation captures the current state for reference during the migration planning process.
What is Data Tools?
Data Tools is a collection of automated services that handle data processing workflows at Path2Response. These services manage file ingestion, conversion, validation, and various operational tasks that support the core data cooperative business.
Architecture
Data Tools consists of three categories of services:
- Embedded Services - Run within the main app.js application
- Micro Services - Standalone services for specific processing tasks
- Deprecated Services - Legacy functionality no longer in active use
1. Embedded Services (app.js)
These services are embedded directly in the main Data Tools application:
| Service | Description |
|---|---|
| autoConvert | Picks up files where the map is enabled and the file tagging matches a map |
| autoPreprocess | Creates documents for custom processes where there is a single file tagged as preprocess and the preprocess is auto-enabled |
| autoEnableRoute | Re-enables routes that show up in SFTP Waiting when the route has been manually disabled |
| autoValidate | Automatically reviews documents where the map has AutoValidate enabled. Marks “Review Complete” for those that pass thresholds |
| oldAccount | Non-functional. When enabled only provides reporting on SmartFile users that last logged in over a year ago. Configuration document indefinitely disabled |
| productMapImport | Finds and processes productMap files provided by Production Operations |
| browseStatus | Checks the titlekey profiles of second party converts and updates #sitetagging Slack channel with status |
| sftpWaiting | Scans SmartFile for client data that has not been pulled down. Mainly used to monitor locked SmartFile paths (brokers that post with the wrong credentials) |
| updateTitlesInZendesk | Updates the list of titlekeys in Zendesk for organizational purposes |
2. Micro Services
Standalone services that handle specific processing workflows:
| Service | Description |
|---|---|
| responseAnalysis | Collects stats on fulfillment files, mail files, list of lists, and response files, as well as how they overlap, in order to provide feedback on order performance |
| fileOperationsMaster | Scans all data rules/incoming routes for data to be pulled. Creates documents picked up by fileOperationsChild |
| fileOperationsChild | Processes documents created by fileOperationsMaster. Initial extraction and cleaning of client provided data into plain text format able to be processed by convert |
| importRunner/convert | Process that picks up files designated to be processed by convert. Picks up “convertrun” documents and appends them with post convert stats |
| preprocessRunner | Picks up new custom process requests and submits them to their relative custom process |
| cassRunner | Picks up files that contain address information that have completed convert and have been marked “Review Complete”. Attaches CASS data objects |
| missingImportRunner | Queries mongo for expected data and posts to #orders-data with data that is not in the anticipated state |
| taskManager | Creates tasks relating to global block mismatch status, Salesforce title sync status, and recency |
| salesForceSync | Pulls information from SalesForce (via Dashboards) to monitor titlekey matching and global block mismatches |
| dsrRunner | Looks for Data Subject Request tickets in Zendesk (usually originating from OneTrust) and performs the appropriate DSR action. Automatically closes Zendesk tickets and corresponding OneTrust requests when finished processing |
3. Deprecated Services
Services no longer in active use:
| Service | Description |
|---|---|
| sameDayOmitRunner | Grabs HHids / IndIds after CASSing data and matching to households. Used for suppression purposes. |
Key Integrations
Data Tools integrates with several internal and external systems:
- SmartFile - SFTP server for client file uploads
- MongoDB - Document storage for processing state
- Slack - Notifications (#sitetagging, #orders-data channels)
- Zendesk - Ticket management and titlekey organization
- Salesforce - Title synchronization via Dashboards
- OneTrust - Data Subject Request origination
Data Flow
Client Data (SmartFile)
↓
fileOperationsMaster (scan routes)
↓
fileOperationsChild (extract/clean)
↓
importRunner/convert (process)
↓
autoValidate (review)
↓
cassRunner (CASS append)
↓
Ready for downstream processing
Migration Considerations
When migrating to BERT, consider:
- Service consolidation - Many embedded services may be candidates for consolidation
- Non-functional services -
oldAccountis disabled and may not need migration - Deprecated services -
sameDayOmitRunnershould be evaluated for retirement - Integration points - SmartFile, Zendesk, Salesforce integrations need BERT equivalents
- Notification patterns - Slack channel notifications should map to BERT alerting
Source: Data Tools Services (Confluence) Last updated from source: 2019-07-25 Documentation created: 2026-01-24