SIENTIAPDE-1478
Enhance PI Web API Scouter Workflow and Documentation - Added support for PI Web API data ingestion, including real-time and historical data retrieval. - Implemented timestamp normalization to ensure consistency across records. - Updated README.md to reflect new features and detailed workflow execution flow. - Enhanced API class with comprehensive error handling and cleanup operations. - Improved CoreScouter workflow with early exit conditions and detailed processing steps. - Updated configuration parameters for better clarity and usability.
This commit is contained in:
@@ -32,10 +32,14 @@ class Scouter:
|
||||
|
||||
This method orchestrates the complete data ingestion process:
|
||||
1. Retrieves the last processed timestamp from Redis
|
||||
2. Loads new data from MongoDB since the last timestamp
|
||||
3. Updates the last processed timestamp
|
||||
2. Loads new data from MongoDB since the last timestamp using collection name
|
||||
format: `raw_{schedule_name}`
|
||||
3. Updates the last processed timestamp with the most recent data point
|
||||
4. Delegates data processing to the CoreScouter workflow
|
||||
|
||||
If no new data is found in MongoDB, the workflow exits early without updating
|
||||
the timestamp or invoking the CoreScouter workflow.
|
||||
|
||||
Args:
|
||||
input_data (dict[str, Any]): Configuration and parameters for the workflow execution.
|
||||
Required fields:
|
||||
|
||||
Reference in New Issue
Block a user