Merge pull request #29 from Aignosi/fix/SIENTIAPDE-1478

Fix/sientiapde 1478

README.md

@@ -5,7 +5,7 @@ A high-performance, scalable data processing and ML model orchestration system b
 ## Features
 
 ### Core Functionality
-- **Multi-Source Data Ingestion**: Support for Kafka topics, direct OPC server access, and real-time triggers
+- **Multi-Source Data Ingestion**: Support for Kafka topics, direct OPC server access, PI Web API endpoints, and real-time triggers
 - **Temporal Workflow Orchestration**: Robust workflow management with automatic retry policies and fault tolerance
 - **Data Quality Gates**: Configurable filtering for null values, out-of-bounds data, and custom validation rules
 - **Time-Series Aggregation**: Flexible aggregation functions (average, median, max, min, latest) with configurable parameters
@@ -102,9 +102,14 @@ The **CoreScouter** workflow implements the core data processing pipeline for in
 1. **Data Quality Gate**: Applies configured filters (null values, out-of-bounds, custom rules)
 2. **Data Aggregation**: Groups data by tag and name, applies aggregation functions
 3. **Data Grouping**: Organizes data and stores temporarily in Redis with TTL
-4. **Data Export**: Persists processed data to PostgreSQL database
+4. **Data Export**: Persists processed data to PostgreSQL database with timestamp conversion
 5. **Metrics Recording**: Writes processing metrics for operational visibility
 
+**Note**: The data export step uses timestamp conversion to ensure consistent datetime
+formatting. The export operation receives the schema, table name, data, and timestamp
+conversion configuration. Conflict resolution and unique column constraints are handled
+by the underlying PostgreSQL activity implementation.
+
 #### Aggregation Functions
 - **`lts`**: Latest value (most recent data point)
 - **`avg`**: Average of all values in the group
@@ -171,6 +176,102 @@ When `debug_data_package` is set to `true`, the workflow stores both raw and pro
 - Validating data transformations
 - Auditing data quality gate decisions
 
+
+### 3. PI Web API Scouter Workflow (`pi_web_api_scouter.py`)
+
+The **PI Web API Scouter** workflow serves as the entry point for PI Web API data processing pipelines. Unlike the standard Scouter workflow that loads data from MongoDB collections, this workflow directly queries PI Web API endpoints to retrieve tag values and processes them for downstream use.
+
+#### Purpose
+- **Direct API Ingestion**: Retrieves data directly from PI Web API endpoints
+- **Real-time Data Processing**: Supports real-time and historical data retrieval
+- **Data Normalization**: Normalizes timestamps to ensure consistency across records
+- **Workflow Orchestration**: Delegates data processing to the CoreScouter workflow
+- **Error Handling**: Comprehensive error handling with retry policies
+
+#### Execution Flow
+1. **Tag Value Retrieval**: Retrieves tag values from PI Web API using configured WebIds and time periods
+2. **Data Normalization**: Normalizes timestamps to ensure all records in a batch share the same timestamp value
+3. **Data Validation**: Validates retrieved data and handles empty responses
+4. **Data Processing**: Delegates data processing to the CoreScouter child workflow
+
+**Note**: The timestamp normalization process converts all timestamps to string format and then sets all records to the maximum timestamp value (lexicographically) found in the dataset. This ensures consistency across all records in a single batch.
+
+#### Key Features
+- **Configurable Time Periods**: Supports flexible time period configurations (e.g., '*-1d', '*-1h')
+- **Data Point Limits**: Configurable maximum data points per tag via `max_count` parameter
+- **Timeout Management**: Configurable API request timeouts for reliable operation
+- **Empty Data Handling**: Gracefully handles empty responses without processing
+- **Standardized Processing**: Uses CoreScouter for consistent data quality and export operations
+
+#### Input Parameters
+```json
+{
+  "model_name": "pi_sensors",
+  "model_id": "pi_001",
+  "schedule_name": "hourly_pi_collection",
+  "pi_web_api_query": {
+    "endpoint": "/streamsets/recorded",
+    "period": "*-1d",
+    "max_count": 10,
+    "api_timeout": 30
+  },
+  "model_tags": {
+    "Temperature": {
+      "webid": "F1AbCdEfGhIjKlMnOpQrStUvWxYz",
+      "aggr_function": "avg",
+      "data_range": [-50, 150]
+    },
+    "Pressure": {
+      "webid": "F2AbCdEfGhIjKlMnOpQrStUvWxYz",
+      "aggr_function": "max",
+      "data_range": [0, 100]
+    }
+  },
+  "trigger_laborious": false,
+  "filters": {
+    "OUT_OF_BOUNDS_FILTER": {"policy": "DISCARD"},
+    "NULL_VALUES_FILTER": {"policy": "DISCARD"}
+  },
+  "schema": "sensor_data",
+  "table_name": "pi_readings",
+  "retention_time": 3600,
+  "fill_missing_tags": false,
+  "debug_data_package": false
+}
+```
+
+**PI Web API Query Parameters:**
+- `endpoint` (str): PI Web API endpoint path (e.g., '/streamsets/recorded')
+- `period` (str): Time period configuration (e.g., '*-1d' for last day, '*-1h' for last hour)
+- `max_count` (int, optional): Maximum data points per tag. Defaults to 1
+- `api_timeout` (int): Request timeout in seconds for PI Web API calls
+
+**Model Tags Configuration:**
+- `webid` (str): PI Web API WebId for the tag
+- `aggr_function` (str): Aggregation method (avg, mdn, max, min, lts)
+- `data_range` (list[int]): [min, max] values for data validation
+
+#### Architecture
+
+```mermaid
+flowchart LR
+    A[1. get_tag_values] --> B{data empty?}
+    B -->|yes| C[Exit]
+    B -->|no| D[2. core_scouter 🔃]
+    
+    A -.-> PI_API[(PI Web API)]
+    D -.-> CoreScouter[CoreScouter Workflow]
+```
+
+#### Data Normalization
+
+The `get_tag_values` activity normalizes timestamps to ensure consistency:
+1. Converts all timestamps to string format using the configured datetime format
+2. Identifies the maximum timestamp value (lexicographically) in the dataset
+3. Sets all records to use this normalized timestamp value
+
+This normalization ensures that all records in a single batch share the same timestamp, which is useful for batch processing and data consistency in downstream operations.
+
 ## 📋 Prerequisites
 
 - Python 3.11+
@@ -179,6 +280,7 @@ When `debug_data_package` is set to `true`, the workflow stores both raw and pro
 - Redis server
 - MongoDB server
 - Kafka cluster (for data ingestion)
+- PI Web API server (for PI Web API Scouter workflow)
 
 **Note**: External dependencies must be available either through:
 - Kubernetes cluster deployment
@@ -365,6 +467,9 @@ The Scouter system exposes comprehensive Prometheus metrics:
 | `REDIS_PORT` | Redis port | `6379` | Yes |
 | `MONGODB_URL` | MongoDB connection URI | `localhost:27017` | Yes |
 | `KAFKA_BOOTSTRAP_SERVERS` | Kafka broker addresses | `localhost:9092` | No |
+| `PI_WEB_API_BASE_URL` | PI Web API base URL | - | Yes (for PI Web API Scouter) |
+| `PI_WEB_API_AUTH_TYPE` | PI Web API authentication type ('basic' or 'bearer') | - | Yes (for PI Web API Scouter) |
+| `PI_WEB_API_AUTH_TOKEN` | PI Web API authentication token | - | Yes (for PI Web API Scouter) |
 | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
 | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
 | `PROJECT_NAME` | Project identifier for notifications | `scouter` | No |
@@ -387,16 +492,16 @@ The worker implements aggressive autoscaling policies for workflow and activity
 **Workflow Poller Behavior:**
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `WORKFLOW_POLLER_BEHAVIUR_MINIMUM` | Minimum workflow pollers | `10` |
+| `WORKFLOW_POLLER_BEHAVIOUR_MINIMUM` | Minimum workflow pollers | `10` |
-| `WORKFLOW_POLLER_BEHAVIUR_INITIAL` | Initial workflow pollers | `100` |
+| `WORKFLOW_POLLER_BEHAVIOUR_INITIAL` | Initial workflow pollers | `100` |
-| `WORKFLOW_POLLER_BEHAVIUR_MAXIMUM` | Maximum workflow pollers | `200` |
+| `WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM` | Maximum workflow pollers | `200` |
 
 **Activity Poller Behavior:**
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `ACTIVITY_POLLER_BEHAVIUR_MINIMUM` | Minimum activity pollers | `10` |
+| `ACTIVITY_POLLER_BEHAVIOUR_MINIMUM` | Minimum activity pollers | `10` |
-| `ACTIVITY_POLLER_BEHAVIUR_INITIAL` | Initial activity pollers | `100` |
+| `ACTIVITY_POLLER_BEHAVIOUR_INITIAL` | Initial activity pollers | `100` |
-| `ACTIVITY_POLLER_BEHAVIUR_MAXIMUM` | Maximum activity pollers | `200` |
+| `ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM` | Maximum activity pollers | `200` |
 
 ### Workflow Configuration
 
@@ -460,11 +565,13 @@ MongoDB pipeline configuration:
 scouter/
 ├── activities/              # Temporal activity implementations
 │   ├── activities.py       # Main activities orchestrator
+│   ├── api.py              # PI Web API operations (tag value retrieval)
 │   ├── redis.py           # Redis operations (caching, timestamps)
 │   ├── gates.py            # Data quality gates and filtering
 │   └── mongodb.py          # MongoDB operations (data loading)
 ├── workflow/                # Temporal workflow definitions
 │   ├── scouter.py          # Main data ingestion workflow
+│   ├── pi_web_api_scouter.py # PI Web API data ingestion workflow
 │   └── sub_workflows/      # Sub-workflow implementations
 │       └── core_scouter.py # Core data processing workflow
 ├── worker/                  # Worker implementation
@@ -484,6 +591,7 @@ The Activities class combines multiple service classes through multiple inherita
 - **Redis**: Timestamp management, data caching, and temporary storage
 - **Gates**: Data quality validation and filtering logic
 - **MongoDB**: Data loading from raw collections
+- **API**: PI Web API tag value retrieval and data normalization
 
 All activities support:
 - Comprehensive logging and error handling

e2e/conftest.py

@@ -30,7 +30,7 @@ TEST_MONGODB_CONNECTION_STRING = 'mongodb://localhost:27017'
 TEST_DATABASE_NAME = 'test_db'
 
 
-@pytest.fixture(scope='session')
+@pytest_asyncio.fixture(scope='session')
 def postgres_container():
     """
     Create a PostgreSQL container using testcontainers.
@@ -44,7 +44,7 @@ def postgres_container():
     postgres.stop()
 
 
-@pytest.fixture
+@pytest_asyncio.fixture
 def postgres_engine(postgres_container):
     """
     Create SQLAlchemy engine for PostgreSQL test database.
@@ -81,7 +81,6 @@ def _create_schema_and_table(engine):
         
         # Create table WITHOUT partitioning (simpler for tests)
         # Same structure as production, but without PARTITION BY RANGE
-        # Use UNIQUE constraint directly since table is not partitioned
         create_table_sql = f"""
             CREATE TABLE IF NOT EXISTS {schema_name}.{table_name} (
                 id SERIAL NOT NULL,
@@ -90,8 +89,7 @@ def _create_schema_and_table(engine):
                 value numeric NULL,
                 "timestamp" timestamptz NOT NULL,
                 created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
-                PRIMARY KEY (id, created_at),
+                PRIMARY KEY (id, created_at)
-                UNIQUE (model_id, timestamp, variable)
             );
         """
         
@@ -99,7 +97,7 @@ def _create_schema_and_table(engine):
         # Transaction is automatically committed when exiting the 'with' block
 
 
-@pytest.fixture(autouse=True)
+@pytest_asyncio.fixture(autouse=True)
 def setup_postgres_schema_and_table(postgres_engine):
     """
     Automatically create necessary schema and table before each test.
@@ -108,15 +106,14 @@ def setup_postgres_schema_and_table(postgres_engine):
     that the sientia_data schema and laborious_data table exist
     with the correct structure before tests execute.
     
-    Note: For tests, we use a non-partitioned table with a UNIQUE constraint
+    Note: For tests, we use a non-partitioned table which is simpler and avoids issues
-    directly in the table definition, which is simpler and avoids issues
     with pandas to_sql recognizing partitioned tables.
     """
     _create_schema_and_table(postgres_engine)
     yield
 
 
-@pytest.fixture
+@pytest_asyncio.fixture
 def mock_logger():
     """Mock logger for testing."""
     logger = MagicMock(spec=Logger)
@@ -128,7 +125,7 @@ def mock_logger():
     return logger
 
 
-@pytest.fixture
+@pytest_asyncio.fixture
 def mock_mongo_client():
     """
     Mock MongoDB client to avoid real connections.
@@ -153,7 +150,7 @@ def mock_mongo_client():
     return mock_client
 
 
-@pytest.fixture
+@pytest_asyncio.fixture
 def notification_handler(mock_logger, mock_mongo_client):
     """
     Create a real NotificationHandler instance with mocked MongoDB client.
@@ -173,7 +170,7 @@ def notification_handler(mock_logger, mock_mongo_client):
         handler.shutdown()
 
 
-@pytest.fixture
+@pytest_asyncio.fixture
 def metrics_controller(mock_logger):
     """
     Create a real MetricsController instance.
@@ -186,7 +183,7 @@ def metrics_controller(mock_logger):
     # MetricsController might have cleanup, but it's optional
 
 
-@pytest.fixture
+@pytest_asyncio.fixture
 def mock_pi_web_api_client():
     """Mock PI Web API client."""
     mock_client = MagicMock()
@@ -212,8 +209,8 @@ def mock_pi_web_api_client():
     return mock_client
 
 
-@pytest_asyncio.fixture
+@pytest_asyncio.fixture(scope='function')
-async def test_activities(
+def test_activities(
     postgres_engine,
     postgres_container,
     mock_logger,

e2e/scenarios.md

@@ -299,25 +299,6 @@ The `pi_web_api_scouter` workflow:
 
 ---
 
-#### Scenario 2.2.2: Zero Affected Rows After Export
-**Description**: PostgreSQL export returns zero affected rows
-
-**Input**:
-- Data that results in `affected_rows: 0` from export
-
-**Expected Behavior**:
-- `export_data_to_postgres` returns `{'affected_rows': 0}`
-- Workflow checks `if data_exported.get('affected_rows', 0) <= 0:` and returns early
-- `write_metrics` NOT called
-- `store_data_package` NOT called
-
-**Assertions**:
-- Early return after export
-- No metrics written
-- Workflow completes without error
-
----
-
 ### 2.3 Error Scenarios
 
 #### Scenario 2.3.1: Redis Connection Error
@@ -338,25 +319,6 @@ The `pi_web_api_scouter` workflow:
 
 ---
 
-#### Scenario 2.3.2: PostgreSQL Unique Constraint Violation
-**Description**: Duplicate data violates unique constraint
-
-**Input**:
-- Data with duplicate `model_id`, `timestamp`, `variable` combination
-- `on_conflict: 'ignore'` configured
-
-**Expected Behavior**:
-- PostgreSQL handles conflict with `ON CONFLICT DO NOTHING`
-- `affected_rows` may be 0 for duplicates
-- Workflow continues normally
-
-**Assertions**:
-- No exception raised
-- Duplicates ignored
-- Workflow continues
-
----
-
 ## 3. Activity-Specific Scenarios
 
 > **Note**: Activity-specific scenarios are better suited for unit tests rather than e2e tests.
@@ -457,7 +419,6 @@ For each scenario, verify:
 2. Scenario 2.1.2: Quality Filters
 3. Scenario 2.1.3: Different Aggregations
 4. Scenario 3.3.5: Invalid Aggregation
-5. Scenario 3.5.2: Conflict Ignore
 
 ### Low Priority (Nice to Have)
 1. Scenario 4.1.2: Retry Success

e2e/test_core_scouter_early_exit.py

@@ -94,100 +94,3 @@ async def test_scenario_2_2_1_empty_data_after_grouping(
     # Should have 0 rows since export_data_to_postgres was not called
     assert row_count == 0, f"Expected no data in PostgreSQL, got {row_count} rows"
 
-
-@pytest.mark.asyncio
-@pytest.mark.integration
-async def test_scenario_2_2_2_zero_affected_rows_after_export(
-    temporal_test_env: WorkflowEnvironment,
-    temporal_worker: Worker,
-    test_activities: Activities,
-    postgres_engine,
-):
-    """
-    Scenario 2.2.2: Zero Affected Rows After Export
-    
-    PostgreSQL export returns zero affected rows, workflow exits early.
-    
-    Note: This scenario is hard to test directly in e2e because we'd need to
-    simulate a conflict or other condition that results in 0 affected rows.
-    We'll test by inserting duplicate data first, then running the workflow again.
-    """
-    client = temporal_test_env.client
-
-    # First, insert some data directly to create a conflict scenario
-    test_data = [
-        {
-            'timestamp': '2024-01-01 12:00:00+0000',
-            'name': 'tag1',
-            'value': 10.5,
-            'tag': 'webid1',
-        },
-    ]
-
-    # Insert data directly into PostgreSQL to create duplicates
-    schema_name = 'sientia_data'
-    table_name = 'laborious_data'
-    full_table_name = f"{schema_name}.{table_name}"
-
-    with postgres_engine.connect() as conn:
-        conn.execute(
-            text(f"""
-                INSERT INTO {full_table_name} (model_id, variable, value, timestamp)
-                VALUES (1, 'tag1', 10.5, '2024-01-01 12:00:00+00:00')
-                ON CONFLICT (model_id, timestamp, variable) DO NOTHING
-            """)
-        )
-        conn.commit()
-
-    # Prepare input data with the same data (will result in conflict)
-    input_data = {
-        'metadata': {
-            'metadata': {
-                'model_id': '1',
-                'model_name': 'Test Model',
-                'schedule_name': 'test-schedule',
-                'workflow_name': 'pi_web_api_scouter',
-            }
-        },
-        'workflow_name': 'pi_web_api_scouter',
-        'schedule_name': 'test-schedule',
-        'model_name': 'Test Model',
-        'model_id': '1',
-        'data': test_data,
-        'trigger_laborious': False,
-        'filters': {},
-        'schema': 'sientia_data',
-        'table_name': 'laborious_data',
-        'retention_time': 3600,
-        'fill_missing_tags': False,
-        'model_tags': {
-            'tag1': {
-                'webid': 'webid1',
-                'aggr_function': 'avg',
-                'data_range': [0, 100],
-                'frequency': 60000,
-            },
-        },
-    }
-
-    # Start workflow
-    handle = await client.start_workflow(
-        CoreScouter.run,
-        input_data,
-        id=f'test-core-scouter-zero-{datetime.now().timestamp()}',
-        task_queue='test-queue',
-    )
-
-    # Wait for workflow completion (should complete without error)
-    await handle.result()
-
-    # Verify the count didn't increase (conflict handled, 0 affected rows)
-    with postgres_engine.connect() as conn:
-        result = conn.execute(
-            text(f"SELECT COUNT(*) FROM {full_table_name} WHERE model_id = 1")
-        )
-        row_count = result.scalar()
-
-    # Should still have 1 row (the original one, duplicate was ignored)
-    assert row_count == 1, f"Expected 1 row (duplicate ignored), got {row_count}"
-

e2e/test_core_scouter_errors.py

@@ -88,102 +88,3 @@ async def test_scenario_2_3_1_redis_connection_error(
         # Restore original method
         test_activities.redis_repository.get = original_get
 
-
-@pytest.mark.asyncio
-@pytest.mark.integration
-async def test_scenario_2_3_2_postgresql_unique_constraint_violation(
-    temporal_test_env: WorkflowEnvironment,
-    temporal_worker: Worker,
-    test_activities: Activities,
-    postgres_engine,
-):
-    """
-    Scenario 2.3.2: PostgreSQL Unique Constraint Violation
-    
-    Duplicate data violates unique constraint, handled gracefully with ON CONFLICT DO NOTHING.
-    """
-    client = temporal_test_env.client
-
-    # Generate unique model_id to avoid conflicts with other tests
-    unique_id = int(datetime.now().timestamp() * 1000) % 1000000
-
-    # First, insert data directly to create a duplicate
-    schema_name = 'sientia_data'
-    table_name = 'laborious_data'
-    full_table_name = f"{schema_name}.{table_name}"
-
-    with postgres_engine.connect() as conn:
-        conn.execute(
-            text(f"""
-                INSERT INTO {full_table_name} (model_id, variable, value, timestamp)
-                VALUES (:model_id, 'tag1', 10.5, '2024-01-01 12:00:00+00:00')
-                ON CONFLICT (model_id, timestamp, variable) DO NOTHING
-            """),
-            {'model_id': unique_id}
-        )
-        conn.commit()
-
-    # Prepare the same data to trigger conflict
-    test_data = [
-        {
-            'timestamp': '2024-01-01 12:00:00+0000',
-            'name': 'tag1',
-            'value': 10.5,
-            'tag': 'webid1',
-        },
-    ]
-
-    input_data = {
-        'metadata': {
-            'metadata': {
-                'model_id': str(unique_id),
-                'model_name': 'Test Model',
-                'schedule_name': 'test-schedule',
-                'workflow_name': 'pi_web_api_scouter',
-            }
-        },
-        'workflow_name': 'pi_web_api_scouter',
-        'schedule_name': 'test-schedule',
-        'model_name': 'Test Model',
-        'model_id': str(unique_id),
-        'data': test_data,
-        'trigger_laborious': False,
-        'filters': {},
-        'schema': 'sientia_data',
-        'table_name': 'laborious_data',
-        'retention_time': 3600,
-        'fill_missing_tags': False,
-        'model_tags': {
-            'tag1': {
-                'webid': 'webid1',
-                'aggr_function': 'avg',
-                'data_range': [0, 100],
-                'frequency': 60000,
-            },
-        },
-    }
-
-    # Start workflow
-    handle = await client.start_workflow(
-        CoreScouter.run,
-        input_data,
-        id=f'test-core-scouter-conflict-{datetime.now().timestamp()}',
-        task_queue='test-queue',
-    )
-
-    # Wait for workflow completion - should complete without error
-    # (conflict is handled gracefully with ON CONFLICT DO NOTHING)
-    await handle.result()
-
-    # Verify no exception was raised and workflow completed
-    # The duplicate should be ignored (0 affected rows), but workflow should complete
-    with postgres_engine.connect() as conn:
-        result = conn.execute(
-            text(f"SELECT COUNT(*) FROM {full_table_name} WHERE model_id = :model_id"),
-            {'model_id': unique_id}
-        )
-        row_count = result.scalar()
-
-    # Should still have 1 row (duplicate was ignored)
-    assert row_count == 1, f"Expected 1 row (duplicate ignored), got {row_count}"
-

e2e/test_pi_web_api_scouter.py

@@ -1,106 +0,0 @@
-"""
-End-to-end tests for PI Web API Scouter workflow.
-"""
-
-from datetime import datetime
-
-import pytest
-from sqlalchemy import inspect, text
-from temporalio.testing import WorkflowEnvironment
-from temporalio.worker import Worker
-
-from scouter.activities.activities import Activities
-from scouter.workflow.pi_web_api_scouter import PIWebAPIScouter
-
-
-@pytest.mark.asyncio
-@pytest.mark.integration
-async def test_pi_web_api_scouter_e2e(
-    temporal_test_env: WorkflowEnvironment,
-    temporal_worker: Worker,
-    test_activities: Activities,
-    mock_pi_web_api_client,
-    postgres_engine,
-):
-    """
-    End-to-end test for PI Web API Scouter workflow.
-
-    This test:
-    1. Starts the workflow with test data
-    2. Verifies PI Web API is called
-    3. Verifies data flows through CoreScouter
-    4. Verifies data is stored in PostgreSQL (schema: sientia_data, table: laborious_data)
-    5. Verifies data is cached in Redis
-    """
-    client = temporal_test_env.client
-
-    # Prepare test input
-    input_data = {
-        'model_name': 'PI Web API Scouter Test Model',
-        'model_id': '1',
-        'schedule_name': 'pi-web-api-scouter-test',
-        'model_tags': {
-            'tag1': {
-                'webid': 'webid1',
-                'aggr_function': 'avg',
-                'data_range': [0, 100],
-                'frequency': 60000,
-            },
-            'tag2': {
-                'webid': 'webid2',
-                'aggr_function': 'avg',
-                'data_range': [0, 100],
-                'frequency': 60000,
-            },
-        },
-        'trigger_laborious': False,
-        'filters': {},
-        'schema': 'sientia_data',
-        'table_name': 'laborious_data',
-        'retention_time': 3600,
-        'fill_missing_tags': False,
-        'pi_web_api_query': {
-            'endpoint': '/streamsets/recorded',
-            'period': '*-1d',
-            'max_count': 10,
-            'api_timeout': 30,
-        },
-    }
-
-    # Start workflow
-    handle = await client.start_workflow(
-        PIWebAPIScouter.run,
-        input_data,
-        id=f'test-workflow-{datetime.now().timestamp()}',
-        task_queue='test-queue',
-    )
-
-    # Wait for workflow completion
-    await handle.result()
-
-    # Verify PI Web API was called
-    mock_pi_web_api_client.get_latest_values_df.assert_called_once()
-
-    # Verify data was stored in PostgreSQL
-    inspector = inspect(postgres_engine)
-    
-    # Schema and table are created by the setup_postgres_schema_and_table fixture
-    schema_name = 'sientia_data'
-    table_name = 'laborious_data'
-    full_table_name = f"{schema_name}.{table_name}"
-    
-    # Check if table exists in the schema
-    table_exists = inspector.has_table(table_name, schema=schema_name)
-    
-    assert table_exists, f"Expected table {full_table_name} to exist in PostgreSQL"
-
-    # Verify data was inserted
-    with postgres_engine.connect() as conn:
-        result = conn.execute(text(f"SELECT COUNT(*) FROM {full_table_name}"))
-        row_count = result.scalar()
-    
-    assert row_count > 0, f"Expected data in PostgreSQL table {full_table_name}, got {row_count} rows"
-
-    # Verify data was cached in Redis
-    keys = await test_activities.redis_repository.keys('*')
-    assert len(keys) > 0, "Expected data in Redis"

e2e/test_pi_web_api_scouter_errors.py

@@ -11,7 +11,7 @@ from temporalio.testing import WorkflowEnvironment
 from temporalio.worker import Worker
 
 from scouter.activities.activities import Activities
-from scouter.utils.clients.pi_web_api_client import PIMSRequestError
+from sientia_do.repository.pi_web_api_client import PIMSRequestError
 from scouter.workflow.pi_web_api_scouter import PIWebAPIScouter
 
 

pyproject.toml

@@ -116,6 +116,7 @@ addopts = [
     "--strict-markers",
 ]
 asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
 markers = [
     "asyncio: marks tests as async",
     "integration: marks tests as integration tests",

requirements.txt

@@ -3,6 +3,6 @@ psycopg2-binary
 sqlalchemy
 redis
 pymongo
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.7.1
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.8.2
 prometheus-client
 pycurl

scouter/activities/api.py

@@ -9,10 +9,9 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
     from sientia_do.observability.sientia_monitoring import SientiaMonitoring
+    from sientia_do.repository.pi_web_api_client import PIWebAPIClient
     from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ
 
-    from scouter.utils.clients.pi_web_api_client import PIWebAPIClient
-
 
 class API(SientiaMonitoring):
     """
@@ -64,6 +63,10 @@ class API(SientiaMonitoring):
     def close(self) -> None:
         """
         Close the PI Web API client and shutdown monitoring services.
+
+        This method performs cleanup operations:
+        - Closes the PI Web API client connection
+        - Shuts down SientiaMonitoring services (metrics, notifications)
         """
         self.pi_web_api_client.close()
         SientiaMonitoring.shutdown(self)
@@ -77,6 +80,11 @@ class API(SientiaMonitoring):
         for a set of configured tags. It returns the data as a list of dictionaries
         suitable for further processing in the workflow.
 
+        The timestamps are normalized to ensure consistency across all records in the
+        response. After converting timestamps to string format, all timestamps are
+        set to the maximum timestamp value (lexicographically) found in the dataset.
+        This ensures all records in a single batch share the same timestamp value.
+
         Args:
             input_data (dict[str, Any]): Activity input parameters.
                 Required fields:
@@ -89,7 +97,7 @@ class API(SientiaMonitoring):
 
         Returns:
             list[dict]: List of data records, each containing:
-                - timestamp: Data point timestamp
+                - timestamp: Normalized timestamp string (all records share the same value)
                 - name: Tag name
                 - value: Numeric value
                 - tag: WebId
@@ -102,6 +110,7 @@ class API(SientiaMonitoring):
         endpoint = input_data['endpoint']
         web_ids = input_data['web_ids']
         period = input_data['period']
+        end_time = input_data.get('end_time', '*')
         max_count = input_data.get('max_count', 1)
         api_timeout = input_data['api_timeout']
 
@@ -112,9 +121,10 @@ class API(SientiaMonitoring):
                 endpoint=endpoint,
                 web_ids=web_ids,
                 start_time=period,
+                end_time=end_time,
                 max_count=max_count,
                 metadata=metadata,
-                timeout=api_timeout,
+                request_timeout=api_timeout,
             )
 
         except Exception as e:
@@ -130,10 +140,12 @@ class API(SientiaMonitoring):
 
         latest_values['timestamp'] = latest_values['timestamp'].dt.strftime(DATETIME_FORMAT_WITH_TZ)
 
-        # Normalize the package timestamp
-        latest_values['timestamp'] = latest_values['timestamp'].max()
-
         self.debug(f'Latest values: {latest_values.to_string()}', metadata=metadata)
+
+        # Normalize the package timestamp
+        valid_timestamp_values = latest_values['timestamp'].dropna()
+        latest_values['timestamp'] = valid_timestamp_values.max()
+
         self.info(f'Gathered {len(latest_values)} tag values', metadata=metadata)
 
         return latest_values.to_dict(orient='records')

scouter/metrics.py

@@ -1,5 +1,4 @@
-from prometheus_client import Counter, Gauge, Histogram
+from prometheus_client import Counter, Gauge
-from sientia_do.observability.metrics import CORE_LABELS as SIENTIA_CORE_LABELS
 
 # Application health and status metrics
 APP_UP = Gauge(
@@ -24,23 +23,3 @@ TAG_CHANGES_MONITOR = Gauge(
     'Current value change of each tag',
     [*CORE_LABELS, 'tag_name'],
 )
-
-
-# Generic REST client metrics
-GENERIC_REST_CLIENT_LAG = Histogram(
-    'scouter_generic_rest_client_lag',
-    'Lag time for a request to a generic REST client',
-    SIENTIA_CORE_LABELS,
-)
-
-GENERIC_REST_READ_COUNT = Counter(
-    'scouter_generic_rest_client_read_count',
-    'Number of reads from a generic REST client',
-    SIENTIA_CORE_LABELS,
-)
-
-GENERIC_REST_READ_ERROR_COUNT = Counter(
-    'scouter_generic_rest_client_read_error_count',
-    'Number of read errors from a generic REST client',
-    SIENTIA_CORE_LABELS,
-)

scouter/utils/clients/pi_web_api_client.py

@@ -1,340 +0,0 @@
-import io
-import json
-import time
-import warnings
-from typing import Any
-from urllib.parse import urlencode
-
-import pandas as pd
-import pycurl
-from sientia_do.notifications.handlers import NotificationHandler
-from sientia_do.observability.logger import Logger
-from sientia_do.observability.metrics_controller import MetricsController
-from sientia_do.observability.sientia_monitoring import SientiaMonitoring
-
-from scouter import metrics
-
-warnings.simplefilter('ignore')  # Ignore warnings such as 'verify=False'
-
-
-class PIMSRequestError(Exception):
-    """Generic error for failed requests to PI Web API using pycurl."""
-
-    pass
-
-
-class PIWebAPIClient(SientiaMonitoring):
-    """
-    Client for interacting with the PI Web API.
-
-    This class provides a robust interface for querying historical and real-time
-    data from OSIsoft PI systems through the PI Web API. It implements:
-    - Asynchronous HTTP requests using pycurl
-    - Authentication support (Basic and Bearer)
-    - Automatic data normalization and timestamp handling
-    - Comprehensive error handling and monitoring
-    - Metrics collection for observability
-
-    The client is designed for high-performance data retrieval with proper
-    connection management and error recovery mechanisms.
-    """
-
-    def __init__(
-        self,
-        base_url: str,
-        auth_config: dict[str, Any],
-        logger: Logger,
-        notification_handler: NotificationHandler,
-        metrics_controller: MetricsController,
-        headers_config: dict[str, Any] | None = None,
-    ) -> None:
-        """
-        Initialize the PI Web API client with connection parameters.
-
-        Args:
-            base_url (str): Base URL of the PI Web API server
-            auth_config (dict[str, Any]): Authentication configuration.
-                Required fields:
-                - type (str): Authentication type ('basic' or 'bearer')
-                - token (str): Authentication token
-            logger (Logger): Logger instance for operation logging
-            notification_handler (NotificationHandler): Handler for system notifications
-            metrics_controller (MetricsController): Controller for metrics collection
-            headers_config (dict[str, Any], optional): HTTP headers configuration.
-                Default headers include content-type, accept, and x-requested-with
-            max_concurrency (int, optional): Maximum number of concurrent requests. Defaults to 8
-        """
-        if headers_config is None:
-            headers_config = {
-                'Content-Type': 'application/json',
-                'Accept': 'application/json',
-                'x-requested-with': 'XMLHttpRequest',
-            }
-        SientiaMonitoring.__init__(self, logger, notification_handler, metrics_controller)
-
-        self.base_url = base_url.rstrip('/')
-
-        # auth_config spec:
-        #     'type': 'basic' or 'bearer',
-        #     'token': 'token',
-        self.auth_config = auth_config
-        self.auth_config['type'] = self.auth_config['type'].lower()
-        self.headers: dict[str, str] = headers_config
-
-        self.authenticate()
-
-    def close(self) -> None:
-        """
-        Close the client and shutdown monitoring services.
-        """
-        SientiaMonitoring.shutdown(self)
-
-    def _to_clean_timestamp(self, series: pd.Series) -> pd.Series:
-        """
-        Convert a Series of timestamps to datetime, UTC, and round to the nearest second.
-
-        Args:
-            series (pd.Series): Series containing timestamp values
-
-        Returns:
-            pd.Series: Cleaned timestamp series in UTC, floored to seconds
-        """
-        series = pd.to_datetime(series, utc=True, errors='coerce')
-        return series.dt.floor('s')
-
-    def _extract_numeric(self, value: Any) -> float | None:
-        """
-        Normalize a value (potentially nested) to float.
-
-        This method handles PI Web API response values that may be nested
-        in dictionaries or other structures, extracting the numeric value.
-
-        Args:
-            value (Any): Value to extract and normalize
-
-        Returns:
-            float | None: Numeric value as float, or None if conversion fails
-        """
-        if isinstance(value, dict):
-            value = value.get('Value', value)
-        return pd.to_numeric(value, errors='coerce')
-
-    def authenticate(self):
-        """
-        Configure authentication headers based on auth_config.
-
-        This method sets up the Authorization header using either Basic or Bearer
-        authentication based on the configured authentication type.
-
-        Raises:
-            ValueError: If authentication type is not 'basic' or 'bearer'
-        """
-        self.logger.info(f'Authenticating with {self.auth_config["type"]} authentication')
-
-        if self.auth_config['type'] == 'basic':
-            self.headers['Authorization'] = f'Basic {self.auth_config["token"]}'
-        elif self.auth_config['type'] == 'bearer':
-            self.headers['Authorization'] = f'Bearer {self.auth_config["token"]}'
-        else:
-            raise ValueError(f'Invalid authentication type: {self.auth_config["type"]}')
-
-    async def _curl_get_json(
-        self,
-        url: str,
-        params: list[tuple[str, str]] | None = None,
-        timeout: int = 30,
-        verify: bool = True,
-        metadata: dict[str, Any] | None = None,
-    ) -> dict[str, Any]:
-        """
-        Perform a GET request using pycurl and return the decoded JSON response.
-
-        This method executes an asynchronous HTTP GET request with proper error handling,
-        metrics collection, and timeout management. It automatically tracks request
-        latency and emits monitoring metrics.
-
-        Args:
-            url (str): Target URL for the GET request
-            params (list[tuple[str, str]], optional): Query parameters as list of tuples.
-                Each tuple contains (parameter_name, parameter_value)
-            timeout (int, optional): Request timeout in seconds. Defaults to 30
-            verify (bool, optional): Verify SSL certificates. Defaults to True
-            metadata (dict[str, Any], optional): Workflow execution metadata for tracking
-
-        Returns:
-            dict[str, Any]: Parsed JSON response body
-
-        Raises:
-            PIMSRequestError: If HTTP error, connection error, or JSON parsing error occurs
-        """
-        if metadata is None:
-            metadata = {}
-
-        buffer = io.BytesIO()
-        c = pycurl.Curl()
-
-        core_labels = self.get_core_labels(metadata=metadata, operation_type='get_json')
-
-        try:
-            if params:
-                query_string = urlencode(params, doseq=True)
-                full_url = f'{url}?{query_string}'
-            else:
-                full_url = url
-
-            c.setopt(pycurl.URL, full_url.encode('utf-8'))
-            c.setopt(pycurl.WRITEDATA, buffer)
-
-            # Configure HTTP headers
-            header_list = [f'{k}: {v}' for k, v in self.headers.items()]
-            if header_list:
-                c.setopt(pycurl.HTTPHEADER, header_list)
-
-            # Set request timeout
-            c.setopt(pycurl.TIMEOUT, timeout)
-
-            # Configure SSL verification
-            if not verify:
-                c.setopt(pycurl.SSL_VERIFYPEER, 0)
-                c.setopt(pycurl.SSL_VERIFYHOST, 0)
-
-            start_time = time.time()
-            try:
-                c.perform()
-            except Exception as e:
-                await self.emit_metric(
-                    metric_object=metrics.GENERIC_REST_READ_ERROR_COUNT,
-                    tags=core_labels,
-                )
-                raise e
-            await self.observe_lag(
-                start_time=start_time,
-                metric_object=metrics.GENERIC_REST_CLIENT_LAG,
-                tags=core_labels,
-            )
-
-            status_code = c.getinfo(pycurl.RESPONSE_CODE)
-            body = buffer.getvalue().decode('utf-8', errors='replace')
-
-            if status_code >= 400:
-                await self.emit_metric(
-                    metric_object=metrics.GENERIC_REST_READ_ERROR_COUNT,
-                    tags=core_labels,
-                )
-                raise PIMSRequestError(f"HTTP {status_code} calling '{full_url}': {body[:200]}")
-
-            await self.emit_metric(
-                metric_object=metrics.GENERIC_REST_READ_COUNT,
-                tags=core_labels,
-            )
-            try:
-                return json.loads(body)
-            except json.JSONDecodeError as e:
-                raise PIMSRequestError(
-                    f"Error decoding JSON response from '{full_url}': {e}; body: {body[:200]}"
-                ) from e
-
-        except pycurl.error as e:
-            raise PIMSRequestError(f"Connection error calling '{url}': {e}") from e
-        finally:
-            c.close()
-
-    async def get_latest_values_df(
-        self,
-        web_ids: dict[str, dict[str, str]],
-        endpoint: str,
-        timeout: int = 30,
-        start_time: str = '*-1d',
-        end_time: str = '*',
-        max_count: int | None = 1,
-        metadata: dict[str, Any] | None = None,
-    ) -> pd.DataFrame:
-        """
-        Retrieve historical values for multiple WebIds using PI Web API streamsets.
-
-        This method queries the PI Web API's /streamsets/recorded endpoint to fetch
-        historical data for multiple tags simultaneously. It returns a normalized
-        DataFrame with timestamps, tag names, values, and WebIds.
-
-        Args:
-            web_ids (dict[str, str | None]): Dictionary mapping tag names to their WebIds.
-                None values are filtered out before querying
-            endpoint (str): PI Web API endpoint path (e.g., '/streamsets/recorded')
-            timeout (int, optional): Request timeout in seconds. Defaults to 30
-            start_time (str, optional): Start time in PI Web API format (e.g., "*-50d").
-                Defaults to "*-1d" (1 day ago)
-            end_time (str, optional): End time in PI Web API format (e.g., "*").
-                Defaults to "*" (current time)
-            max_count (int, optional): Maximum number of data points per series.
-                Defaults to 1. If None, maxCount parameter is not sent
-            metadata (dict[str, Any], optional): Workflow execution metadata for tracking
-
-        Returns:
-            pd.DataFrame: DataFrame with columns:
-                - timestamp: Cleaned timestamp (UTC, floored to seconds)
-                - name: Tag name
-                - value: Numeric value (normalized)
-                - tag: WebId of the tag
-                Returns empty DataFrame if no data is found
-
-        Raises:
-            PIMSRequestError: If API request fails or returns invalid data
-        """
-        if metadata is None:
-            metadata = {}
-
-        url = f'{self.base_url}{endpoint}'
-
-        # Build query parameters with WebIds (filtering out None values)
-        params: list[tuple[str, str]] = [('webid', web_id['webid']) for web_id in web_ids.values()]
-        # Invert startTime and endTime to get descending order (most recent first)
-        # PI Web API returns descending order when endTime < startTime
-        params.extend(
-            [
-                ('startTime', end_time),  # Use end_time as startTime (inverted)
-                ('endtime', start_time),  # Use start_time as endTime (inverted)
-                ('selectedFields', 'Items.Name;Items.Items.Timestamp;Items.Items.Value'),
-            ]
-        )
-
-        if max_count is not None:
-            params.append(('maxCount', str(max_count)))
-
-        data = await self._curl_get_json(
-            url=url,
-            params=params,
-            timeout=timeout,
-            verify=False,
-            metadata=metadata,
-        )
-
-        self.debug(f'Raw data from PI Web API: {data}', metadata=metadata)
-
-        raw_data = data.get('Items', [])
-
-        records = []
-        for entry in raw_data:
-            tag_name = entry.get('Name')
-            series_items = entry.get('Items', [])
-            for it in series_items:
-                if isinstance(it, dict) and 'Timestamp' in it and 'Value' in it:
-                    ts = it.get('Timestamp')
-                    val = it.get('Value')
-                    web_id = web_ids[tag_name]['webid']
-                    if ts is not None:
-                        records.append(
-                            {
-                                'timestamp': ts,
-                                'name': tag_name,
-                                'value': self._extract_numeric(val),
-                                'tag': web_id,
-                            }
-                        )
-
-        if not records:
-            return pd.DataFrame()
-
-        df = pd.DataFrame.from_records(records)
-        df['timestamp'] = self._to_clean_timestamp(df['timestamp'])
-
-        return df

scouter/utils/connectors_config.py

@@ -2,31 +2,6 @@ from os import getenv
 from typing import Any
 
 
-def build_postgres_config() -> dict[str, Any]:
-    """
-    Build PostgreSQL connection configuration from environment variables.
-
-    Returns:
-        dict[str, Any]: PostgreSQL configuration dictionary with keys:
-            - host: Database hostname (default: localhost)
-            - port: Database port (default: 5432)
-            - user: Database username (default: sientia)
-            - password: Database password (default: sientia)
-            - dbname: Database name (default: sientia)
-            - min_connections: Minimum connection pool size (default: 5)
-            - max_connections: Maximum connection pool size (default: 20)
-    """
-    return {
-        'host': getenv('POSTGRES_HOST', 'localhost'),
-        'port': int(getenv('POSTGRES_PORT', '5432')),
-        'user': getenv('POSTGRES_USER', 'sientia'),
-        'password': getenv('POSTGRES_PASSWORD', 'sientia'),
-        'dbname': getenv('POSTGRES_DBNAME', 'sientia'),
-        'min_connections': int(getenv('POSTGRES_MIN_CONNECTIONS', '5')),
-        'max_connections': int(getenv('POSTGRES_MAX_CONNECTIONS', '20')),
-    }
-
-
 def build_kafka_config() -> dict[str, Any]:
     """
     Build Kafka configuration from environment variables.
@@ -42,74 +17,3 @@ def build_kafka_config() -> dict[str, Any]:
         'polling_time': int(getenv('KAFKA_POLLING_TIME', '1000')),
         'group_id': 'scouter-group',
     }
-
-
-def build_redis_config() -> dict[str, Any]:
-    """
-    Build Redis connection configuration from environment variables.
-
-    Returns:
-        dict[str, Any]: Redis configuration dictionary with keys:
-            - host: Redis server hostname (default: localhost)
-            - port: Redis server port (default: 6379)
-            - username: Redis authentication username (default: None)
-            - password: Redis authentication password (default: None)
-    """
-    return {
-        'host': getenv('REDIS_HOST', 'localhost'),
-        'port': int(getenv('REDIS_PORT', '6379')),
-        'username': getenv('REDIS_USERNAME', None),
-        'password': getenv('REDIS_PASSWORD', None),
-    }
-
-
-def build_mongodb_config() -> dict[str, Any]:
-    """
-    Build MongoDB connection configuration from environment variables.
-
-    Returns:
-        dict[str, Any]: MongoDB configuration dictionary with keys:
-            - connection_string: Complete MongoDB connection URI
-            - database_name: Target database name (default: sientia)
-    """
-    username = getenv('MONGODB_USERNAME', 'sientia')
-    password = getenv('MONGODB_PASSWORD', 'sientia')
-    uri = getenv('MONGODB_URL', 'localhost:27017')
-
-    connection_string = f'mongodb://{username}:{password}@{uri}'
-    return {
-        'connection_string': connection_string,
-        'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
-    }
-
-
-def build_api_config() -> dict[str, Any]:
-    """
-    Build API connection configuration from environment variables.
-
-    Returns:
-        dict[str, Any]: API configuration dictionary with keys:
-            - base_url: API base URL (default: https://pi.example.com)
-            - auth_type: API authentication type (default: basic)
-            - auth_token: API authentication token (default: None)
-    """
-    return {
-        'base_url': getenv('PI_WEB_API_BASE_URL', 'https://pi.example.com'),
-        'auth_type': getenv('PI_WEB_API_AUTH_TYPE', 'basic'),
-        'auth_token': getenv('PI_WEB_API_AUTH_TOKEN', None),
-    }
-
-
-def build_druid_config() -> dict[str, Any]:
-    """
-    Build Apache Druid connection configuration from environment variables.
-
-    Returns:
-        dict[str, Any]: Druid configuration dictionary with keys:
-            - host: Druid server hostname (default: localhost)
-            - port: Druid server port (default: 8082)
-    """
-    return {
-        'host': getenv('DRUID_HOST', 'localhost'),
-        'port': int(getenv('DRUID_PORT', '8082')),
-    }

scouter/worker/prepare_worker.py

@@ -7,17 +7,19 @@ from sientia_do.observability.logger import Logger
 from temporalio.client import Client
 from temporalio.worker import PollerBehaviorAutoscaling, Worker
 
+# Worker configuration parameters with default values
+# See worker_parameters.md for detailed documentation
 parameters = [
     ('MAX_CONCURRENT_WORKFLOW_TASKS', '200'),
     ('MAX_CONCURRENT_ACTIVITIES', '200'),
     ('MAX_CONCURRENT_LOCAL_ACTIVITIES', '200'),
     ('MAX_CACHED_WORKFLOWS', '200'),
-    ('WORKFLOW_POLLER_BEHAVIUR_MINIMUM', '10'),
+    ('WORKFLOW_POLLER_BEHAVIOUR_MINIMUM', '10'),
-    ('WORKFLOW_POLLER_BEHAVIUR_INITIAL', '100'),
+    ('WORKFLOW_POLLER_BEHAVIOUR_INITIAL', '100'),
-    ('WORKFLOW_POLLER_BEHAVIUR_MAXIMUM', '200'),
+    ('WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM', '200'),
-    ('ACTIVITY_POLLER_BEHAVIUR_MINIMUM', '10'),
+    ('ACTIVITY_POLLER_BEHAVIOUR_MINIMUM', '10'),
-    ('ACTIVITY_POLLER_BEHAVIUR_INITIAL', '100'),
+    ('ACTIVITY_POLLER_BEHAVIOUR_INITIAL', '100'),
-    ('ACTIVITY_POLLER_BEHAVIUR_MAXIMUM', '200'),
+    ('ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM', '200'),
 ]
 
 
@@ -60,13 +62,13 @@ def prepare_worker(
         ],
         max_cached_workflows=local_workflow_parameters['MAX_CACHED_WORKFLOWS'],
         workflow_task_poller_behavior=PollerBehaviorAutoscaling(
-            minimum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIUR_MINIMUM'],
+            minimum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MINIMUM'],
-            initial=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIUR_INITIAL'],
+            initial=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_INITIAL'],
-            maximum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIUR_MAXIMUM'],
+            maximum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM'],
         ),
         activity_task_poller_behavior=PollerBehaviorAutoscaling(
-            minimum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIUR_MINIMUM'],
+            minimum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MINIMUM'],
-            initial=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIUR_INITIAL'],
+            initial=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_INITIAL'],
-            maximum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIUR_MAXIMUM'],
+            maximum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM'],
         ),
     )

scouter/worker/worker.py

@@ -1,8 +1,6 @@
 from temporalio import client, workflow
 from temporalio.runtime import PrometheusConfig, Runtime, TelemetryConfig
 
-from scouter.worker.prepare_worker import prepare_worker
-
 with workflow.unsafe.imports_passed_through():
     import asyncio
     import os
@@ -11,15 +9,16 @@ with workflow.unsafe.imports_passed_through():
     from prometheus_client import start_http_server
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.observability.logger import get_logger
-
+    from sientia_do.utils.connectors_config import (
-    from scouter import metrics
-    from scouter.activities.activities import Activities
-    from scouter.utils.connectors_config import (
         build_api_config,
         build_mongodb_config,
         build_postgres_config,
         build_redis_config,
     )
+
+    from scouter import metrics
+    from scouter.activities.activities import Activities
+    from scouter.worker.prepare_worker import prepare_worker
     from scouter.workflow.pi_web_api_scouter import PIWebAPIScouter
     from scouter.workflow.scouter import Scouter
     from scouter.workflow.sub_workflows.core_scouter import CoreScouter
@@ -40,13 +39,13 @@ MAX_CACHED_WORKFLOWS = int(os.getenv('MAX_CACHED_WORKFLOWS', '200'))
 
 # Temporal docs also recommends an autoscaling policy, with agrresive limits to prioritize latency over throughput.
 
-WORKFLOW_POLLER_BEHAVIUR_MINIMUM = int(os.getenv('WORKFLOW_POLLER_BEHAVIUR_MINIMUM', '10'))
+WORKFLOW_POLLER_BEHAVIOUR_MINIMUM = int(os.getenv('WORKFLOW_POLLER_BEHAVIOUR_MINIMUM', '10'))
-WORKFLOW_POLLER_BEHAVIUR_INITIAL = int(os.getenv('WORKFLOW_POLLER_BEHAVIUR_INITIAL', '100'))
+WORKFLOW_POLLER_BEHAVIOUR_INITIAL = int(os.getenv('WORKFLOW_POLLER_BEHAVIOUR_INITIAL', '100'))
-WORKFLOW_POLLER_BEHAVIUR_MAXIMUM = int(os.getenv('WORKFLOW_POLLER_BEHAVIUR_MAXIMUM', '200'))
+WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM = int(os.getenv('WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM', '200'))
 
-ACTIVITY_POLLER_BEHAVIUR_MINIMUM = int(os.getenv('ACTIVITY_POLLER_BEHAVIUR_MINIMUM', '10'))
+ACTIVITY_POLLER_BEHAVIOUR_MINIMUM = int(os.getenv('ACTIVITY_POLLER_BEHAVIOUR_MINIMUM', '10'))
-ACTIVITY_POLLER_BEHAVIUR_INITIAL = int(os.getenv('ACTIVITY_POLLER_BEHAVIUR_INITIAL', '100'))
+ACTIVITY_POLLER_BEHAVIOUR_INITIAL = int(os.getenv('ACTIVITY_POLLER_BEHAVIOUR_INITIAL', '100'))
-ACTIVITY_POLLER_BEHAVIUR_MAXIMUM = int(os.getenv('ACTIVITY_POLLER_BEHAVIUR_MAXIMUM', '200'))
+ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM = int(os.getenv('ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM', '200'))
 
 
 async def main():

scouter/worker/worker_parameters.md

@@ -0,0 +1,113 @@
+# Worker Parameters Documentation
+
+This document explains each configuration parameter used in the `prepare_worker.py` file for configuring Temporal workers.
+
+## Overview
+
+All parameters can be configured via environment variables using the pattern: `{WORKFLOW_NAME}_{PARAMETER_NAME}`. If not set, default values are used as specified below.
+
+## Concurrency Parameters
+
+### MAX_CONCURRENT_WORKFLOW_TASKS
+- **Default**: `200`
+- **Description**: Maximum number of concurrent workflow tasks that can be processed simultaneously by the worker. This controls how many workflow executions can be actively running at the same time.
+- **Usage**: Set via `max_concurrent_workflow_tasks` in the Worker configuration.
+- **Impact**: Higher values allow more workflows to run concurrently but consume more resources. Lower values provide better resource control but may limit throughput.
+
+### MAX_CONCURRENT_ACTIVITIES
+- **Default**: `200`
+- **Description**: Maximum number of concurrent activity tasks that can be executed simultaneously by the worker. Activities are the actual work units that perform business logic.
+- **Usage**: Set via `max_concurrent_activities` in the Worker configuration.
+- **Impact**: Controls the parallelism of activity execution. Higher values increase throughput but require more system resources (CPU, memory, network connections).
+
+### MAX_CONCURRENT_LOCAL_ACTIVITIES
+- **Default**: `200`
+- **Description**: Maximum number of concurrent local activity tasks that can be executed simultaneously. Local activities run in the same process as the workflow, without requiring a separate activity worker.
+- **Usage**: Set via `max_concurrent_local_activities` in the Worker configuration.
+- **Impact**: Similar to regular activities, but local activities have lower latency and overhead since they don't require network round-trips. Useful for lightweight operations.
+
+## Caching Parameters
+
+### MAX_CACHED_WORKFLOWS
+- **Default**: `200`
+- **Description**: Maximum number of workflow instances that can be cached in memory by the worker. Cached workflows allow faster resumption of execution without reloading state.
+- **Usage**: Set via `max_cached_workflows` in the Worker configuration.
+- **Impact**: Higher values improve performance for frequently accessed workflows but consume more memory. Lower values reduce memory usage but may require more frequent state reloads.
+
+## Understanding Pollers in Temporal
+
+**Pollers** are components of Temporal Workers that continuously request tasks from the Temporal service's Task Queues via synchronous RPCs. There are separate pollers for workflow tasks and activity tasks.
+
+### How Pollers Work
+
+Pollers send requests to the Temporal service to retrieve tasks from Task Queues. When a task is available, the poller retrieves it and the Worker processes it using registered Workflow or Activity handlers. This architecture provides:
+- **Load Balancing**: Workers only poll when they have capacity, distributing load across multiple processes
+- **Fault Tolerance**: Tasks persist in queues if a Worker fails, allowing recovery
+- **Task Routing**: Tasks can be routed to specific Worker processes
+
+### Autoscaling Poller Behavior
+
+Temporal supports autoscaling that dynamically adjusts the number of concurrent pollers based on workload. The system scales up during high load and down during low load, maintaining a baseline for responsiveness. Autoscaling is configured with `minimum`, `initial`, and `maximum` parameters that define the scaling bounds.
+
+## Workflow Poller Behavior (Autoscaling)
+
+These parameters control the autoscaling behavior of the workflow task poller, which retrieves workflow tasks from the Temporal server.
+
+### WORKFLOW_POLLER_BEHAVIOUR_MINIMUM
+- **Default**: `10`
+- **Description**: Minimum number of concurrent pollers for workflow tasks. The poller count will never go below this value.
+- **Usage**: Set via `minimum` in `PollerBehaviorAutoscaling` for `workflow_task_poller_behavior`.
+- **Impact**: Ensures a baseline level of polling activity even during low load periods.
+
+### WORKFLOW_POLLER_BEHAVIOUR_INITIAL
+- **Default**: `100`
+- **Description**: Initial number of concurrent pollers for workflow tasks when the worker starts.
+- **Usage**: Set via `initial` in `PollerBehaviorAutoscaling` for `workflow_task_poller_behavior`.
+- **Impact**: Determines the starting point for poller scaling. Higher values provide faster initial task acquisition but consume more resources.
+
+### WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM
+- **Default**: `200`
+- **Description**: Maximum number of concurrent pollers allowed for workflow tasks. The poller count will not exceed this value even under high load.
+- **Usage**: Set via `maximum` in `PollerBehaviorAutoscaling` for `workflow_task_poller_behavior`.
+- **Impact**: Caps the resource consumption for workflow task polling. Prevents excessive polling that could overwhelm the Temporal server or worker.
+
+## Activity Poller Behavior (Autoscaling)
+
+These parameters control the autoscaling behavior of the activity task poller, which retrieves activity tasks from the Temporal server.
+
+### ACTIVITY_POLLER_BEHAVIOUR_MINIMUM
+- **Default**: `10`
+- **Description**: Minimum number of concurrent pollers for activity tasks. The poller count will never go below this value.
+- **Usage**: Set via `minimum` in `PollerBehaviorAutoscaling` for `activity_task_poller_behavior`.
+- **Impact**: Ensures a baseline level of polling activity even during low load periods.
+
+### ACTIVITY_POLLER_BEHAVIOUR_INITIAL
+- **Default**: `100`
+- **Description**: Initial number of concurrent pollers for activity tasks when the worker starts.
+- **Usage**: Set via `initial` in `PollerBehaviorAutoscaling` for `activity_task_poller_behavior`.
+- **Impact**: Determines the starting point for poller scaling. Higher values provide faster initial task acquisition but consume more resources.
+
+### ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM
+- **Default**: `200`
+- **Description**: Maximum number of concurrent pollers allowed for activity tasks. The poller count will not exceed this value even under high load.
+- **Usage**: Set via `maximum` in `PollerBehaviorAutoscaling` for `activity_task_poller_behavior`.
+- **Impact**: Caps the resource consumption for activity task polling. Prevents excessive polling that could overwhelm the Temporal server or worker.
+
+## Configuration Example
+
+To override these parameters, set environment variables using the pattern:
+```
+{WORKFLOW_NAME}_{PARAMETER_NAME}={value}
+```
+
+For example, if your workflow is named `ScouterWorkflow`:
+```bash
+SCOUTERWORKFLOW_MAX_CONCURRENT_ACTIVITIES=500
+SCOUTERWORKFLOW_WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM=300
+```
+
+## Notes
+
+- All parameter values are converted to integers before use.
+- The autoscaling poller behavior dynamically adjusts the number of pollers between the minimum and maximum values based on workload.
+- These parameters should be tuned based on your specific workload characteristics, available resources, and performance requirements.

scouter/workflow/pi_web_api_scouter.py

@@ -33,28 +33,33 @@ class PIWebAPIScouter:
 
         This method orchestrates the complete data ingestion process from PI Web API:
         1. Retrieves tag values from PI Web API using configured WebIds
-        2. Validates and normalizes the retrieved data
+        2. Validates and normalizes the retrieved data (timestamps are normalized)
         3. Delegates data processing to the CoreScouter workflow
 
+        If no data is retrieved from the PI Web API, the workflow exits early without
+        invoking the CoreScouter workflow.
+
         Args:
             input_data (dict[str, Any]): Configuration and parameters for the workflow execution.
                 Required fields:
+                - schedule_name (str): Unique identifier for the data collection schedule
                 - model_name (str): Name of the data model being processed
                 - model_id (str): Unique identifier for the data model
-                - schedule_name (str): Unique identifier for the data collection schedule
+                - pi_web_api_query (dict[str, Any]): PI Web API query configuration containing:
-                - endpoint (str): PI Web API endpoint path (e.g., '/streamsets/recorded')
+                    - endpoint (str): PI Web API endpoint path (e.g., '/streamsets/recorded')
-                - web_ids (dict[str, str | None]): Mapping of tag names to WebIds
+                    - period (str): Time period configuration (e.g., '*-1d', '*-1h')
-                - period (dict[str, str]): Time period configuration with 'start_time'
+                    - api_timeout (int): Request timeout in seconds for PI Web API calls
-                - api_timeout (int): Request timeout in seconds for PI Web API calls
+                    - max_count (int, optional): Maximum data points per tag. Defaults to 1
-                - max_count (int, optional): Maximum data points per tag. Defaults to 1
                 - trigger_laborious (bool): Flag to enable intensive data processing
                 - filters (dict[str, str]): Data quality filters configuration
                 - schema (str): Target database schema for data export
                 - table_name (str): Target table name for data export
                 - retention_time (int): Data retention period in Redis (seconds)
-                - model_tags (dict[str, Any]): Tag-specific configuration including:
+                - model_tags (dict[str, Any]): Tag-specific configuration mapping tag names
+                  to WebIds and processing rules, including:
+                    - webid (str): PI Web API WebId for the tag
                     - data_range: [min, max] values for data validation
-                    - aggr_function: Aggregation method (avg, mdn, max, min, lts)
+                    - aggr_func: Aggregation method (avg, mdn, max, min, lts)
                     - frequency: Data collection frequency in milliseconds
                     - topics: List of Kafka topics for data routing
 

scouter/workflow/scouter.py

@@ -32,10 +32,14 @@ class Scouter:
 
         This method orchestrates the complete data ingestion process:
         1. Retrieves the last processed timestamp from Redis
-        2. Loads new data from MongoDB since the last timestamp
+        2. Loads new data from MongoDB since the last timestamp using collection name
-        3. Updates the last processed timestamp
+           format: `raw_{schedule_name}`
+        3. Updates the last processed timestamp with the most recent data point
         4. Delegates data processing to the CoreScouter workflow
 
+        If no new data is found in MongoDB, the workflow exits early without updating
+        the timestamp or invoking the CoreScouter workflow.
+
         Args:
             input_data (dict[str, Any]): Configuration and parameters for the workflow execution.
                 Required fields:

scouter/workflow/sub_workflows/core_scouter.py

@@ -23,7 +23,8 @@ class CoreScouter:
     - Metrics collection and monitoring
 
     The workflow is designed for high-throughput data processing with configurable
-    quality gates and aggregation strategies.
+    quality gates and aggregation strategies. It is typically invoked as a child
+    workflow by parent workflows such as Scouter or PIWebAPIScouter.
     """
 
     @workflow.run
@@ -35,9 +36,14 @@ class CoreScouter:
         1. Data Quality Gate: Applies configurable filters for data validation
         2. Data Aggregation: Groups and aggregates data using specified functions
         3. Data Grouping: Organizes data by tags and applies retention policies
-        4. Data Export: Persists processed data to PostgreSQL
+        4. Data Export: Persists processed data to PostgreSQL with timestamp conversion
         5. Metrics Collection: Records processing metrics for monitoring
 
+        The workflow implements early exit conditions:
+        - If held_data is empty after grouping, the workflow exits without exporting
+        - If data export results in zero or negative affected_rows, the workflow exits
+          without writing metrics or storing debug packages
+
         Args:
             input_data (dict[str, Any]): Complete workflow configuration and data.
                 Required fields:
@@ -53,6 +59,9 @@ class CoreScouter:
                 - table_name (str): Target database table
                 - retention_time (int): Redis data retention period (seconds)
                 - model_tags (dict[str, Any]): Tag-specific processing rules
+                - fill_missing_tags (bool): Enable filling of missing tag values
+                - debug_data_package (bool, optional): Store data packages for debugging.
+                  When True, stores both raw and processed data in MongoDB for debugging
 
         Returns:
             None: This workflow processes data but doesn't return results
@@ -110,8 +119,6 @@ class CoreScouter:
                 'table_name': input_data['table_name'],
                 'data': held_data,
                 'timestamp_conversion': {'column': 'timestamp', 'format': DATETIME_FORMAT_WITH_TZ},
-                'on_conflict': 'ignore',
-                'unique_columns': ['model_id', 'timestamp', 'variable'],
             },
             retry_policy=retry_policy,
             start_to_close_timeout=timedelta(seconds=60),

tests/activities/test_api.py

@@ -139,9 +139,10 @@ async def test_get_tag_values_success(api_activity):
             'tag3': {'webid': 'webid3', 'aggr_function': 'avg', 'data_range': [0, 100]},
         },
         start_time='*-1d',
+        end_time='*',
         max_count=10,
         metadata=metadata['metadata'],
-        timeout=30,
+        request_timeout=30,
     )
 
     assert len(result) == 3
@@ -193,9 +194,10 @@ async def test_get_tag_values_with_default_max_count(api_activity):
         endpoint='/streamsets/recorded',
         web_ids={'tag1': {'webid': 'webid1', 'aggr_function': 'avg', 'data_range': [0, 100]}},
         start_time='*-1h',
+        end_time='*',
         max_count=1,
         metadata=metadata['metadata'],
-        timeout=15,
+        request_timeout=15,
     )
 
     assert len(result) == 1

tests/utils/clients/test_pi_web_api_client.py

@@ -1,588 +0,0 @@
-import json
-from unittest.mock import AsyncMock, MagicMock, patch
-
-import pandas as pd
-import pycurl
-import pytest
-
-from scouter.utils.clients.pi_web_api_client import PIMSRequestError, PIWebAPIClient
-
-
-@pytest.fixture
-def mock_logger():
-    return MagicMock()
-
-
-@pytest.fixture
-def mock_notification_handler():
-    return AsyncMock()
-
-
-@pytest.fixture
-def mock_metrics_controller():
-    return AsyncMock()
-
-
-@pytest.fixture
-def auth_config_basic():
-    return {'type': 'basic', 'token': 'test_token_123'}
-
-
-@pytest.fixture
-def auth_config_bearer():
-    return {'type': 'bearer', 'token': 'bearer_token_456'}
-
-
-@pytest.fixture
-def pi_client(mock_logger, mock_notification_handler, mock_metrics_controller, auth_config_basic):
-    return PIWebAPIClient(
-        base_url='https://pi.example.com',
-        auth_config=auth_config_basic,
-        logger=mock_logger,
-        notification_handler=mock_notification_handler,
-        metrics_controller=mock_metrics_controller,
-    )
-
-
-def test_init_with_basic_auth(
-    mock_logger, mock_notification_handler, mock_metrics_controller, auth_config_basic
-):
-    """Test initialization with basic authentication"""
-    client = PIWebAPIClient(
-        base_url='https://pi.example.com/',
-        auth_config=auth_config_basic,
-        logger=mock_logger,
-        notification_handler=mock_notification_handler,
-        metrics_controller=mock_metrics_controller,
-    )
-
-    assert client.base_url == 'https://pi.example.com'
-    assert client.auth_config['type'] == 'basic'
-    assert client.headers['Authorization'] == 'Basic test_token_123'
-    assert client.headers['Content-Type'] == 'application/json'
-    assert client.headers['Accept'] == 'application/json'
-    mock_logger.info.assert_called_with('Authenticating with basic authentication')
-
-
-def test_init_with_bearer_auth(
-    mock_logger, mock_notification_handler, mock_metrics_controller, auth_config_bearer
-):
-    """Test initialization with bearer authentication"""
-    client = PIWebAPIClient(
-        base_url='https://pi.example.com',
-        auth_config=auth_config_bearer,
-        logger=mock_logger,
-        notification_handler=mock_notification_handler,
-        metrics_controller=mock_metrics_controller,
-    )
-
-    assert client.base_url == 'https://pi.example.com'
-    assert client.auth_config['type'] == 'bearer'
-    assert client.headers['Authorization'] == 'Bearer bearer_token_456'
-    mock_logger.info.assert_called_with('Authenticating with bearer authentication')
-
-
-def test_init_with_custom_headers(
-    mock_logger, mock_notification_handler, mock_metrics_controller, auth_config_basic
-):
-    """Test initialization with custom headers"""
-    custom_headers = {
-        'Content-Type': 'application/xml',
-        'Custom-Header': 'custom_value',
-    }
-
-    client = PIWebAPIClient(
-        base_url='https://pi.example.com',
-        auth_config=auth_config_basic,
-        logger=mock_logger,
-        notification_handler=mock_notification_handler,
-        metrics_controller=mock_metrics_controller,
-        headers_config=custom_headers,
-    )
-
-    assert client.headers['Content-Type'] == 'application/xml'
-    assert client.headers['Custom-Header'] == 'custom_value'
-    assert client.headers['Authorization'] == 'Basic test_token_123'
-
-
-def test_authenticate_invalid_type(mock_logger, mock_notification_handler, mock_metrics_controller):
-    """Test that invalid authentication type raises ValueError"""
-    invalid_auth_config = {'type': 'invalid', 'token': 'test_token'}
-
-    with pytest.raises(ValueError) as exc_info:
-        PIWebAPIClient(
-            base_url='https://pi.example.com',
-            auth_config=invalid_auth_config,
-            logger=mock_logger,
-            notification_handler=mock_notification_handler,
-            metrics_controller=mock_metrics_controller,
-        )
-
-    assert 'Invalid authentication type: invalid' in str(exc_info.value)
-
-
-@patch('scouter.utils.clients.pi_web_api_client.SientiaMonitoring.shutdown')
-def test_close(mock_shutdown, pi_client):
-    """Test close method calls shutdown"""
-    pi_client.close()
-
-    mock_shutdown.assert_called_once()
-
-
-def test_to_clean_timestamp(pi_client):
-    """Test timestamp cleaning and normalization"""
-    timestamps = pd.Series(
-        [
-            '2025-01-15T10:30:45.123456Z',
-            '2025-01-15T10:30:46.789012Z',
-            '2025-01-15T10:30:47.999999Z',
-        ]
-    )
-
-    result = pi_client._to_clean_timestamp(timestamps)
-
-    assert isinstance(result, pd.Series)
-    assert result.dtype == 'datetime64[ns, UTC]'
-    # Verify microseconds are floored to seconds
-    assert result[0] == pd.Timestamp('2025-01-15T10:30:45Z')
-    assert result[1] == pd.Timestamp('2025-01-15T10:30:46Z')
-    assert result[2] == pd.Timestamp('2025-01-15T10:30:47Z')
-
-
-def test_to_clean_timestamp_with_invalid_values(pi_client):
-    """Test timestamp cleaning with invalid values returns NaT"""
-    timestamps = pd.Series(['invalid', 'not_a_date', '2025-01-15T10:30:45Z'])
-
-    result = pi_client._to_clean_timestamp(timestamps)
-
-    assert pd.isna(result[0])
-    assert pd.isna(result[1])
-    assert result[2] == pd.Timestamp('2025-01-15T10:30:45Z')
-
-
-def test_extract_numeric_with_float(pi_client):
-    """Test extracting numeric value from float"""
-    result = pi_client._extract_numeric(42.5)
-
-    assert result == pytest.approx(42.5)
-
-
-def test_extract_numeric_with_int(pi_client):
-    """Test extracting numeric value from int"""
-    result = pi_client._extract_numeric(42)
-
-    assert result == pytest.approx(42.0)
-
-
-def test_extract_numeric_with_string(pi_client):
-    """Test extracting numeric value from string"""
-    result = pi_client._extract_numeric('123.45')
-
-    assert result == pytest.approx(123.45)
-
-
-def test_extract_numeric_with_dict(pi_client):
-    """Test extracting numeric value from dictionary"""
-    result = pi_client._extract_numeric({'Value': 99.9})
-
-    assert result == pytest.approx(99.9)
-
-
-def test_extract_numeric_with_invalid_value(pi_client):
-    """Test extracting numeric value from invalid value returns None/NaN"""
-    result = pi_client._extract_numeric('invalid_number')
-
-    assert pd.isna(result)
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_success(mock_curl_class, pi_client):
-    """Test successful GET request with JSON response"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    response_data = {'status': 'success', 'data': [1, 2, 3]}
-    response_json = json.dumps(response_data).encode('utf-8')
-
-    def mock_perform():
-        buffer = mock_curl.setopt.call_args_list[1][0][1]
-        buffer.write(response_json)
-
-    mock_curl.perform.side_effect = mock_perform
-    mock_curl.getinfo.return_value = 200
-
-    result = await pi_client._curl_get_json('https://pi.example.com/api/test')
-
-    assert result == response_data
-    mock_curl.setopt.assert_any_call(pycurl.TIMEOUT, 30)
-    mock_curl.perform.assert_called_once()
-    mock_curl.close.assert_called_once()
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_with_params(mock_curl_class, pi_client):
-    """Test GET request with query parameters"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    response_data = {'result': 'ok'}
-    response_json = json.dumps(response_data).encode('utf-8')
-
-    def mock_perform():
-        buffer = mock_curl.setopt.call_args_list[1][0][1]
-        buffer.write(response_json)
-
-    mock_curl.perform.side_effect = mock_perform
-    mock_curl.getinfo.return_value = 200
-
-    params = [('key1', 'value1'), ('key2', 'value2')]
-    result = await pi_client._curl_get_json('https://pi.example.com/api', params=params)
-
-    assert result == response_data
-    # Verify URL includes query parameters
-    set_url_call = [call for call in mock_curl.setopt.call_args_list if call[0][0] == pycurl.URL][0]
-    assert b'key1=value1' in set_url_call[0][1]
-    assert b'key2=value2' in set_url_call[0][1]
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_http_error(mock_curl_class, pi_client):
-    """Test GET request with HTTP error response"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    error_response = b'{"error": "Not found"}'
-
-    def mock_perform():
-        buffer = mock_curl.setopt.call_args_list[1][0][1]
-        buffer.write(error_response)
-
-    mock_curl.perform.side_effect = mock_perform
-    mock_curl.getinfo.return_value = 404
-
-    with pytest.raises(PIMSRequestError) as exc_info:
-        await pi_client._curl_get_json('https://pi.example.com/api/notfound')
-
-    assert 'HTTP 404' in str(exc_info.value)
-    mock_curl.close.assert_called_once()
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_connection_error(mock_curl_class, pi_client):
-    """Test GET request with connection error"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    mock_curl.perform.side_effect = pycurl.error('Connection failed')
-
-    with pytest.raises(PIMSRequestError) as exc_info:
-        await pi_client._curl_get_json('https://pi.example.com/api/test')
-
-    assert 'Connection error' in str(exc_info.value)
-    mock_curl.close.assert_called_once()
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_invalid_json(mock_curl_class, pi_client):
-    """Test GET request with invalid JSON response"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    invalid_json = b'This is not valid JSON'
-
-    def mock_perform():
-        buffer = mock_curl.setopt.call_args_list[1][0][1]
-        buffer.write(invalid_json)
-
-    mock_curl.perform.side_effect = mock_perform
-    mock_curl.getinfo.return_value = 200
-
-    with pytest.raises(PIMSRequestError) as exc_info:
-        await pi_client._curl_get_json('https://pi.example.com/api/test')
-
-    assert 'Error decoding JSON response' in str(exc_info.value)
-    mock_curl.close.assert_called_once()
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_with_custom_timeout(mock_curl_class, pi_client):
-    """Test GET request with custom timeout"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    response_data = {'status': 'ok'}
-    response_json = json.dumps(response_data).encode('utf-8')
-
-    def mock_perform():
-        buffer = mock_curl.setopt.call_args_list[1][0][1]
-        buffer.write(response_json)
-
-    mock_curl.perform.side_effect = mock_perform
-    mock_curl.getinfo.return_value = 200
-
-    await pi_client._curl_get_json('https://pi.example.com/api/test', timeout=60)
-
-    mock_curl.setopt.assert_any_call(pycurl.TIMEOUT, 60)
-
-
-@pytest.mark.asyncio
-@patch('scouter.utils.clients.pi_web_api_client.pycurl.Curl')
-async def test_curl_get_json_without_ssl_verify(mock_curl_class, pi_client):
-    """Test GET request with SSL verification disabled"""
-    mock_curl = MagicMock()
-    mock_curl_class.return_value = mock_curl
-
-    response_data = {'status': 'ok'}
-    response_json = json.dumps(response_data).encode('utf-8')
-
-    def mock_perform():
-        buffer = mock_curl.setopt.call_args_list[1][0][1]
-        buffer.write(response_json)
-
-    mock_curl.perform.side_effect = mock_perform
-    mock_curl.getinfo.return_value = 200
-
-    await pi_client._curl_get_json('https://pi.example.com/api/test', verify=False)
-
-    mock_curl.setopt.assert_any_call(pycurl.SSL_VERIFYPEER, 0)
-    mock_curl.setopt.assert_any_call(pycurl.SSL_VERIFYHOST, 0)
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_success(mock_curl_get_json, pi_client):
-    """Test successful retrieval of latest values"""
-    mock_curl_get_json.return_value = {
-        'Items': [
-            {
-                'Name': 'tag1',
-                'Items': [
-                    {'Timestamp': '2025-01-15T10:30:00Z', 'Value': 42.5},
-                    {'Timestamp': '2025-01-15T10:31:00Z', 'Value': 43.0},
-                ],
-            },
-            {
-                'Name': 'tag2',
-                'Items': [
-                    {'Timestamp': '2025-01-15T10:30:00Z', 'Value': 100.0},
-                ],
-            },
-        ]
-    }
-
-    web_ids = {
-        'tag1': {'webid': 'webid1'},
-        'tag2': {'webid': 'webid2'},
-    }
-
-    result = await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-        start_time='*-1d',
-        end_time='*',
-        max_count=10,
-    )
-
-    assert isinstance(result, pd.DataFrame)
-    assert len(result) == 3
-    assert list(result.columns) == ['timestamp', 'name', 'value', 'tag']
-    assert result['name'].tolist() == ['tag1', 'tag1', 'tag2']
-    assert result['value'].tolist() == [42.5, 43.0, 100.0]
-
-    mock_curl_get_json.assert_called_once()
-    call_args = mock_curl_get_json.call_args
-    assert call_args[1]['url'] == 'https://pi.example.com/streamsets/recorded'
-    assert call_args[1]['timeout'] == 30
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_with_custom_params(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df with custom parameters"""
-    mock_curl_get_json.return_value = {
-        'Items': [
-            {
-                'Name': 'tag1',
-                'Items': [
-                    {'Timestamp': '2025-01-15T10:30:00Z', 'Value': 42.5},
-                ],
-            }
-        ]
-    }
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-    metadata = {'model_id': 'test_model'}
-
-    result = await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-        start_time='*-7d',
-        end_time='*-1d',
-        max_count=100,
-        timeout=60,
-        metadata=metadata,
-    )
-
-    assert isinstance(result, pd.DataFrame)
-    assert len(result) == 1
-
-    mock_curl_get_json.assert_called_once()
-    call_args = mock_curl_get_json.call_args
-    params = call_args[1]['params']
-
-    # Verify parameters (inverted: startTime uses end_time, endTime uses start_time)
-    assert ('startTime', '*-1d') in params
-    assert ('endtime', '*-7d') in params
-    assert ('maxCount', '100') in params
-    assert call_args[1]['timeout'] == 60
-    assert call_args[1]['metadata'] == metadata
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_empty_response(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df with empty response"""
-    mock_curl_get_json.return_value = {'Items': []}
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-
-    result = await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-    )
-
-    assert isinstance(result, pd.DataFrame)
-    assert len(result) == 0
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_no_items_in_tag(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df when tag has no items"""
-    mock_curl_get_json.return_value = {
-        'Items': [
-            {
-                'Name': 'tag1',
-                'Items': [],
-            }
-        ]
-    }
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-
-    result = await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-    )
-
-    assert isinstance(result, pd.DataFrame)
-    assert len(result) == 0
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_with_missing_timestamp(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df filters out items with missing timestamp"""
-    mock_curl_get_json.return_value = {
-        'Items': [
-            {
-                'Name': 'tag1',
-                'Items': [
-                    {'Timestamp': '2025-01-15T10:30:00Z', 'Value': 42.5},
-                    {'Value': 43.0},  # Missing Timestamp
-                    {'Timestamp': None, 'Value': 44.0},  # None Timestamp
-                ],
-            }
-        ]
-    }
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-
-    result = await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-    )
-
-    assert isinstance(result, pd.DataFrame)
-    assert len(result) == 1  # Only the first item should be included
-    assert result['value'].tolist() == [42.5]
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_with_nested_value(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df with nested value extraction"""
-    mock_curl_get_json.return_value = {
-        'Items': [
-            {
-                'Name': 'tag1',
-                'Items': [
-                    {'Timestamp': '2025-01-15T10:30:00Z', 'Value': {'Value': 42.5}},
-                ],
-            }
-        ]
-    }
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-
-    result = await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-    )
-
-    assert isinstance(result, pd.DataFrame)
-    assert len(result) == 1
-    assert result['value'].tolist() == [42.5]
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_default_max_count(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df uses default max_count of 1"""
-    mock_curl_get_json.return_value = {'Items': []}
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-
-    await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-    )
-
-    call_args = mock_curl_get_json.call_args
-    params = call_args[1]['params']
-
-    assert ('maxCount', '1') in params
-    # Verify default time parameters are inverted (startTime uses end_time default, endTime uses start_time default)
-    assert ('startTime', '*') in params  # Default end_time
-    assert ('endtime', '*-1d') in params  # Default start_time
-
-
-@pytest.mark.asyncio
-@patch.object(PIWebAPIClient, '_curl_get_json', new_callable=AsyncMock)
-async def test_get_latest_values_df_with_none_max_count(mock_curl_get_json, pi_client):
-    """Test get_latest_values_df does not send maxCount parameter when max_count is None"""
-    mock_curl_get_json.return_value = {'Items': []}
-
-    web_ids = {'tag1': {'webid': 'webid1'}}
-
-    await pi_client.get_latest_values_df(
-        web_ids=web_ids,
-        endpoint='/streamsets/recorded',
-        max_count=None,
-    )
-
-    call_args = mock_curl_get_json.call_args
-    params = call_args[1]['params']
-
-    # Verify maxCount parameter is not present when max_count is None
-    assert ('maxCount', '1') not in params
-    assert ('maxCount', None) not in params
-    # Verify time parameters are still present
-    assert ('startTime', '*') in params
-    assert ('endtime', '*-1d') in params

tests/utils/test_connectors_config.py

@@ -4,12 +4,7 @@ from unittest.mock import patch
 import pytest
 
 from scouter.utils.connectors_config import (
-    build_api_config,
-    build_druid_config,
     build_kafka_config,
-    build_mongodb_config,
-    build_postgres_config,
-    build_redis_config,
 )
 
 
@@ -19,50 +14,6 @@ def mock_env_vars():
         yield
 
 
-@pytest.mark.usefixtures('mock_env_vars')
-def test_build_postgres_config_defaults():
-    """Test that build_postgres_config returns default values when no env vars are set"""
-    config = build_postgres_config()
-
-    assert config == {
-        'host': 'localhost',
-        'port': 5432,
-        'user': 'sientia',
-        'password': 'sientia',
-        'dbname': 'sientia',
-        'min_connections': 5,
-        'max_connections': 20,
-    }
-
-
-@pytest.mark.usefixtures('mock_env_vars')
-def test_build_postgres_config_with_env_vars():
-    """Test that build_postgres_config uses env vars when set"""
-    with patch.dict(
-        os.environ,
-        {
-            'POSTGRES_HOST': 'db.example.com',
-            'POSTGRES_PORT': '5433',
-            'POSTGRES_USER': 'admin',
-            'POSTGRES_PASSWORD': 'secret',
-            'POSTGRES_DBNAME': 'test_db',
-            'POSTGRES_MIN_CONNECTIONS': '3',
-            'POSTGRES_MAX_CONNECTIONS': '15',
-        },
-    ):
-        config = build_postgres_config()
-
-        assert config == {
-            'host': 'db.example.com',
-            'port': 5433,
-            'user': 'admin',
-            'password': 'secret',
-            'dbname': 'test_db',
-            'min_connections': 3,
-            'max_connections': 15,
-        }
-
-
 @pytest.mark.usefixtures('mock_env_vars')
 def test_build_kafka_config_defaults():
     """Test that build_kafka_config returns default values when no env vars are set"""
@@ -89,114 +40,3 @@ def test_build_kafka_config_with_env_vars():
             'polling_time': 5000,
             'group_id': 'scouter-group',
         }
-
-
-@pytest.mark.usefixtures('mock_env_vars')
-def test_build_redis_config_defaults():
-    """Test that build_redis_config returns default values when no env vars are set"""
-    config = build_redis_config()
-
-    assert config == {'host': 'localhost', 'port': 6379, 'username': None, 'password': None}
-
-
-@pytest.mark.usefixtures('mock_env_vars')
-def test_build_redis_config_with_env_vars():
-    """Test that build_redis_config uses env vars when set"""
-    with patch.dict(
-        os.environ,
-        {
-            'REDIS_HOST': 'redis.example.com',
-            'REDIS_PORT': '6380',
-            'REDIS_USERNAME': 'test',
-            'REDIS_PASSWORD': 'test',
-        },
-    ):
-        config = build_redis_config()
-
-        assert config == {
-            'host': 'redis.example.com',
-            'port': 6380,
-            'username': 'test',
-            'password': 'test',
-        }
-
-
-def test_build_mongodb_config_defaults():
-    """Test that build_mongodb_config returns default values when no env vars are set"""
-    os.environ['MONGODB_URL'] = 'localhost:27017'
-    os.environ['MONGODB_DATABASE_NAME'] = 'sientia'
-    os.environ['MONGODB_USERNAME'] = 'sientia'
-    os.environ['MONGODB_PASSWORD'] = 'sientia'
-
-    config = build_mongodb_config()
-
-    assert config == {
-        'connection_string': 'mongodb://sientia:sientia@localhost:27017',  # NOSONAR
-        'database_name': 'sientia',
-    }
-
-
-def test_build_mongodb_config_with_env_vars():
-    """Test that build_mongodb_config uses env vars when set"""
-    with patch.dict(
-        os.environ,
-        {
-            'MONGODB_URL': 'mongodb.example.com:27017',
-            'MONGODB_DATABASE_NAME': 'test_db',
-            'MONGODB_USERNAME': 'test',
-            'MONGODB_PASSWORD': 'test',
-        },
-    ):
-        config = build_mongodb_config()
-
-        assert config == {
-            'connection_string': 'mongodb://test:test@mongodb.example.com:27017',
-            'database_name': 'test_db',
-        }
-
-
-@pytest.mark.usefixtures('mock_env_vars')
-def test_build_api_config_defaults():
-    """Test that build_api_config returns default values when no env vars are set"""
-    config = build_api_config()
-
-    assert config == {
-        'base_url': 'https://pi.example.com',
-        'auth_type': 'basic',
-        'auth_token': None,
-    }
-
-
-@pytest.mark.usefixtures('mock_env_vars')
-def test_build_api_config_with_env_vars():
-    """Test that build_api_config uses env vars when set"""
-    with patch.dict(
-        os.environ,
-        {
-            'PI_WEB_API_BASE_URL': 'https://api.production.com',
-            'PI_WEB_API_AUTH_TYPE': 'bearer',
-            'PI_WEB_API_AUTH_TOKEN': 'secret_token_123',
-        },
-    ):
-        config = build_api_config()
-
-        assert config == {
-            'base_url': 'https://api.production.com',
-            'auth_type': 'bearer',
-            'auth_token': 'secret_token_123',
-        }
-
-
-def test_build_druid_config_defaults():
-    """Test that build_druid_config returns default values when no env vars are set"""
-    config = build_druid_config()
-
-    assert config == {'host': 'localhost', 'port': 8082}
-
-
-def test_build_druid_config_with_env_vars():
-    """Test that build_druid_config uses env vars when set"""
-    with patch.dict(os.environ, {'DRUID_HOST': 'druid.example.com', 'DRUID_PORT': '8083'}):
-        config = build_druid_config()
-
-        assert config == {'host': 'druid.example.com', 'port': 8083}

tests/workflow/sub_workflows/test_core_scouter.py

@@ -118,8 +118,6 @@ async def test_core_scouter_workflow_success(mock_workflow, core_scouter):
                         'column': 'timestamp',
                         'format': DATETIME_FORMAT_WITH_TZ,
                     },
-                    'on_conflict': 'ignore',
-                    'unique_columns': ['model_id', 'timestamp', 'variable'],
                 },
                 retry_policy=ANY,
                 start_to_close_timeout=ANY,
@@ -305,8 +303,6 @@ async def test_core_scouter_workflow_with_zero_affected_rows(mock_workflow, core
                 'column': 'timestamp',
                 'format': DATETIME_FORMAT_WITH_TZ,
             },
-            'on_conflict': 'ignore',
-            'unique_columns': ['model_id', 'timestamp', 'variable'],
         },
         retry_policy=ANY,
         start_to_close_timeout=ANY,
@@ -377,8 +373,6 @@ async def test_core_scouter_workflow_without_debug_data_package(mock_workflow, c
                         'column': 'timestamp',
                         'format': DATETIME_FORMAT_WITH_TZ,
                     },
-                    'on_conflict': 'ignore',
-                    'unique_columns': ['model_id', 'timestamp', 'variable'],
                 },
                 retry_policy=ANY,
                 start_to_close_timeout=ANY,

values.yaml

@@ -163,7 +163,7 @@ env:
   - name: GITHUB_REPO_URL
     value: "git@github.com:Aignosi/sientia-dataops-scouter_temporal.git"
   - name: GITHUB_BRANCH
-    value: "fix/SIENTIAPDE-1445"
+    value: "fix/SIENTIAPDE-1478"
   - name: PYTHON_APP
     value: "scouter.worker.worker"
 
@@ -242,18 +242,18 @@ env:
   - name: SCOUTER_MAX_CACHED_WORKFLOWS
     value: "200"
 
-  - name: SCOUTER_WORKFLOW_POLLER_BEHAVIUR_MINIMUM
+  - name: SCOUTER_WORKFLOW_POLLER_BEHAVIOUR_MINIMUM
     value: "10"
-  - name: SCOUTER_WORKFLOW_POLLER_BEHAVIUR_INITIAL
+  - name: SCOUTER_WORKFLOW_POLLER_BEHAVIOUR_INITIAL
     value: "100"
-  - name: SCOUTER_WORKFLOW_POLLER_BEHAVIUR_MAXIMUM
+  - name: SCOUTER_WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM
     value: "200"
 
-  - name: SCOUTER_ACTIVITY_POLLER_BEHAVIUR_MINIMUM
+  - name: SCOUTER_ACTIVITY_POLLER_BEHAVIOUR_MINIMUM
     value: "10"
-  - name: SCOUTER_ACTIVITY_POLLER_BEHAVIUR_INITIAL
+  - name: SCOUTER_ACTIVITY_POLLER_BEHAVIOUR_INITIAL
     value: "100"
-  - name: SCOUTER_ACTIVITY_POLLER_BEHAVIUR_MAXIMUM
+  - name: SCOUTER_ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM
     value: "200"