gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1811

Browse Source 
Enhance OPC UA communication and metrics tracking

- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
This commit is contained in:
vitor-aignosi
2026-05-15 15:28:28 -03:00
parent 473bd0b03f
commit 638d5b70b4
15 changed files with 1370 additions and 424 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
e2e/conftest.py
View File

@@ -2,7 +2,14 @@
Pytest configuration and fixtures for E2E tests.
"""
import sys
from unittest.mock import AsyncMock, MagicMock, patch
# E2E workflows under test do not run ModelAnalysis; stub before Activities import.
_model_analysis_module = MagicMock()
_model_analysis_module.ModelAnalysis = MagicMock
sys.modules.setdefault('sientia', MagicMock())
sys.modules.setdefault('sientia.ModelAnalysis', _model_analysis_module)
from io import BytesIO
import pandas as pd

13
e2e/helpers.py
View File

@@ -64,7 +64,8 @@ def assert_prediction(
prediction: float = 0.5,
prediction_confidence: int | Decimal = 0,
prediction_status: str = 'Good',
comments: str = '',
comments: str | None = None,
comments_contains: str | None = None,
) -> None:
"""
Assert exactly one prediction row exists for model_id with expected columns.
@@ -75,7 +76,8 @@ def assert_prediction(
prediction: Expected prediction value.
prediction_confidence: Expected confidence (int or Decimal for numeric column).
prediction_status: Expected status string.
comments: Expected comments string.
comments: Expected exact comments string (optional).
comments_contains: Substring expected in comments when queued (optional).
"""
import pytest
@@ -98,7 +100,12 @@ def assert_prediction(
str(prediction_confidence)
), f'Expected prediction_confidence={prediction_confidence}, got {row[2]}'
assert row[3] == prediction_status, f"Expected prediction_status='{prediction_status}', got {row[3]}"
assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
if comments is not None:
assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
if comments_contains is not None:
assert comments_contains in row[4], (
f"Expected comments to contain '{comments_contains}', got {row[4]}"
)
def assert_continue(

42
e2e/scenarios.md
View File

@@ -495,6 +495,48 @@ These paths do **not** rely on Temporal activity retries for export failures: th
---
#### Scenario 3.2.4: OPC Session / Channel Bad* (Tier-1)
**Description**: OPC write fails with a Tier-1 session or channel status (e.g. `BadSessionIdInvalid`) while transport may still appear open on the client
**Input**:
- Valid prediction and OPC output config
- Mock or server returning Tier-1 `UaStatusCodeError` on write (no write retry in the same activity)
**Expected Behavior**:
- `write_opc_data` fails forward for affected tags; background reconnect may be scheduled if `OPC_RECONNECTION_INTERVAL` allows
- Workflow **completes**
- PostgreSQL row uses **`prediction_confidence` 14** and comment prefix `OPC UA session/channel error:` (including OPC status name)
- `opc_write_attempts_total` records `result=BadSessionIdInvalid` (or matching status); no second write attempt in the same activity
**Assertions**:
- Workflow completes
- `prediction_confidence = 14`
- `comments` matches `OPC UA session/channel error:%`
- Generic OPC error confidence **12** is not used for this case
**Reference**: [docs/opc-communication.md](../docs/opc-communication.md), plan `.cursor/plans/opc_bad_reconnect_ac4c6045.plan.md`
---
#### Scenario 3.2.5: OPC Write Blocked During Reconnect
**Description**: A write is attempted while the repository is reconnecting (session not ready)
**Input**:
- Valid prediction
- Simulated slow reconnect (e.g. delayed `connect`) or concurrent writes where the first triggers reconnect
**Expected Behavior**:
- Second write (or parallel write) is rejected **immediately** when reconnect is in progress or `_session_ready` is cleared — **without** calling `write_value`
- No wait/sleep on the write path; no duplicate `connect` from parallel writers (connection lock)
- `prediction_confidence = 14`, `comments = OPC UA reconnect in progress` (distinguish from Tier-1 `Bad*` via comment prefix in SQL)
**Assertions**:
- At most one reconnect sequence (`disconnect` + `connect`) for the overlapping window
- No write retry after failure
- Tests in `test_opc_repository` (unit) and optional e2e in `test_predictions_batch_format_export.py`
---
#### Scenario 3.2.3: PI Web API Partial Write Error
**Description**: Two prediction tags attempt to be written to PI Web API, but only one succeeds

137
e2e/test_predictions_batch_format_export.py
View File

@@ -153,7 +153,6 @@ async def test_scenario_3_1_1_default_prediction_export(
'addr_1',
0,
'float',
ANY,
{
'model_id': 311,
'model_name': 'test_model',
@@ -165,7 +164,6 @@ async def test_scenario_3_1_1_default_prediction_export(
'addr_2',
2,
'float',
ANY,
{
'model_id': 311,
'model_name': 'test_model',
@@ -252,20 +250,28 @@ async def test_scenario_3_1_2_export_with_opc_only(
opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
opc_write_data.assert_has_calls(
[
call('addr_1', 0.5, 'float', ANY,
{
'model_id': 312,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call('addr_2', 0, 'float', ANY,
{
'model_id': 312,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call(
'addr_1',
0.5,
'float',
{
'model_id': 312,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
call(
'addr_2',
0,
'float',
{
'model_id': 312,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
]
)
@@ -500,20 +506,28 @@ async def test_scenario_3_1_5_export_without_transformed_data(
opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
opc_write_data.assert_has_calls(
[
call('addr_1', 0.5, 'float', ANY,
{
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call('addr_2', 0, 'float', ANY,
{
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call(
'addr_1',
0.5,
'float',
{
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
call(
'addr_2',
0,
'float',
{
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
]
)
@@ -646,7 +660,68 @@ async def test_scenario_3_2_2_opc_write_error(
prediction_confidence=12,
comments='Some data could not be written to OPC servers',
)
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_2_4_opc_session_bad_error(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.2.4: OPC session/channel Tier-1 Bad* (e.g. BadSessionIdInvalid).
PostgreSQL stores prediction_confidence 14 and a stable session error comment.
"""
client = temporal_test_env.client
model_id = 324
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
opc_write_data.return_value = (
False,
{
'notification_id': 'OPC_WRITE_DATA_ERROR_1',
'message': 'BadSessionIdInvalid',
'block': 'opc_repository',
'level': NotificationLevel.ERROR,
'attachment_content': 'BadSessionIdInvalid',
'opc_error_kind': 'session_bad',
'opc_status': 'BadSessionIdInvalid',
},
)
input_data = get_base_input_data(model_id)
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
input_data['pi_web_api_output_config'] = None
await start_and_await_workflow(
client, PredictionsBatch.run, input_data, make_workflow_id('test-opc-session-bad')
)
assert_prediction(
postgres_engine,
model_id,
prediction_confidence=14,
comments='OPC UA session/channel error: BadSessionIdInvalid',
)
@pytest.mark.asyncio