Fix flaky shutdown assertions in test suite
GC-collected instances during tests also invoke patched shutdown,
making assert_called_once_with flaky. Count calls for this instance
specifically instead of asserting total call count.
Pin sqlalchemy<2.1 to fix psycopg driver break; use tmp_path in retrain
- sqlalchemy 2.1 defaults postgresql:// URLs to psycopg (v3), but sientia_do
only ships psycopg2-binary, breaking with "No module named psycopg"
- Replace hardcoded 'tmp' mkdtemp mock with pytest tmp_path fixture in
retrain_model tests to avoid writing to a fake path
Simplify bandit nosec comments and mock metrics_controller with real spe
- drift.py/simple_metrics.py: move nosec rationale to a code comment above
the query instead of an inline suppression comment, keeping bandit happy
with a plain `# nosec`
- tests: replace bare `MagicMock()` with `MagicMock(spec=MetricsController)`
for metrics_controller across activities/repository tests so mocks fail
fast on attribute typos instead of silently accepting any call
SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QTZPOC-20: Refactor ModelImport status engine for lazy initialization, add byte-counted size validation to bundle extraction, optimize MLflow run ID lookups, and ensure workflow cancellation propagates correctly.
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.
Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
QTZPOC-20: Enhance system robustness and test coverage across bundle processing, activity error handling, and worker operations.
* Strengthen bundle reader validation with new tests for archive integrity, security vulnerabilities (e.g., path traversal, duplicate entries, special file types), and I/O error resilience.
* Improve error handling and status propagation in PI Web API, MLflow, and OPC activities, including partial write scenarios and OPC session management.
* Add comprehensive tests for worker startup, configuration, graceful shutdown, and Prometheus metrics server initialization.
* Introduce a new utility for intelligent DataFrame debugging, summarizing large dataframes for concise logging, and refine exception chain summarization.
* Update code coverage configuration to exclude @overload decorator lines.
QTZPOC-20: Improve structured logging and reporting for all model import pipeline failures by introducing a dedicated activity and centralizing error classification. This ensures consistent observability, with all workflow failures now reported before cleanup. Also, adjusted import step timeouts and refined manual run script cleanup logic.
QTZPOC-20: Implement model import workflow for .sientia bundles
Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:
* New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
* Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
* Updated .env.example and README.md with new configurations and usage instructions.
* Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
* Refactors existing utility scripts into a scripts/ directory.
Renamed the OPC.close asynchronous method to OPC.aclose to align with common Python conventions for asynchronous context managers and methods, improving clarity. All call sites and tests have been updated accordingly.
SIENTIAPDE-1646 Refactor MLFlow tests and update artifact handling
- Enhanced test cases for MLFlow to improve clarity and accuracy in data handling.
- Updated references in tests to use 'evaluation_data.csv' and 'test_data.csv' instead of 'retrain_input.csv' and 'train_data.csv'.
- Introduced a new helper function for creating prediction frames to streamline test setup.
Enhance MLFlow reference data handling and testing
- Updated the artifact handling in MLFlow to prioritize 'retrain_input.csv' over 'train_data.csv' when resolving reference data.
- Introduced new methods for resolving artifact names and locating downloaded CSV files.
- Modified the `get_reference_data` method to improve artifact resolution and error handling.
- Expanded unit tests to cover scenarios for missing artifacts and preference logic between retrain and train data CSVs.
SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
SIENTIAPDE-1811 Implement OPC write error handling and refactor tag writing logic
- Introduced a new function `_apply_opc_write_error` to manage session and reconnect flags based on OPC write error responses.
- Refactored the `_write_tags_from_config` method to streamline the writing of OPC tags for both prediction and confidence data.
- Enhanced unit tests to cover various scenarios for OPC write errors, including session bad and reconnect in progress cases.
- Updated existing tests to validate the new logic and ensure robust error handling.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Refactor OPC and OpcRepository classes to remove logger dependency and streamline logging calls
- Removed logger attribute from OPC and OpcRepository classes, replacing direct logger calls with class methods for logging.
- Updated write_data method signatures to eliminate logger parameter, simplifying the interface.
- Adjusted related tests to reflect changes in method signatures and logging behavior.
Implement new scheduling configurations for model retraining and drift analysis in input_sample.json
- Added three new schedule configurations: `minimal-retrain-test-runtime`, `drift-test-runtime`, and `simple-metrics-test-runtime`.
- Each configuration includes parameters such as model ID, workflow type, frequency, and specific queries for data retrieval.
- Enhanced the structure to support active status and updated timestamps for better tracking of schedule states.
Update E2E test report and enhance drift analysis handling
- Updated the E2E test report metrics to reflect the latest test results, showing 47 collected tests with all passing.
- Removed outdated sections related to failed tests and their causes, streamlining the report.
- Implemented a regression fix in the drift analysis to handle empty merged frames, ensuring workflows skip export when no drift metrics are available.
- Enhanced the `insert_sample_data` and `insert_sample_prediction` functions to allow customizable timestamps for better test accuracy.
- Refactored E2E tests to improve clarity and maintainability, particularly in handling repeat scenarios with distinct timestamps.
Refactor ModelMetrics to utilize DriftAnalysis for drift detection
- Replaced ModelAnalysis with DriftAnalysis in the ModelMetrics class to enhance drift detection capabilities.
- Updated method signatures and documentation to reflect the changes in target_name and return values.
- Adjusted data handling to ensure compatibility with the new analysis methods and improved clarity in the drift metrics dataframe preparation.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Refactor MLFlow retraining logic to always use retrain method
- Removed the conditional logic for full retraining, ensuring the `retrain` method is always called.
- Updated the documentation in the `retrain_model` method to reflect the changes in retraining flow.
- Adjusted tests to verify that the `retrain` method is invoked correctly, while ensuring `train` is not called when the full retrain flag is set.
Update dependencies and refactor MLFlow activities
- Replaced direct GitHub dependencies in `requirements.txt` with specific versioned packages for `sientia_do` and `sientia_model`.
- Refactored imports in `activities.py` to streamline the code structure.
- Enhanced the `MLFlow` class in `mlflow.py` by introducing a method to resolve model aliases, improving flexibility in model lookups.
- Simplified shutdown logic in `worker.py` for better readability.
- Added new tests for MLFlow activities and improved existing test coverage for data handling and model retraining processes.
Enhance environment configuration and update dependencies
- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
SIENTIAPDE-1712 Enhance logging across various classes by adding logger parameters and improving debug statements. This update includes adjustments in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes for better traceability and observability during operations.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Refactor metrics and API handling for improved consistency and clarity
- Removed the SIENTIA_CORE_LABELS constant and replaced it with CORE_LABELS for uniformity across metrics.
- Updated the API class to ensure operation_type is always included in core labels for PI Web API metrics.
- Simplified metric tag handling in the Gates class by consolidating common tags into a single core_tags dictionary.
- Enhanced the PredictionProcess class to improve error handling and variable naming for clarity.
Update environment variables in values.yaml and enhance metric labels in metrics.py
- Changed POSTGRES_USER and POSTGRES_PASSWORD values in values.yaml for improved security.
- Added 'runtime' label to metrics in metrics.py for better environment identification.
- Updated CORE_LABELS to include 'runtime' for consistency across metrics.
- Modified type hint for data parameter in PredictionProcess to use a dictionary for better clarity.
- Adjusted tests to reflect changes in core labels and MinIO configuration.
Refactor MinioDataFramePayload usage across activities
- Updated instances of MinioDataFramePayload initialization in Gates, MLFlow, and Storage classes to use the new from_dict method for better data reconstruction from dictionaries.
- Enhanced the PredictionProcess workflow to utilize the updated payload handling.
- Added passthrough fixtures in tests to accommodate the new from_dict method for consistent testing behavior.
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Refactor PI Web API response handling in tests
- Updated test cases in test_api.py to handle response data as lists instead of dictionaries for consistency with the API's expected output format.
- Adjusted mock responses to reflect the new structure, ensuring tests accurately simulate API behavior.
- Enhanced clarity in test descriptions and improved overall test coverage for response processing scenarios.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
SIENTIAPDE-1478
Enhance end-to-end tests for PredictionsBatch workflow
- Added new test scenarios for input and transform gates handling CONTINUE, STOP, and REPEAT policies.
- Implemented sample data insertion functions for testing various prediction outcomes.
- Updated existing tests to verify behavior under different input conditions and response validations.
- Refactored test structure for clarity and maintainability.
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
SIENTIAPDE-1478
Implement PI Web API response processing and metrics tracking
- Added a new method in the API class to process responses from the PI Web API, validating tag writes and emitting metrics for success and errors.
- Enhanced error handling for missing WebIds and tag names in responses, with appropriate logging and notifications.
- Updated tests to cover various scenarios for processing PI Web API responses, ensuring robust functionality and metrics emission.
- Refactored existing methods to integrate the new response processing logic, improving overall code clarity and maintainability.
Refactor Activities and API Integration for PI Web API
- Reintroduced the API import in the Activities class for proper integration.
- Cleaned up whitespace and formatting in the API class and related tests for improved readability.
- Updated test cases to ensure consistent formatting in error messages and configuration structures for PI Web API.
- Enhanced connectors_config.py with additional whitespace for better organization.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Enhance data handling and export processes in Laborious workflows
- Updated `gates.py` to improve data quality validation, filtering, and formatting operations, including enhanced metrics recording.
- Refined `mlflow.py` to better manage model transformations and reference data retrieval from MLflow Model Registry.
- Enhanced `format_and_export_prediction.py` to support separate export of transformed data, improving flexibility in data handling.
- Added comprehensive test coverage for new functionalities, including transformed data formatting and retrain report generation.
- Improved documentation in `README.md` to reflect changes in activities and workflows, ensuring clarity on data processing and export paths.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
