SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QTZPOC-13: Introduce joblib fallback for MLflow model loading
Enhances model loading resilience by attempting to deserialize models directly with joblib if MLflow's native flavor loaders fail. This mechanism handles cases where .pkl files are present in the artifact.
Adds joblib as a core dependency and includes new end-to-end and unit tests to validate fallback behavior, covering success, no-fallback, and double-failure scenarios. Also updates .gitignore for .claude/ files and incorporates minor refactorings and linting adjustments across E2E tests and utility functions.
Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
Update .gitignore and input_sample.json for improved configuration management
- Added new patterns to .gitignore to exclude 'mlruns/', 'relatorio*', and 'openspec/*' directories from version control.
- Updated the 'updated_at' date in input_sample.json to reflect a new timestamp for better tracking of schedule states.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Update .gitignore and remove values.yaml for project cleanup
- Added .ruff_cache/ and catboost_info/ to .gitignore to prevent tracking of temporary files and caches.
- Deleted values.yaml to remove outdated configuration settings, streamlining the project structure.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Update .gitignore, values.yaml, and model_repository.py for improved data handling and logging
- Added 'catboost_info/' to .gitignore to prevent tracking of additional temporary files.
- Updated GITHUB_BRANCH in values.yaml to reflect the current branch for model retraining.
- Enhanced model_repository.py to drop duplicate timestamps in treated data and streamline attribute logging during model retraining.
Update .gitignore to include 'tmp/' directory and ensure '.env' is listed
- Added 'tmp/' to the .gitignore file to prevent temporary files from being tracked.
- Confirmed that '.env' is included to avoid committing sensitive environment variables.
Update .gitignore to include 'tmp/' directory and ensure '.env' is listed
- Added 'tmp/' to the .gitignore file to prevent temporary files from being tracked.
- Confirmed that '.env' is included to avoid committing sensitive environment variables.
- Expanded the .env file with configurations for MongoDB, Postgres, MlFlow, and Temporal.
- Refactored OPC class methods to be asynchronous, including init_opc, write_data, manage_output_tags, and shutdown.
- Updated the worker to initialize OPC asynchronously and adjusted shutdown handling for activities.
Refactor tests for Postgres activities and improve error handling
- Updated test_postgres.py to enhance the testing of load_custom_query method, including cases for None data and date conversion.
- Refactored repeat_last_prediction tests to use mocks for SQLAlchemy session execution.
- Added tests for export_data_to_postgres method, covering both success and error scenarios.
- Improved the initialization tests for Activities class to ensure proper instantiation of dependencies.
- Enhanced test coverage for OPC repository connection validation.
- Updated tests for prediction workflows to streamline input handling and improve clarity.
- Introduced tests for connectors configuration to validate environment variable handling for MLFlow, OPC, and Postgres.
- Added tests for logger utility to ensure default settings are correctly applied.
