SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
Refactor ModelMetrics to utilize DriftAnalysis for drift detection
- Replaced ModelAnalysis with DriftAnalysis in the ModelMetrics class to enhance drift detection capabilities.
- Updated method signatures and documentation to reflect the changes in target_name and return values.
- Adjusted data handling to ensure compatibility with the new analysis methods and improved clarity in the drift metrics dataframe preparation.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
Update values.yaml to rename worker references and adjust GitHub branch for SIENTIAPDE-1811. Changed nameOverride, fullnameOverride, and service account name to "sientia-laborious-legacy-worker" and updated the GITHUB_BRANCH value to "fix/SIENTIAPDE-1811".
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
SIENTIAPDE-1646 Add new runtime configuration for single environment in values.yaml
- Introduced a new runtime named "single" with a replica count of 1.
- Added an environment variable "GITHUB_BRANCH" set to "0.4.2" for the new runtime.
SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
Update .gitignore and input_sample.json for improved configuration management
- Added new patterns to .gitignore to exclude 'mlruns/', 'relatorio*', and 'openspec/*' directories from version control.
- Updated the 'updated_at' date in input_sample.json to reflect a new timestamp for better tracking of schedule states.
Refactor OPC and OpcRepository classes to remove logger dependency and streamline logging calls
- Removed logger attribute from OPC and OpcRepository classes, replacing direct logger calls with class methods for logging.
- Updated write_data method signatures to eliminate logger parameter, simplifying the interface.
- Adjusted related tests to reflect changes in method signatures and logging behavior.
Implement new scheduling configurations for model retraining and drift analysis in input_sample.json
- Added three new schedule configurations: `minimal-retrain-test-runtime`, `drift-test-runtime`, and `simple-metrics-test-runtime`.
- Each configuration includes parameters such as model ID, workflow type, frequency, and specific queries for data retrieval.
- Enhanced the structure to support active status and updated timestamps for better tracking of schedule states.
SIENTIAPDE-1646 Add new scheduling configurations and remove outdated documentation
- Introduced new scheduling configurations for minimal retrain, drift analysis, and simple metrics in `input_sample.json`.
- Removed obsolete documentation files related to drift analysis and E2E test reports to streamline project resources.
- Updated E2E tests for minimal retrain to enhance reporting and error handling during model retraining processes.
Refactor MLflow configuration to use environment variable for tracking URL
- Removed the `_build_mlflow_tracking_url` function and replaced its usage with `getenv` to directly retrieve the `MLFLOW_URL` environment variable.
- This change simplifies the configuration process by allowing users to specify the tracking URL directly through an environment variable.
SIENTIAPDE-1646 Update dependencies and refactor OPC integration
- Replaced `opcua` with `asyncua` in `requirements-local.txt` and `requirements.txt` to utilize the async capabilities.
- Updated `values.yaml` to change the GitHub branch from `feature/SIENTIAPDE-1646` to `release/SIENTIAPDE-1646`.
- Refactored `opc.py` and `opc_repository.py` to accommodate the new `asyncua` library, ensuring compatibility with the synchronous API.
- Adjusted tests in `test_opc_repository.py` to reflect changes in the client implementation and maintain functionality.
Update E2E test report and enhance drift analysis handling
- Updated the E2E test report metrics to reflect the latest test results, showing 47 collected tests with all passing.
- Removed outdated sections related to failed tests and their causes, streamlining the report.
- Implemented a regression fix in the drift analysis to handle empty merged frames, ensuring workflows skip export when no drift metrics are available.
- Enhanced the `insert_sample_data` and `insert_sample_prediction` functions to allow customizable timestamps for better test accuracy.
- Refactored E2E tests to improve clarity and maintainability, particularly in handling repeat scenarios with distinct timestamps.
Refactor ModelMetrics to utilize DriftAnalysis for drift detection
- Replaced ModelAnalysis with DriftAnalysis in the ModelMetrics class to enhance drift detection capabilities.
- Updated method signatures and documentation to reflect the changes in target_name and return values.
- Adjusted data handling to ensure compatibility with the new analysis methods and improved clarity in the drift metrics dataframe preparation.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Refactor MLFlow retraining logic to always use retrain method
- Removed the conditional logic for full retraining, ensuring the `retrain` method is always called.
- Updated the documentation in the `retrain_model` method to reflect the changes in retraining flow.
- Adjusted tests to verify that the `retrain` method is invoked correctly, while ensuring `train` is not called when the full retrain flag is set.
Update dependencies and refactor MLFlow activities
- Replaced direct GitHub dependencies in `requirements.txt` with specific versioned packages for `sientia_do` and `sientia_model`.
- Refactored imports in `activities.py` to streamline the code structure.
- Enhanced the `MLFlow` class in `mlflow.py` by introducing a method to resolve model aliases, improving flexibility in model lookups.
- Simplified shutdown logic in `worker.py` for better readability.
- Added new tests for MLFlow activities and improved existing test coverage for data handling and model retraining processes.
Enhance environment configuration and update dependencies
- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
SIENTIAPDE-1712 Enhance logging across various classes by adding logger parameters and improving debug statements. This update includes adjustments in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes for better traceability and observability during operations.
Update values.yaml to adjust workflow and activity poller configurations for improved performance, increasing minimum, initial, and maximum values. Additionally, add new minimal retrain parameters to enhance retraining capabilities.
SIENTIAPDE-1712 Enhance logging in MLFlowRepository class by introducing a dedicated _debug_dataframe method for conditional logging of DataFrame content. This update improves observability during model transformation, prediction, and retraining processes while managing log output effectively.
SIENTIAPDE-1712 Refactor logging in MLFlowRepository class by removing the debug_dataframe method and replacing it with direct debug statements for improved clarity. This change enhances the logging of DataFrame content during model transformation, prediction, and retraining processes, ensuring better observability without excessive log output.
Update values.yaml to rename SIE_MINIO_OFFLOAD_THRESHOLD_MEGABYTES to SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES and adjust its value to 0.5 for improved configuration clarity.
SIENTIAPDE-1712 Refactor logging in MinioDataFramePayload class to utilize custom_debug method for improved clarity and consistency. Enhanced DataFrame size logging by integrating a dedicated debug method, streamlining the logging process.
SIENTIAPDE-1712 Enhance logging in Gates, MLFlow, and Storage classes by integrating logger parameter for improved traceability. This update allows for better monitoring of operations and data handling across these components.
SIENTIAPDE-1712 Implement debug logging in MinioDataFramePayload class for enhanced traceability. Added a static method for conditional logging and integrated debug statements throughout methods to capture DataFrame size estimates, upload actions, and retrieval processes, improving overall observability.
SIENTIAPDE-1712 Refactor debug logging for DataFrames across multiple classes. Introduced a new method to log DataFrame content conditionally based on row count in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes, improving debugging capabilities while managing log output effectively.
SIENTIAPDE-1712 Implement debug logging for DataFrames in MLFlow and MLFlowRepository classes. Added a method to log DataFrame content conditionally based on row count, enhancing debugging capabilities while preventing excessive log output.
Refactor prediction process to use execute_activity_method for MLFlow model transformation and prediction requests, enhancing consistency in workflow execution.
Update values.yaml to enhance PredictionsBatch worker tuning parameters
- Renamed SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES to SIE_MINIO_OFFLOAD_THRESHOLD_MEGABYTES for consistency.
- Added multiple configuration options for PredictionsBatch to optimize workflow and activity concurrency, including max concurrent tasks, activities, local activities, cached workflows, and poller behaviors.
Update Gates and FormatAndExportPrediction classes to standardize timestamp usage
- Changed 'last_timestamp' to 'timestamp' in the Gates class for consistency in input data handling.
- Updated the FormatAndExportPrediction class to reflect the same change in the output data structure.
Update Gates and FormatAndExportPrediction classes to use 'last_timestamp' for improved data handling
- Modified the Gates class to utilize 'last_timestamp' when only one row is present, ensuring accurate timestamp assignment.
- Updated the FormatAndExportPrediction class to replace 'timestamp' with 'last_timestamp' in the output data structure.
Enhance prediction export functionality by adding 'on_conflict' and 'unique_columns' parameters to the payload initialization in FormatAndExportPrediction class.
Update pyproject.toml to remove specific test file from exclusion list and bump sientia-dataops-library dependency version to 1.10.4 in requirements-light.txt
