Commit Graph

100 Commits

Author SHA1 Message Date
vitor-aignosi
6fb28a0050 SIENTIAPDE-1172
Update dependencies in requirements.txt and increment image tag in values.yaml

- Updated sientia-dataops-library version from 1.3.4 to 1.3.5 in requirements.txt.
- Incremented image tag in values.yaml from 0.2.5 to 0.2.7.
2025-07-29 13:19:29 -03:00
vitor-aignosi
c9dc00fd08 SIENTIAPDE-1172
Update GITHUB_BRANCH in values.yaml and enhance error notification handling in MLFlow

- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for alerts orchestration.
- Added NotificationLevel.ERROR to error notifications in MLFlow for retraining and production model updates.
- Updated tests to verify error notification levels for model management workflows.
2025-07-29 12:45:28 -03:00
vitor-aignosi
26887c820c Merge pull request #12 from Aignosi/SIENTIAPDE-1171-criar-pipeline-de-retreino-laborious
Sientiapde 1171 criar pipeline de retreino laborious
2025-07-24 11:46:47 -03:00
vitor-aignosi
a34bdcd839 SIENTIAPDE-1171
Refactor data merging in MLFlowRepository to simplify merge operation in fit method. Removed unnecessary parameters for improved readability and maintainability.
2025-07-24 11:24:21 -03:00
vitor-aignosi
1261e4a8aa SIENTIAPDE-1171
Enhance data merging in MLFlowRepository by specifying the 'on' parameter for improved join accuracy. This change ensures that the merge operation correctly aligns data based on the 'id' field.
2025-07-24 10:56:05 -03:00
vitor-aignosi
a2492e244d SIENTIAPDE-1171
Enhance data merging in MLFlowRepository by specifying merge parameters

- Updated the merge operation in the fit method to include 'how' and 'validate' parameters for improved data integrity and flexibility during model training.
2025-07-24 10:30:20 -03:00
vitor-aignosi
c738e8b0df SIENTIAPDE-1171
Refactor model_repository and enhance test coverage for MLFlow functionalities

- Updated model_repository to ensure the 'temp' directory is created if it doesn't exist using `exist_ok=True`.
- Added new tests for retraining and updating production models, including error handling scenarios.
- Improved existing tests for model management workflows to ensure robustness and reliability.
2025-07-24 10:24:08 -03:00
vitor-aignosi
4283730e7a SIENTIAPDE-1171
Update GITHUB_BRANCH in values.yaml and refactor MLFlowRepository for model retraining

- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for retraining.
- Refactored MLFlowRepository by introducing create_model_experiment and perform_model_retrain methods to streamline model retraining processes.
- Updated test cases to cover new functionalities and ensure proper error handling in model management workflows.
2025-07-23 16:51:33 -03:00
vitor-aignosi
89b9892a5b SIENTIAPDE-1171
Update dependencies and enhance ML model retraining functionality

- Updated sientia-dataops-library version in requirements.txt from 1.3.3 to 1.3.4.
- Incremented image tag in values.yaml from 0.2.4 to 0.2.5 and added a new environment variable MONGODB_TTL_INDEX_HOURS.
- Introduced new methods in MLFlowRepository for model retraining and production model updates, including error handling and logging.
- Added retrain_model and update_production_model activities in mlflow.py to support model management workflows.
- Modified MongoDB connection settings in connectors_config.py for improved security and configuration flexibility.
2025-07-23 12:02:16 -03:00
Bruno Domingues
9410de5f82 Merge pull request #11 from Aignosi/SIENTIAPDE-1151-criar-testes-de-stress
Sientiapde 1151 criar testes de stress
2025-07-22 12:22:46 -03:00
vitor-aignosi
4f315a1506 Merge branch 'main' into SIENTIAPDE-1151-criar-testes-de-stress 2025-07-22 11:34:16 -03:00
Bruno Domingues
1844118091 Merge pull request #10 from Aignosi/SIENTIAPDE-1163-alterar-dinamica-de-notificacoes-para-usar-o-mongodb-ao-inves-do-kafka
Sientiapde 1163 alterar dinamica de notificacoes para usar o mongodb ao inves do kafka
2025-07-22 10:14:23 -03:00
vitor-aignosi
40cf93d500 SIENTIAPDE-1151
Update GITHUB_BRANCH in values.yaml to reflect new stress testing branch and modify OPC error handling to set success flag instead of continuing on error.
2025-07-18 14:42:22 -03:00
vitor-aignosi
71d256d192 SIENTIAPDE-1151
Enhance OPC error handling by adding detailed notifications for missing servers during write operations. The notification includes the list of available OPC servers for better debugging.
2025-07-18 14:33:07 -03:00
vitor-aignosi
b440c7d547 SIENTIAPDE-1163
SIENTIAPDE-1163 Enhance OPC tests to improve error handling and notifications. Added tests for MongoDB configuration and refined connection error handling in OPC repository.
2025-07-17 15:21:10 -03:00
vitor-aignosi
488674c7e1 SIENTIAPDE-1163
Update requirements.txt to replace local dataops library path with git repository link for version 1.3.3.
2025-07-17 12:45:29 -03:00
vitor-aignosi
9bcf882d00 SIENTIAPDE-1163
Update .gitignore to include git_log and enhance README.md with instructions for generating a git log summary.
2025-07-17 12:37:54 -03:00
vitor-aignosi
515881b404 SIENTIAPDE-1163
Add MongoDB configuration to values.yaml for integration
2025-07-17 12:31:24 -03:00
vitor-aignosi
694ad265c8 SIENTIAPDE-1163
Refactor OPC connection handling to improve error notifications and update GITHUB_BRANCH in values.yaml for MongoDB integration. Change requirements.txt to point to local dataops library path.
2025-07-17 11:02:41 -03:00
vitor-aignosi
c17792020e SIENTIAPDE-1163
Update sientia-dataops-library dependency version to 1.3.0 in requirements.txt for improved functionality.
2025-07-16 09:23:11 -03:00
vitor-aignosi
b6b0dba735 SIENTIAPDE-1163
Update notification handler references to use CoreNotificationHandler and add MongoDB configuration to connectors. Update GITHUB_BRANCH in values.yaml for optimization tasks.
2025-07-16 09:19:40 -03:00
Matheus Demoner
16bba47323 Merge pull request #9 from Aignosi/SIENTIAPDE-1148-separar-scouter-laborious-e-orchestrator-por-namespaces
Sientiapde 1148 separar scouter laborious e orchestrator por namespaces
2025-07-14 10:36:36 -03:00
vitor-aignosi
5787d9f7a0 SIENTIAPDE-1148
Refactor Gates activity to correctly sort data by timestamp and update exception handling in worker to include NOSONAR comment for clarity.
2025-07-11 12:09:24 -03:00
vitor-aignosi
dfaab046fc SIENTIAPDE-1148 2025-07-11 11:42:02 -03:00
vitor-aignosi
42a39ac9ef Merge pull request #8 from Aignosi/SIENTIAPDE-1141-criar-testes-de-carga
Sientiapde 1141 criar testes de carga
2025-07-11 09:25:01 -03:00
vitor-aignosi
ceb82354ea SIENTIAPDE-1141
SIENTIAPDE-1141 Update worker.py to configure concurrency limits for workflow tasks and activities, enhancing performance and resource management.
2025-07-10 16:28:15 -03:00
vitor-aignosi
20a624b26d SIENTIAPDE-1141
Update values.yaml to increase replicaCount to 5, change GITHUB_BRANCH to SIENTIAPDE-1141-criar-testes-de-carga, and adjust POSTGRES_MAX_CONNECTIONS to 30 for improved performance and testing.
2025-07-10 15:35:29 -03:00
Matheus Demoner
3a01a9a244 Merge pull request #7 from Aignosi/SIENTIAPDE-1110-criar-testes-e-2-e
Sientiapde 1110 criar testes e 2 e
2025-07-08 19:14:17 -03:00
vitor-aignosi
f4610fec5b SIENTIAPDE-1110
Update dependencies in requirements.txt and bump image tag in values.yaml to version 0.2.4 for consistency and improvements.
2025-07-07 16:34:34 -03:00
vitor-aignosi
50e6479c49 SIENTIAPDE-1110
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
2025-07-04 15:19:14 -03:00
vitor-aignosi
58216a91ce SIENTIAPDE-1110
Enhance OpcRepository write_data method to set SourceTimestamp using individual datetime components, improving precision in timestamping.
2025-06-27 17:07:33 -03:00
vitor-aignosi
d2e4561dfd SIENTIAPDE-1110
Enhance OpcRepository write_data method to include SourceTimestamp using DateTime, improving data accuracy and traceability.
2025-06-27 17:03:59 -03:00
vitor-aignosi
80ad2d54ad SIENTIAPDE-1110
Refactor Gates and OpcRepository for improved logging and error handling. Removed debug statement in Gates and enhanced disconnect method in OpcRepository to log disconnection status and handle exceptions.
2025-06-27 15:52:33 -03:00
vitor-aignosi
90c6beb673 SIENTIAPDE-1110
Refactor logging in OpcRepository to consistently use 'schedule_name' from metadata for improved error reporting and traceability.
2025-06-27 12:31:23 -03:00
vitor-aignosi
fd9d933931 SIENTIAPDE-1110
Update OPC activity to utilize custom logger in write_data method, improving error handling and traceability.
2025-06-27 12:27:40 -03:00
vitor-aignosi
566818b308 SIENTIAPDE-1110
Refactor logging in OPC and OpcRepository to utilize custom logging methods, enhancing error handling and traceability with metadata integration.
2025-06-27 12:24:15 -03:00
vitor-aignosi
119decac16 SIENTIAPDE-1110
Refactor OPC activity and OpcRepository to include metadata parameter in write_data methods, improving error handling and logging capabilities for better traceability.
2025-06-27 11:05:47 -03:00
vitor-aignosi
f692e23a28 SIENTIAPDE-1110
Update connectors_config to use dynamic OPC_ID as key for configuration, enhancing flexibility in OPC setup.
2025-06-27 10:49:49 -03:00
vitor-aignosi
a1b7d8e024 SIENTIAPDE-1110
Enhance logging in OPC activity by adding debug statements for output configuration and server details to improve traceability.
2025-06-27 10:45:39 -03:00
vitor-aignosi
cb8ffd57c0 SIENTIAPDE-1110
Fixes and tries
2025-06-27 10:44:20 -03:00
vitor-aignosi
ce69b0839f SIENTIAPDE-1110
SIENTIAPDE-1110 Refactor OPC activity to use server IDs instead of names, update values.yaml for OPC_ID, and enhance OpcRepository initialization for improved clarity and consistency.
2025-06-27 10:37:06 -03:00
vitor-aignosi
82356ca3b0 SIENTIAPDE-1110
Update process_confidence method in OPC activity to include metadata parameter for improved data handling.
2025-06-27 10:23:52 -03:00
vitor-aignosi
f1eba071aa SIENTIAPDE-1110
Refactor metadata handling in PredictionProcess to improve clarity by explicitly naming the metadata key in the workflow execution parameters.
2025-06-27 10:20:28 -03:00
vitor-aignosi
f76ae839a9 SIENTIAPDE-1110
Refactor metadata handling in PredictionsBatch and PredictionProcess to improve clarity in input preparation.
2025-06-27 10:06:38 -03:00
vitor-aignosi
73a5ac1e38 SIENTIAPDE-1110
Add metadata logging in PredictionProcess for enhanced traceability.
2025-06-27 10:03:19 -03:00
vitor-aignosi
1325e7b7e3 SIENTIAPDE-1110
Update OPC_URL in values.yaml and enhance input data logging in PredictionProcess for improved traceability.
2025-06-27 10:00:34 -03:00
vitor-aignosi
b1e14aa8fe SIENTIAPDE-1110
Enhance logging in Gates activity by adding input data debug statement for improved traceability.
2025-06-27 09:47:52 -03:00
vitor-aignosi
98ea2a7f75 SIENTIAPDE-1110
Update dependencies and enhance logging in activities; bump version in requirements and values.yaml
2025-06-27 09:17:24 -03:00
vitor-aignosi
cefef0b1e9 SIENTIAPDE-1110
Enhance MLFlow activity to sort data by created_at and remove duplicates; update tests to reflect changes
2025-06-25 15:33:31 -03:00
Bruno Domingues
2be78b1aea SIENTIAPDE-1111: refactor sonarqube pipeline. 2025-06-23 12:21:55 -03:00