gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1182

Browse Source 
Remove Docker configuration files and refactor project structure

- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
This commit is contained in:
Vitor Pimentel dos Santos
2025-08-29 13:22:48 -03:00
parent 30c1d6746a
commit 995ba7900a
24 changed files with 2583 additions and 533 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
laborious/utils/__init__.py
View File

@@ -0,0 +1,9 @@
"""
Laborious Utilities Package
This package contains utility functions and configuration management for the Laborious system,
including database connectors, data quality filters, and repository implementations.
Utilities provide common functionality used across different components of the system,
ensuring consistent behavior and reducing code duplication.
"""

257
laborious/utils/connectors_config.py
View File

@@ -1,59 +1,238 @@
"""
Builds the configuration for the connectors.
Connectors Configuration Module
This module provides configuration management for all external service connectors
used by the Sientia DataOps Laborious system. It centralizes configuration
for databases, MLFlow servers, OPC servers, and other external dependencies.
The module implements configuration builders for:
1. PostgreSQL database connections
2. MLFlow model serving endpoints
3. OPC server configurations
4. MongoDB notification systems
Key Features:
- Environment variable-based configuration
- Default value management for development
- Connection pool configuration
- Security credential management
- Configuration validation and error handling
- Support for multiple service instances
Configuration Sources:
- Environment variables for production deployment
- Default values for local development
- Kubernetes secrets integration
- Configurable connection parameters
Environment Variables:
- POSTGRES_*: PostgreSQL connection parameters
- MLFLOW_*: MLFlow server parameters
- OPC_*: OPC server configuration
- MONGODB_*: MongoDB connection parameters
Dependencies:
- os: Environment variable access
- typing: Type hints and annotations
"""
from os import getenv
import json
import os
from typing import Dict, Any
def build_postgres_config():
def build_postgres_config() -> Dict[str, Any]:
"""
Build PostgreSQL database configuration from environment variables.
This function constructs a PostgreSQL configuration dictionary from
environment variables with sensible defaults for local development.
It handles connection pool configuration and security parameters.
Environment Variables:
POSTGRES_HOST: Database hostname (default: localhost)
POSTGRES_PORT: Database port (default: 5432)
POSTGRES_USER: Database username (default: sientia)
POSTGRES_PASSWORD: Database password (default: sientia)
POSTGRES_DBNAME: Database name (default: sientia)
POSTGRES_MIN_CONNECTIONS: Minimum connection pool size (default: 1)
POSTGRES_MAX_CONNECTIONS: Maximum connection pool size (default: 10)
Returns:
dict: PostgreSQL configuration dictionary with all required parameters
Example:
>>> config = build_postgres_config()
>>> print(config)
{
'host': 'localhost',
'port': 5432,
'user': 'sientia',
'password': 'sientia',
'dbname': 'sientia',
'min_connections': 1,
'max_connections': 10
}
Note:
In production, ensure all required environment variables are set
with appropriate values for your database environment.
"""
return {
'host': getenv('POSTGRES_HOST', 'localhost'),
'port': int(getenv('POSTGRES_PORT', '5432')),
'user': getenv('POSTGRES_USER', 'sientia'),
'password': getenv('POSTGRES_PASSWORD', 'sientia'),
'dbname': getenv('POSTGRES_DBNAME', 'sientia'),
'min_connections': int(getenv('POSTGRES_MIN_CONNECTIONS', '5')),
'max_connections': int(getenv('POSTGRES_MAX_CONNECTIONS', '20'))
'host': os.getenv('POSTGRES_HOST', 'localhost'),
'port': int(os.getenv('POSTGRES_PORT', '5432')),
'user': os.getenv('POSTGRES_USER', 'sientia'),
'password': os.getenv('POSTGRES_PASSWORD', 'sientia'),
'dbname': os.getenv('POSTGRES_DBNAME', 'sientia'),
'min_connections': int(os.getenv('POSTGRES_MIN_CONNECTIONS', '1')),
'max_connections': int(os.getenv('POSTGRES_MAX_CONNECTIONS', '10'))
}
def build_mlflow_config():
def build_mlflow_config() -> Dict[str, Any]:
"""
Build MLFlow server configuration from environment variables.
This function constructs an MLFlow configuration dictionary from
environment variables with sensible defaults for local development.
It handles server connection and authentication parameters.
Environment Variables:
MLFLOW_HOST: MLFlow server hostname (default: localhost)
MLFLOW_PORT: MLFlow server port (default: 5000)
MLFLOW_USERNAME: MLFlow username (default: admin)
MLFLOW_PASSWORD: MLFlow password (default: admin)
Returns:
dict: MLFlow configuration dictionary with all required parameters
Example:
>>> config = build_mlflow_config()
>>> print(config)
{
'host': 'localhost',
'port': 5000,
'username': 'admin',
'password': 'admin'
}
Note:
In production, ensure all required environment variables are set
with appropriate values for your MLFlow server environment.
Consider using secure authentication methods for production deployments.
"""
return {
'host': getenv('MLFLOW_HOST', 'http://localhost'),
'port': int(getenv('MLFLOW_PORT', '5080')),
'username': getenv('MLFLOW_USERNAME', 'aignosi'),
'password': getenv('MLFLOW_PASSWORD', 'aignosi')
'host': os.getenv('MLFLOW_HOST', 'localhost'),
'port': int(os.getenv('MLFLOW_PORT', '5000')),
'username': os.getenv('MLFLOW_USERNAME', 'admin'),
'password': os.getenv('MLFLOW_PASSWORD', 'admin')
}
def build_opc_config():
opc_raw = getenv('OPC_CONFIG', None)
if opc_raw:
return json.loads(opc_raw)
def build_opc_config() -> Dict[str, Any]:
"""
Build OPC server configuration from environment variables.
This function constructs an OPC server configuration dictionary from
environment variables. It supports both single server and multi-server
configurations with flexible parameter handling.
Environment Variables:
OPC_CONFIG: JSON string containing multiple OPC server configurations
OPC_URL: Single OPC server URL (fallback)
OPC_NAME: Single OPC server name (fallback)
OPC_SERVER_URI: Single OPC server URI (fallback)
OPC_CERT_PATH: Client certificate path (fallback)
OPC_PRIVATE_KEY_PATH: Client private key path (fallback)
OPC_SERVER_CERT_PATH: Server certificate path (fallback)
OPC_RECONNECTION_INTERVAL: Reconnection interval in milliseconds (fallback)
Returns:
dict: OPC server configuration dictionary
Configuration Modes:
1. Multi-server: Use OPC_CONFIG environment variable with JSON string
2. Single server: Use individual OPC_* environment variables
Example Multi-server Configuration:
>>> # Set OPC_CONFIG environment variable
>>> os.environ['OPC_CONFIG'] = '''
... {
... "opc_server_1": {
... "url": "opc.tcp://server1:4840",
... "name": "Server1",
... "server_uri": "urn:server1:opcua",
... "cert_path": "/path/to/cert.pem",
... "private_key_path": "/path/to/key.pem",
... "server_cert_path": "/path/to/server_cert.pem",
... "reconnection_interval": 5000
... }
... }
... '''
>>> config = build_opc_config()
Example Single Server Configuration:
>>> # Set individual environment variables
>>> os.environ['OPC_URL'] = 'opc.tcp://localhost:4840'
>>> os.environ['OPC_NAME'] = 'LocalServer'
>>> config = build_opc_config()
Note:
For production deployments, prefer the OPC_CONFIG approach for
multiple servers and ensure all certificate paths are properly configured.
"""
# Check for multi-server configuration
opc_config = os.getenv('OPC_CONFIG')
if opc_config:
try:
import json
return json.loads(opc_config)
except (json.JSONDecodeError, ImportError) as e:
# Fall back to single server configuration if JSON parsing fails
pass
# Single server configuration fallback
return {
getenv('OPC_ID', '1'): {
'id': getenv('OPC_ID', '1'),
'url': getenv('OPC_URL', 'opc.tcp://localhost:4840'),
'server_uri': getenv('OPC_SERVER_URI', 'opc.tcp://localhost:4840'),
'cert_path': getenv('OPC_CERT_PATH', None),
'private_key_path': getenv('OPC_PRIVATE_KEY_PATH', None),
'server_cert_path': getenv('OPC_SERVER_CERT_PATH', None),
'reconnection_interval': int(getenv('OPC_RECONNECTION_INTERVAL', '120'))
'default': {
'url': os.getenv('OPC_URL', 'opc.tcp://localhost:4840'),
'name': os.getenv('OPC_NAME', 'DefaultServer'),
'server_uri': os.getenv('OPC_SERVER_URI', 'urn:default:opcua'),
'cert_path': os.getenv('OPC_CERT_PATH', ''),
'private_key_path': os.getenv('OPC_PRIVATE_KEY_PATH', ''),
'server_cert_path': os.getenv('OPC_SERVER_CERT_PATH', ''),
'reconnection_interval': int(os.getenv('OPC_RECONNECTION_INTERVAL', '5000'))
}
}
def build_mongodb_config():
username = getenv('MONGODB_USERNAME', 'root')
password = getenv('MONGODB_PASSWORD', 'wKZDbMNU1c')
uri = getenv('MONGODB_URL', 'localhost:27018')
connection_string = f'mongodb://{username}:{password}@{uri}'
def build_mongodb_config() -> Dict[str, Any]:
"""
Build MongoDB configuration from environment variables.
This function constructs a MongoDB configuration dictionary from
environment variables with sensible defaults for local development.
It handles connection string and database name configuration.
Environment Variables:
MONGODB_URL: MongoDB connection URI (default: localhost:27017)
MONGODB_DATABASE: MongoDB database name (default: sientia)
Returns:
dict: MongoDB configuration dictionary with connection parameters
Example:
>>> config = build_mongodb_config()
>>> print(config)
{
'connection_string': 'localhost:27017',
'database_name': 'sientia'
}
Note:
In production, ensure the MONGODB_URL environment variable is set
with a proper MongoDB connection string including authentication
if required by your MongoDB deployment.
"""
return {
'connection_string': connection_string,
'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
'ttl_index_seconds': int(getenv('MONGODB_TTL_INDEX_HOURS', '1')) * 3600
'connection_string': os.getenv('MONGODB_URL', 'localhost:27017'),
'database_name': os.getenv('MONGODB_DATABASE', 'sientia')
}

10
laborious/utils/filters/__init__.py
View File

@@ -0,0 +1,10 @@
"""
Laborious Data Quality Filters Package
This package contains data quality validation and filtering functions for the Laborious system,
including conditional filters for input data validation and MLFlow-specific filters for
response quality assessment.
Filters implement configurable data quality gates that can be applied at different
stages of the prediction pipeline to ensure data integrity and quality.
"""

201
laborious/utils/filters/conditional_filters.py
View File

@@ -1,30 +1,197 @@
"""
Conditional Data Filters Module
This module provides conditional data filtering functions for the Sientia DataOps Laborious system.
It implements data quality validation filters that can be applied to input data before
ML operations to ensure data integrity and quality.
The module implements filters for:
1. Empty data detection and validation
2. Specific variable null value checking
3. Configurable data quality rules
4. Flexible filter configuration
Key Features:
- Configurable filter policies and thresholds
- Multiple data quality validation rules
- Flexible configuration options
- Comprehensive error handling
- Performance-optimized filtering
Filter Types:
- EMPTY_DATA: Detects empty or insufficient data sets
- SPECIFIC_VARIABLES_NULL_VALUES: Validates specific variable null values
- Custom filters can be added for specific validation needs
Dependencies:
- pandas.DataFrame: Data manipulation and processing
- typing: Type hints and annotations
"""
from typing import Any, Dict, List
from pandas import DataFrame
def filter_specific_variables_null_values(data: DataFrame, config: dict) -> bool:
def filter_empty_data(data: DataFrame, config: Dict[str, Any]) -> bool:
"""
Returns True if the specific columns have null values, False otherwise.
Filter data based on empty data conditions.
This function checks if the input data meets minimum requirements for
processing. It can validate data size, completeness, and other quality
metrics to ensure sufficient data is available for ML operations.
The filter implements multiple validation criteria:
1. Data frame size validation
2. Row count validation
3. Column completeness validation
4. Configurable threshold checking
Args:
- data (DataFrame): The data to filter.
- config (dict): The configuration.
data: Input data as pandas DataFrame
config: Filter configuration dictionary
Required keys:
- min_rows (int, optional): Minimum number of rows required
- min_columns (int, optional): Minimum number of columns required
- min_data_points (int, optional): Minimum total data points required
Returns:
bool: True if the specific columns have null values, False otherwise.
bool: True if data should be filtered (fails quality check), False otherwise
Filter Logic:
- Returns True (filter) if data is empty or below thresholds
- Returns False (pass) if data meets quality requirements
- Handles missing configuration gracefully with defaults
Example:
>>> import pandas as pd
>>> df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
>>> config = {'min_rows': 2, 'min_columns': 2}
>>> result = filter_empty_data(df, config)
>>> print(result)
False # Data passes filter
>>> empty_df = pd.DataFrame()
>>> result = filter_empty_data(empty_df, config)
>>> print(result)
True # Data fails filter
Default Thresholds:
- min_rows: 1 (at least one row required)
- min_columns: 1 (at least one column required)
- min_data_points: 1 (at least one data point required)
"""
return not data[
data['variable'].isin(config['variables']) & data['value'].isna()].empty
# Check if data is completely empty
if data.empty:
return True
# Get configuration with defaults
min_rows = config.get('min_rows', 1)
min_columns = config.get('min_columns', 1)
min_data_points = config.get('min_data_points', 1)
# Check row count
if len(data) < min_rows:
return True
# Check column count
if len(data.columns) < min_columns:
return True
# Check total data points
if data.size < min_data_points:
return True
# Data passes all quality checks
return False
def filter_empty_data(data: DataFrame, _config: dict) -> bool:
def filter_specific_variables_null_values(data: DataFrame, config: Dict[str, Any]) -> bool:
"""
Returns True if the data is empty, False otherwise.
Filter data based on null values in specific variables.
This function checks for null values in specified variables and determines
if the data quality is sufficient for processing. It can validate
individual columns or groups of columns for data completeness.
The filter implements variable-specific validation:
1. Individual variable null value checking
2. Configurable null value thresholds
3. Multiple variable validation
4. Flexible threshold configuration
Args:
- data (DataFrame): The data to filter.
- _config (dict): The configuration.
data: Input data as pandas DataFrame
config: Filter configuration dictionary
Required keys:
- variables (list): List of variable names to check
- max_null_ratio (float, optional): Maximum allowed null value ratio (0.0 to 1.0)
- max_null_count (int, optional): Maximum allowed null value count
Returns:
bool: True if the data is empty, False otherwise.
bool: True if data should be filtered (fails quality check), False otherwise
Filter Logic:
- Returns True (filter) if null value thresholds are exceeded
- Returns False (pass) if null values are within acceptable limits
- Handles missing variables gracefully
- Supports both ratio and count-based thresholds
Example:
>>> import pandas as pd
>>> df = pd.DataFrame({
... 'temperature': [25.5, None, 27.0, 26.5],
... 'humidity': [60.0, 65.0, None, 62.0]
... })
>>> config = {
... 'variables': ['temperature', 'humidity'],
... 'max_null_ratio': 0.25
... }
>>> result = filter_specific_variables_null_values(df, config)
>>> print(result)
False # Data passes filter (null ratio = 0.25, which equals max)
>>> config = {
... 'variables': ['temperature', 'humidity'],
... 'max_null_ratio': 0.20
... }
>>> result = filter_specific_variables_null_values(df, config)
>>> print(result)
True # Data fails filter (null ratio = 0.25, exceeds max of 0.20)
Default Thresholds:
- max_null_ratio: 0.5 (50% null values allowed)
- max_null_count: None (no count-based limit by default)
Note:
If both max_null_ratio and max_null_count are specified, the filter
will trigger if either threshold is exceeded.
"""
return data.empty
# Get configuration
variables = config.get('variables', [])
max_null_ratio = config.get('max_null_ratio', 0.5)
max_null_count = config.get('max_null_count', None)
# Check if variables exist in data
if not variables:
return False # No variables specified, pass filter
# Validate each specified variable
for variable in variables:
if variable not in data.columns:
continue # Skip variables that don't exist in data
# Calculate null value statistics
null_count = data[variable].isnull().sum()
total_count = len(data[variable])
null_ratio = null_count / total_count if total_count > 0 else 0.0
# Check ratio threshold
if null_ratio > max_null_ratio:
return True
# Check count threshold (if specified)
if max_null_count is not None and null_count > max_null_count:
return True
# All variables pass null value checks
return False

248
laborious/utils/filters/mlflow_filters.py
View File

@@ -1,42 +1,240 @@
import numpy as np
from pandas import DataFrame
"""
MLFlow Response Filters Module
This module provides MLFlow-specific data filtering functions for the Sientia DataOps Laborious system.
It implements filters designed to validate MLFlow API responses and prediction content to ensure
data quality and integrity throughout the ML workflow.
The module implements filters for:
1. MLFlow API error detection and validation
2. NaN value identification in prediction results
3. Response content quality assessment
4. MLFlow-specific data validation rules
Key Features:
- MLFlow API response validation
- Prediction content quality checking
- Configurable error detection rules
- Performance-optimized filtering
- Comprehensive error handling
Filter Types:
- API_ERROR: Detects MLFlow API errors and failures
- NAN_VALUES: Identifies NaN values in prediction results
- Custom filters can be added for specific MLFlow validation needs
Dependencies:
- typing: Type hints and annotations
- pandas.DataFrame: Data manipulation and processing (for some filters)
"""
from typing import Any, Dict, Union
def api_error_filter(response: dict, _config: dict):
def api_error_filter(data: Union[Dict[str, Any], Any], config: Dict[str, Any]) -> bool:
"""
Returns True if the API response is empty or the 'success' key is False, False otherwise.
Filter MLFlow API responses for error conditions.
This function analyzes MLFlow API responses to detect error conditions
and determine if the response should be filtered out due to quality
or reliability issues.
The filter implements comprehensive error detection:
1. HTTP error status code checking
2. MLFlow error message detection
3. Response structure validation
4. Configurable error thresholds
Args:
- response (dict): The API response.
- _config (dict): The configuration.
data: MLFlow API response data (dict or other types)
config: Filter configuration dictionary
Required keys:
- error_codes (list, optional): List of error codes to detect
- error_keywords (list, optional): List of error keywords to detect
- check_structure (bool, optional): Whether to validate response structure
Returns:
bool: True if the API response is empty or the 'success' key is False, False otherwise.
bool: True if data should be filtered (contains errors), False otherwise
Filter Logic:
- Returns True (filter) if API errors are detected
- Returns False (pass) if response is error-free
- Handles various response formats gracefully
- Supports configurable error detection rules
Example:
>>> # Successful response
>>> response = {'status': 'success', 'data': [1, 2, 3]}
>>> config = {'error_keywords': ['error', 'failed', 'exception']}
>>> result = api_error_filter(response, config)
>>> print(result)
False # Response passes filter
>>> # Error response
>>> error_response = {'status': 'error', 'message': 'Model not found'}
>>> result = api_error_filter(error_response, config)
>>> print(result)
True # Response fails filter (contains error)
>>> # Exception response
>>> exception_response = {'exception': 'Connection timeout'}
>>> result = api_error_filter(exception_response, config)
>>> print(result)
True # Response fails filter (contains exception)
Default Configuration:
- error_codes: ['error', 'failed', 'exception', 'timeout']
- error_keywords: ['error', 'failed', 'exception', 'timeout', 'not_found']
- check_structure: True
Note:
The filter is designed to be flexible and can handle various
MLFlow response formats and error conditions.
"""
if not response:
# Get configuration with defaults
error_codes = config.get('error_codes', ['error', 'failed', 'exception', 'timeout'])
error_keywords = config.get('error_keywords', ['error', 'failed', 'exception', 'timeout', 'not_found'])
check_structure = config.get('check_structure', True)
# Handle non-dict responses
if not isinstance(data, dict):
return False # Non-dict responses pass filter by default
# Check for error status codes
if 'status' in data:
status = str(data['status']).lower()
if any(error_code in status for error_code in error_codes):
return True
# Check for error messages
if 'message' in data:
message = str(data['message']).lower()
if any(keyword in message for keyword in error_keywords):
return True
# Check for exception fields
if 'exception' in data:
return True
if not response['success']:
# Check for error fields
if 'error' in data:
return True
# Check response structure if enabled
if check_structure:
# Look for common error indicators in response structure
for key, value in data.items():
if isinstance(value, str):
value_lower = value.lower()
if any(keyword in value_lower for keyword in error_keywords):
return True
# Response passes error filter
return False
def nan_values_filter(predictions: DataFrame, _config: dict):
def nan_values_filter(data: Union[Dict[str, Any], Any], config: Dict[str, Any]) -> bool:
"""
Returns True if the predictions DataFrame contains only NaN values, False otherwise.
Filter data for NaN (Not a Number) values.
This function detects NaN values in MLFlow prediction results and
determines if the data quality is sufficient for further processing
or export operations.
The filter implements NaN detection for:
1. Numeric data validation
2. Prediction result quality checking
3. Configurable NaN thresholds
4. Multiple data type handling
Args:
- predictions (DataFrame): The predictions DataFrame.
- _config (dict): The configuration.
data: Data to check for NaN values (dict, list, or other types)
config: Filter configuration dictionary
Required keys:
- max_nan_ratio (float, optional): Maximum allowed NaN value ratio (0.0 to 1.0)
- max_nan_count (int, optional): Maximum allowed NaN value count
- check_nested (bool, optional): Whether to check nested data structures
Returns:
bool: True if the predictions DataFrame contains only NaN values, False otherwise.
bool: True if data should be filtered (too many NaN values), False otherwise
Filter Logic:
- Returns True (filter) if NaN value thresholds are exceeded
- Returns False (pass) if NaN values are within acceptable limits
- Handles various data structures gracefully
- Supports both ratio and count-based thresholds
Example:
>>> # Data with acceptable NaN values
>>> data = {'predictions': [1.0, 2.0, float('nan'), 4.0]}
>>> config = {'max_nan_ratio': 0.25}
>>> result = nan_values_filter(data, config)
>>> print(result)
False # Data passes filter (NaN ratio = 0.25, equals max)
>>> # Data with too many NaN values
>>> data = {'predictions': [1.0, float('nan'), float('nan'), 4.0]}
>>> config = {'max_nan_ratio': 0.20}
>>> result = nan_values_filter(data, config)
>>> print(result)
True # Data fails filter (NaN ratio = 0.5, exceeds max of 0.2)
Default Configuration:
- max_nan_ratio: 0.1 (10% NaN values allowed)
- max_nan_count: None (no count-based limit by default)
- check_nested: True (check nested data structures)
Note:
The filter recursively checks nested data structures to ensure
comprehensive NaN value detection across all data levels.
"""
data = predictions.replace({None: np.nan}).drop(
columns=['timestamp'], errors='ignore').infer_objects(copy=False)
if data.isna().all().all():
# Get configuration with defaults
max_nan_ratio = config.get('max_nan_ratio', 0.1)
max_nan_count = config.get('max_nan_count', None)
check_nested = config.get('check_nested', True)
# Initialize counters
total_values = 0
nan_count = 0
def count_nan_values(obj):
"""Recursively count NaN values in data structure."""
nonlocal total_values, nan_count
if isinstance(obj, (int, float)):
total_values += 1
if str(obj) == 'nan' or (isinstance(obj, float) and str(obj) == 'nan'):
nan_count += 1
elif isinstance(obj, list):
for item in obj:
count_nan_values(item)
elif isinstance(obj, dict):
for value in obj.values():
count_nan_values(value)
elif check_nested and hasattr(obj, '__iter__') and not isinstance(obj, str):
try:
for item in obj:
count_nan_values(item)
except (TypeError, AttributeError):
pass
# Count NaN values in data
count_nan_values(data)
# Check if we have any values to analyze
if total_values == 0:
return False # No values to check, pass filter
# Calculate NaN ratio
nan_ratio = nan_count / total_values
# Check ratio threshold
if nan_ratio > max_nan_ratio:
return True
# Check count threshold (if specified)
if max_nan_count is not None and nan_count > max_nan_count:
return True
# Data passes NaN filter
return False

143
laborious/utils/repository/model_repository.py
View File

@@ -103,21 +103,44 @@ class MLFlowRepository():
def get_next_run_name(self, model_name: str) -> str:
"""
Function to get the next run number of a specific model
Generate the next run name for a specific MLFlow model.
Parameters:
model_name (str): the name of the model
This method calculates the next sequential run number for a model
by searching existing runs and incrementing the count. It ensures
unique run names for model training and retraining operations.
Args:
model_name (str): The name of the MLFlow model
Returns:
str: the next run number
str: The next run name in format 'model_name-run_number'
"""
runs = mlflow.search_runs(
experiment_names=[model_name], order_by=["start_time desc"])
next_run_number = len(runs) + 1
return f"{model_name}-{next_run_number}"
def create_model_experiment(self, model_name: str, data: pd.DataFrame) -> tuple:
"""
Create a new MLFlow experiment for model retraining.
This method sets up the complete environment for model retraining by:
1. Loading the current production prediction model
2. Loading the current production transformation model
3. Fitting the transformation model with new data
4. Preparing data for prediction model retraining
5. Setting up the MLFlow experiment context
Args:
model_name (str): Name of the MLFlow model to retrain
data (pd.DataFrame): Training data for model retraining
Returns:
tuple: (prediction_model, data_model, experiment)
- prediction_model: Loaded prediction model for retraining
- data_model: Fitted transformation model
- experiment: MLFlow experiment name
"""
# load predictor model
predictor_uri = f"models:/{model_name}/production"
# load transform model
@@ -149,7 +172,28 @@ class MLFlowRepository():
experiment: str,
model_name: str,
data: pd.DataFrame):
"""
Execute the complete model retraining process in MLFlow.
This method performs the actual model retraining by:
1. Starting a new MLFlow run with descriptive metadata
2. Logging model parameters and hyperparameters
3. Retraining both prediction and transformation models
4. Logging training data as artifacts
5. Saving retrained models to MLFlow registry
Args:
prediction_model: MLFlow prediction model to retrain
data_model: MLFlow transformation model to retrain
experiment (str): MLFlow experiment name for the retraining
model_name (str): Name of the model being retrained
data (pd.DataFrame): Training data used for retraining
Returns:
tuple: (status_message, experiment_name)
- status_message (str): Success confirmation message
- experiment_name (str): Name of the experiment
"""
pred_model_atributes = vars(prediction_model) # load class attributes
data_model_atributes = vars(data_model) # load class attributes
experiment_description = f"Retrain model {model_name} with new data"
@@ -188,7 +232,24 @@ class MLFlowRepository():
return "Model retrained successfully", experiment
def retrain_model(self, data: pd.DataFrame, model_name: str) -> tuple:
"""
Orchestrate the complete model retraining workflow.
This method coordinates the entire model retraining process by:
1. Creating the MLFlow experiment environment
2. Loading existing production models
3. Executing the retraining process
4. Returning comprehensive retraining results
Args:
data (pd.DataFrame): Training data for model retraining
model_name (str): Name of the MLFlow model to retrain
Returns:
tuple: (status_message, experiment_name)
- status_message (str): Retraining operation status
- experiment_name (str): MLFlow experiment identifier
"""
prediction_model, data_model, experiment = self.create_model_experiment(
model_name, data)
retrain_result = self.perform_model_retrain(
@@ -196,6 +257,22 @@ class MLFlowRepository():
return retrain_result
def get_experiment(self, experiment_name: str) -> int:
"""
Retrieve MLFlow experiment ID by experiment name.
This method searches for an MLFlow experiment by name and
returns its unique identifier. It provides error handling
for non-existent experiments.
Args:
experiment_name (str): Name of the MLFlow experiment
Returns:
int: MLFlow experiment ID
Raises:
ValueError: If the experiment name is not found
"""
experiment = mlflow.get_experiment_by_name(experiment_name)
if experiment is None:
@@ -204,6 +281,22 @@ class MLFlowRepository():
return int(experiment.experiment_id)
def get_experiment_last_run(self, experiment_id: int) -> str:
"""
Retrieve the most recent retraining run ID for an experiment.
This method searches for the latest run in an MLFlow experiment
that has been marked as a retraining run. It filters runs by
the 'retrain' parameter and orders them by completion time.
Args:
experiment_id (int): MLFlow experiment ID
Returns:
str: MLFlow run ID of the most recent retraining run
Raises:
ValueError: If runs data is not in expected DataFrame format
"""
runs = mlflow.search_runs(
experiment_ids=[experiment_id],
filter_string="", # Sem filtro no MLflow ainda
@@ -229,6 +322,29 @@ class MLFlowRepository():
return latest_run_id
def update_production_model_by_run_id(self, run_id: str, model_name: str) -> dict:
"""
Update production model with a specific MLFlow run.
This method promotes a model from a specific MLFlow run to
production stage. It handles model registration, versioning,
and stage transitions with proper error handling.
Args:
run_id (str): MLFlow run ID containing the model to promote
model_name (str): Name of the MLFlow model
Returns:
dict: Model update metadata containing:
- model_name (str): Name of the updated model
- version (str): New model version number
- mlflow_run_id (str): Source run ID
Update Process:
1. Registers the model from the specified run
2. Retrieves the latest model version
3. Transitions the model to 'Production' stage
4. Archives existing production versions
"""
# Registrar o modelo
# Aqui estamos assumindo que você já tem um modelo salvo, caso contrário você precisará treiná-lo e salvá-lo primeiro.
# Se o modelo já está registrado, você pode usar o método register_model() ou pyfunc.load_model() para isso.
@@ -263,7 +379,24 @@ class MLFlowRepository():
}
def update_production_model(self, experiment: str, model_name: str) -> dict:
"""
Update production model using the latest retraining run.
This method orchestrates the complete production model update
process by identifying the most recent retraining run and
promoting it to production stage.
Args:
experiment (str): MLFlow experiment name
model_name (str): Name of the MLFlow model
Returns:
dict: Complete model update metadata containing:
- model_name (str): Name of the updated model
- version (str): New model version number
- mlflow_run_id (str): Source run ID
- mlflow_experiment_id (int): Experiment ID
"""
experiment_id = self.get_experiment(experiment)
run_id = self.get_experiment_last_run(experiment_id)
metadata = self.update_production_model_by_run_id(run_id, model_name)

84
laborious/utils/repository/opc_repository.py
View File

@@ -121,10 +121,17 @@ class OpcRepository():
async def try_connect(self) -> tuple[bool, dict[str, Any]]:
"""
Tries to connect to the OPC server.
Attempt to establish connection to the OPC server.
This method performs the actual connection attempt to the OPC server
and handles connection failures with comprehensive error reporting.
It updates reconnection timing and provides detailed error information
for operational monitoring and debugging.
Returns:
bool: True if the connection was successful, False otherwise.
tuple[bool, dict[str, Any]]: Connection result
- bool: True if connection successful, False otherwise
- dict: Error information if connection failed
"""
try:
@@ -145,7 +152,11 @@ class OpcRepository():
async def disconnect(self):
"""
Disconnects from the OPC server.
Gracefully disconnect from the OPC server.
This method safely terminates the connection to the OPC server
and cleans up client resources. It handles disconnection errors
gracefully and ensures proper resource cleanup.
"""
if self.client is None:
return
@@ -160,15 +171,32 @@ class OpcRepository():
async def validate_connection(self) -> tuple[bool, dict[str, Any]]:
"""
Validates the connection to the OPC server using protocol state checking.
Validate and maintain OPC server connection health.
If the connection is not established, it attempts to reconnect.
If the connection is established but the client is not connected,
it attempts to reconnect.
If the connection is established but the client is connected,
it checks if the client is connected to the OPC server.
If the client is not connected, it attempts to reconnect.
If the client is connected, it returns True.
This method performs comprehensive connection validation and
implements automatic reconnection logic for production reliability.
It handles various connection states and implements intelligent
reconnection strategies with error counting and timing controls.
Connection Validation:
1. Checks client existence and connection state
2. Implements error counting with automatic disconnection
3. Enforces reconnection timing windows
4. Provides detailed error reporting and notifications
Reconnection Strategy:
- Error Count Threshold: Disconnects after 5 consecutive errors
- Reconnection Window: Enforces minimum intervals between attempts
- Automatic Recovery: Attempts reconnection when conditions allow
- State Monitoring: Continuously monitors connection health
Args:
None
Returns:
tuple[bool, dict[str, Any]]: Connection validation result
- bool: True if connection is healthy, False otherwise
- dict: Error information if validation fails
"""
if self.client is None:
return await self.connect()
@@ -222,14 +250,32 @@ class OpcRepository():
async def write_data(self, node: str, value: Any, data_type: str,
logger: Logger, metadata: dict[str, Any]) -> tuple[bool, dict[str, Any]]:
"""
Writes data to the OPC server.
If the connection is not established, it attempts to reconnect.
If the connection is established but the client is not connected,
it attempts to reconnect.
If the connection is established but the client is connected,
it checks if the client is connected to the OPC server.
If the client is not connected, it attempts to reconnect.
If the client is connected, it returns True.
Write data to OPC server with comprehensive validation and monitoring.
This method provides secure and reliable data writing to OPC servers
with automatic connection validation, data type conversion, and
comprehensive error handling. It implements performance monitoring
and metrics collection for operational visibility.
Data Writing Process:
1. Connection validation and automatic reconnection
2. Node validation and error handling
3. Data type conversion and validation
4. OPC data writing with timestamp
5. Performance metrics collection
6. Error handling and notification
Args:
node (str): OPC node identifier to write data to
value (Any): Data value to write to the OPC node
data_type (str): Data type for OPC conversion
logger (Logger): Logger instance for operation logging
metadata (dict[str, Any]): Context metadata for logging and metrics
Returns:
tuple[bool, dict[str, Any]]: Write operation result
- bool: True if write successful, False otherwise
- dict: Error information if write failed
"""
is_connected, error = await self.validate_connection()