SIENTIAPDE-1005

Enhance README and configuration files; add Redis authentication and update workflows

README.md

@@ -1,2 +1,94 @@
-# sientia-dataops-scouter_temporal
+# Sientia DataOps Scouter
-Scouter version in Temporal
+
+The Sientia DataOps Scouter is a Temporal-based workflow application that processes industrial data from OPC collectors. It aggregates and filters data received from OPC collectors via Kafka topics, direct access to the OPC server, or active trigger (real time applications). The processed data is then stored or forwarded for further analysis.
+
+## Key Features
+
+- Data ingestion from multiple sources:
+  - OPC collectors through Kafka
+  - Direct access to OPC servers
+  - Real-time triggers for immediate processing
+- Data aggregation and filtering
+- Workflow orchestration using Temporal.io
+- Integration with Redis for caching and PostgreSQL for storage
+- Scalable deployment using Kubernetes
+
+## Workflows
+
+### Core Scouter
+
+The Core Scouter is the main workflow processes the received data. Steps:
+
+- data_quality_gate: Filters the data received from the OPC collector.
+- aggregate_data: Aggregates the data received from the OPC collector.
+- group_and_hold_data: Groups the data received from the OPC collector.
+- export_data_to_postgres: Exports the data received from the OPC collector to PostgreSQL.
+
+### Scouter
+
+The Scouter is the batch basic workflow that extracts data from the source and processes it using the Core Scouter workflow. Steps:
+
+- load_from_kafka: Loads data from a kafka topic.
+- core_scouter: Processes the data using the Core Scouter workflow.
+
+#### Workflow inputs:
+
+- `topic` (str): Kafka topic name where data is received
+- `schedule_name` (str): Name of the schedule that triggers the workflow
+- `model_name` (str): Name of the model being used for processing
+- `model_id` (int): Unique identifier for the model
+- `trigger_laborious` (bool): Flag indicating if laborious direct processing is required (real time applications)
+- `filters` (dict): Dictionary containing data filtering rules
+  - `NULL_VALUES_FILTER`: Configuration for handling null values
+    - `policy`: Policy for null values ("KEEP" or "DISCARD")
+  - `OUT_OF_BOUNDS_FILTER`: Configuration for handling out-of-bounds values
+    - `policy`: Policy for out-of-bounds values ("KEEP" or "DISCARD")
+- `schema` (str): Database schema name where data will be stored
+- `table_name` (str): Name of the table where data will be stored
+- `retention_time` (int): Time in seconds that data will be retained in Redis
+- `model_tags` (dict): Configuration for different OPC tags
+  - The key is the tag name and the value is a dictionary containing:
+    - `data_range`: List of two numbers [min, max] defining valid data range
+    - `aggr_function`: Aggregation function to use ("lts", "mdn", "avg", "max", "min")
+
+## Fake Data
+
+The Fake Data activity is used to generate fake data for testing purposes. Steps:
+
+- generate_and_send_data: Generates fake data and sends it to a kafka topic.
+
+#### Workflow inputs:
+
+- `topic` (str): Kafka topic name where data is received
+
+## Environment variables
+
+- `POSTGRES_HOST`
+- `POSTGRES_PORT`
+- `POSTGRES_USER`
+- `POSTGRES_PASSWORD`
+- `POSTGRES_DBNAME`
+- `POSTGRES_MIN_CONNECTIONS`
+- `POSTGRES_MAX_CONNECTIONS`
+
+- `KAFKA_BOOTSTRAP_SERVERS`
+- `KAFKA_POLLING_TIME`
+
+- `REDIS_HOST`
+- `REDIS_PORT`
+- `REDIS_USERNAME`
+- `REDIS_PASSWORD`
+
+- `LOG_LEVEL`
+- `PROJECT_NAME`
+
+- `TEMPORAL_HOST`
+- `TEMPORAL_NAMESPACE`
+
+## Application deployment
+
+The application can be deployed using the following command:
+
+```bash
+helm upgrade --install sientia-dataops-opc-ingestor sientia/sientia-module -n sientia-opc --create-namespace -f ./values.yaml
+```

input_sample.json

@@ -1,6 +1,5 @@
 {
     "topic": "fake_data",
-    "workflow_name": "scouter-fake-pipeline",
     "schedule_name": "scouter-fake-pipeline",
     "model_name": "fake_model",
     "model_id": 1,

scouter/activities/redis.py

@@ -8,20 +8,24 @@ with workflow.unsafe.imports_passed_through():
     import json
     from typing import Any
     from pandas import DataFrame
-    import numpy as np
     from datetime import datetime
 
 
 class Redis(BaseActivity):
     def __init__(self, host: str, port: int,
+                 username: str, password: str,
                  logger: Logger, notification_handler: NotificationHandler):
         self.host = host
         self.port = port
+        self.username = username
+        self.password = password
 
         self.redis_client = redis.Redis(
             host=self.host,
             port=self.port,
-            decode_responses=True
+            decode_responses=True,
+            username=self.username,
+            password=self.password
         )
 
         BaseActivity.__init__(self, logger, notification_handler)

scouter/utils/connectors_config.py

@@ -25,4 +25,6 @@ def build_redis_config():
     return {
         'host': getenv('REDIS_HOST', 'localhost'),
         'port': int(getenv('REDIS_PORT', '6379')),
+        'username': getenv('REDIS_USERNAME', None),
+        'password': getenv('REDIS_PASSWORD', None)
     }

scouter/workflow/fake_data.py

@@ -17,8 +17,6 @@ class FakeData:
         Args:
             workflow_input (dict[str, Any]): The input data containing:
                 topic (str): The Kafka topic to send data to
-                num_messages (int, optional): Number of messages to generate. 
-                Defaults to random.randint(1, len(self.tags)).
         """
         await workflow.execute_activity_method(
             Faker.generate_and_send_data,

scouter/workflow/scouter.py

@@ -18,7 +18,6 @@ class Scouter:
         Args:
             input_data (dict[str, Any]): The data to process. Contains:
                 topic (str): The topic to load data from.
-                workflow_name (str): The name of the workflow.
                 schedule_name (str): The name of the schedule.
                 model_name (str): The name of the model.
                 model_id (str): The id of the model.
@@ -27,9 +26,22 @@ class Scouter:
                 schema (str): The schema of the table to export data to.
                 table_name (str): The name of the table to export data to.
                 retention_time (int): The retention time for data in redis in seconds.
-                model_tags (dict[str, Any]): The tags of the model. And it's respective configuration.
+                model_tags (dict[str, Any]): The tags of the model.
+                    And it's respective configuration.
         """
 
+        input_data['workflow_name'] = 'scouter'
+
+        await workflow.execute_local_activity_method(
+            Activities.prepare_activity,
+            {
+                'workflow_name': input_data['workflow_name'],
+                'schedule_name': input_data['schedule_name'],
+                'model_name': input_data['model_name'],
+                'model_id': input_data['model_id']
+            }
+        )
+
         data = await workflow.execute_activity_method(
             Activities.load_from_kafka,
             {

tests/activities/test_redis.py

@@ -14,20 +14,26 @@ def redis_activity(_mock_redis_client):
     logger = MagicMock()
     notification_handler = MagicMock(spec=NotificationHandler)
     return Redis(host='localhost', port=6379,
-                 logger=logger, notification_handler=notification_handler)
+                 logger=logger, notification_handler=notification_handler,
+                 username='test', password='test')
 
 
 @patch('scouter.activities.redis.redis.Redis')
 def test_redis_initialization(mock_redis_client):
     """Test Redis activity initialization"""
     redis_activity = Redis(host='localhost', port=6379,
-                           logger=MagicMock(), notification_handler=MagicMock())
+                           logger=MagicMock(), notification_handler=MagicMock(),
+                           username='test', password='test')
     assert redis_activity.host == 'localhost'
     assert redis_activity.port == 6379
+    assert redis_activity.username == 'test'
+    assert redis_activity.password == 'test'
     mock_redis_client.assert_called_once_with(
         host='localhost',
         port=6379,
-        decode_responses=True
+        decode_responses=True,
+        username='test',
+        password='test'
     )
 
 

tests/utils/test_connectors_config.py

@@ -90,7 +90,9 @@ def test_build_redis_config_defaults():
 
     assert config == {
         'host': 'localhost',
-        'port': 6379
+        'port': 6379,
+        'username': None,
+        'password': None
     }
 
 
@@ -99,11 +101,15 @@ def test_build_redis_config_with_env_vars():
     """Test that build_redis_config uses env vars when set"""
     with patch.dict(os.environ, {
         'REDIS_HOST': 'redis.example.com',
-        'REDIS_PORT': '6380'
+        'REDIS_PORT': '6380',
+        'REDIS_USERNAME': 'test',
+        'REDIS_PASSWORD': 'test'
     }):
         config = build_redis_config()
 
         assert config == {
             'host': 'redis.example.com',
-            'port': 6380
+            'port': 6380,
+            'username': 'test',
+            'password': 'test'
         }

tests/workflow/test_scouter.py

@@ -16,7 +16,20 @@ async def test_scouter_workflow(mock_workflow, scouter):
     mock_workflow.execute_activity_method.return_value = 'test_data'
     await scouter.run(
         input_data={
-            'topic': 'test_topic'
+            'topic': 'test_topic',
+            'schedule_name': 'test_schedule',
+            'model_name': 'test_model',
+            'model_id': 'test_model_id'
+        }
+    )
+
+    mock_workflow.execute_local_activity_method.assert_called_once_with(
+        Activities.prepare_activity,
+        {
+            'workflow_name': 'scouter',
+            'schedule_name': 'test_schedule',
+            'model_name': 'test_model',
+            'model_id': 'test_model_id'
         }
     )
 
@@ -33,7 +46,11 @@ async def test_scouter_workflow(mock_workflow, scouter):
         'core_scouter',
         {
             'topic': 'test_topic',
-            'data': 'test_data'
+            'data': 'test_data',
+            'workflow_name': 'scouter',
+            'schedule_name': 'test_schedule',
+            'model_name': 'test_model',
+            'model_id': 'test_model_id'
         }
     )
 
@@ -44,7 +61,20 @@ async def test_scouter_workflow_empty(mock_workflow, scouter):
     mock_workflow.execute_activity_method.return_value = {}
     await scouter.run(
         input_data={
-            'topic': 'test_topic'
+            'topic': 'test_topic',
+            'schedule_name': 'test_schedule',
+            'model_name': 'test_model',
+            'model_id': 'test_model_id'
+        }
+    )
+
+    mock_workflow.execute_local_activity_method.assert_called_once_with(
+        Activities.prepare_activity,
+        {
+            'workflow_name': 'scouter',
+            'schedule_name': 'test_schedule',
+            'model_name': 'test_model',
+            'model_id': 'test_model_id'
         }
     )
 

values.yaml

@@ -0,0 +1,183 @@
+# Default values for sientia-module.
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+
+# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
+replicaCount: 1
+
+# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
+image:
+  repository: aignosi.azurecr.io/sientia-module
+  # This sets the pull policy for images.
+  pullPolicy: Always
+  # Overrides the image tag whose default is the chart appVersion.
+  tag: "0.0.1"
+
+# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
+imagePullSecrets:
+- name: docker-hub-secret
+# This is to override the chart name.
+nameOverride: "sientia-scouter-worker"
+fullnameOverride: "sientia-scouter-worker"
+namespace: sientia
+
+# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
+serviceAccount:
+  # Specifies whether a service account should be created
+  create: true
+  # Automatically mount a ServiceAccount's API credentials?
+  automount: true
+  # Annotations to add to the service account
+  annotations: {}
+  # The name of the service account to use.
+  # If not set and create is true, a name is generated using the fullname template
+  name: "sientia-scouter-worker"
+
+# This is for setting Kubernetes Annotations to a Pod.
+# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
+podAnnotations: {}
+# This is for setting Kubernetes Labels to a Pod.
+# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
+podLabels: {}
+
+podSecurityContext: {}
+  # fsGroup: 2000
+
+securityContext: {}
+  # capabilities:
+  #   drop:
+  #   - ALL
+  # readOnlyRootFilesystem: true
+  # runAsNonRoot: true
+  # runAsUser: 1000
+
+
+resources: {}
+  # We usually recommend not to specify default resources and to leave this as a conscious
+  # choice for the user. This also increases chances charts run on environments with little
+  # resources, such as Minikube. If you do want to specify resources, uncomment the following
+  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
+  # limits:
+  #   cpu: 100m
+  #   memory: 128Mi
+  # requests:
+  #   cpu: 100m
+  #   memory: 128Mi
+
+# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
+livenessProbe:
+  exec:
+    command:
+      - sh
+      - -c
+      - pgrep -f "scouter.worker.worker"
+  initialDelaySeconds: 20
+  periodSeconds: 30
+
+readinessProbe:
+  exec:
+    command:
+      - sh
+      - -c
+      - pgrep -f "scouter.worker.worker"
+  initialDelaySeconds: 10
+  periodSeconds: 15
+
+
+# This section is for setting up autoscaling more information can be found here: https://kubernetes.io/docs/concepts/workloads/autoscaling/
+autoscaling:
+  enabled: false
+  minReplicas: 1
+  maxReplicas: 100
+  targetCPUUtilizationPercentage: 80
+  # targetMemoryUtilizationPercentage: 80
+
+# Additional volumes on the output Deployment definition.
+volumes: []
+# - name: foo
+#   secret:
+#     secretName: mysecret
+#     optional: false
+
+# Additional volumeMounts on the output Deployment definition.
+volumeMounts: []
+# - name: foo
+#   mountPath: "/etc/foo"
+#   readOnly: true
+
+nodeSelector: {}
+
+tolerations: []
+
+affinity: {}
+
+service:
+  enabled: false
+  type: ClusterIP
+  port: 4840
+  targetPort: 4840
+  
+
+env:
+  # Entrypoint variables 
+  - name: GITHUB_REPO_URL
+    value: "git@github.com:Aignosi/sientia-dataops-scouter_temporal.git"
+  - name: GITHUB_BRANCH
+    value: "SIENTIAPDE-1005-implementar-os-workflows-mapeados-utilizando-as-workers-e-activities-apropriadas"
+  - name: PYTHON_APP
+    value: "scouter.worker.worker"
+
+  # Application variables
+  - name: POSTGRES_HOST
+    value: "paradedb-rw.paradedb.svc.cluster.local"
+  - name: POSTGRES_PORT
+    value: "5432"
+  - name: POSTGRES_USER
+    value: "sientia"
+  - name: POSTGRES_PASSWORD
+    value: "sientia"
+  - name: POSTGRES_DBNAME
+    value: "sientia"
+  - name: POSTGRES_MIN_CONNECTIONS
+    value: "10"
+  - name: POSTGRES_MAX_CONNECTIONS
+    value: "20"
+
+  - name: KAFKA_BOOTSTRAP_SERVERS
+    value: "kafka.kafka.svc.cluster.local:9092"
+  - name: KAFKA_POLLING_TIME
+    value: "1000"
+
+  - name: REDIS_HOST
+    value: "redis-master.redis.svc.cluster.local"
+  - name: REDIS_PORT
+    value: "6379"
+  - name: REDIS_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: redis
+        key: redis-username
+  - name: REDIS_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: redis
+        key: redis-password
+
+  - name: LOG_LEVEL
+    value: "INFO"
+  - name: PROJECT_NAME
+    value: "sientia-scouter"
+
+  - name: TEMPORAL_HOST
+    value: "temporal.temporal.svc.cluster.local:7233"
+  - name: TEMPORAL_NAMESPACE
+    value: "default"
+
+ssh:
+  enabled: true
+  secretName: git-ssh-key-temp
+  sshPath: /mnt/.ssh
+  knownHostsPath: /mnt/known_hosts
+
+# kubectl create secret docker-registry docker-hub-secret --namespace sientia --docker-server=http://aignosi.azurecr.io --docker-username=aignosi --docker-password=5I5zpQ6sRaHqX1hD3dr+2mo647yO3FRc359/wu6gsP+ACRDRz5mp
+# helm upgrade --install sientia-dataops-scouter sientia/sientia-module -n sientia --create-namespace -f ./values.yaml