ci(workflow): Enhance quality gate, streamline validation, and add AI guidance

- **Quality Gate (.github/workflows/quality-gate.yml):**
- Expanded push triggers to include main, release/**, and feature/** branches.
- Added concurrency control to cancel in-progress runs, preventing redundant checks.
- Refined workflow permissions to be more specific (contents: read, pull-requests: write, issues: write), enhancing security.
- **Local Validation (validate.sh):**
- Removed the validate.sh script, as its checks are now integrated into or superseded by the CI pipeline.
- **AI Guidance (CLAUDE.md):**
- Introduced CLAUDE.md to provide comprehensive guidance for Claude Code, detailing project architecture, commands, and conventions for AI assistants.
- **SonarQube Configuration (sonar-project.properties):**
- Removed hardcoded sonar.projectVersion.

.github/workflows/quality-gate.yml

@@ -1,6 +1,11 @@
 name: Quality gate
 
 on:
+  push:
+    branches:
+      - main
+      - 'release/**'
+      - 'feature/**'
   pull_request:
     branches:
       - main
@@ -8,10 +13,17 @@ on:
       - 'feature/**'
     types: [ opened, synchronize, reopened ]
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   quality-gate:
     uses: Aignosi/github_workflow_templates/.github/workflows/python-quality-gate.yml@main
-    permissions: write-all
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
     with:
       project_name: 'ingestor'
       repositories: 'sientia-dataops-library'

CLAUDE.md

@@ -0,0 +1,74 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project
+
+OPC UA data ingestion service. Pulls tags from OPC UA servers, streams to Kafka (optional) and persists to MongoDB. Horizontally scaled via Redis-coordinated slot leasing — multiple pods share tag load without central orchestration.
+
+Depends on internal shared lib `sientia-dataops-library` (imported as `sientia_do`, pinned in `requirements.txt` via git+ssh) for logging, notifications, metrics controller, and Mongo/Redis repositories. Base classes/utilities from `sientia_do` won't be visible in this repo — check that lib's source if behavior there is unclear.
+
+## Commands
+
+```bash
+python3.11 -m venv venv && source ./venv/bin/activate
+pip install -r requirements.txt -r requirements-dev.txt
+
+./run_local.sh                          # loads .env, runs python -m ingestor.app
+./run_coverage.sh                       # pytest --cov=ingestor --cov-report=html, opens report
+
+pytest                                  # all tests
+pytest tests/unit                       # unit only
+pytest tests/functional                 # functional only (spins up real-ish clients, see conftest.py)
+pytest tests/unit/test_ingestor.py::TestClass::test_name   # single test
+pytest --cov=ingestor --cov-report=html # coverage report -> htmlcov/
+
+ruff check .                            # lint
+ruff format .                           # format
+mypy ingestor                           # type check
+bandit -r ingestor                      # security scan
+```
+
+External deps (Redis, MongoDB, Kafka) must be reachable — via `kubectl port-forward`, local docker-compose, or cloud services. Config lives in `.env` (copy from `.env.example`).
+
+## Architecture
+
+Manager-based pipeline, one manager per concern, composed by `IngestorManager`:
+
+```
+app.py (asyncio loop, signals, Prometheus server)
+  -> Ingestor (ingestor.py) — lifecycle: prepare_ingestor() -> loop() -> shutdown()
+       -> IngestorManager (managers/ingestor_manager.py) — central coordinator
+            -> ResourceManager (managers/resource_manager.py) — Redis slot leases, heartbeats, active-ingestor registry
+            -> OpcManager (managers/opc_manager.py), one instance per OPC server — connection, subscriptions, datachange callbacks
+            -> DataManager (managers/data_manager.py) — MongoDB persistence + optional Kafka publish
+```
+
+**Slot model**: OPC tag configs live in Redis under keys like `slot:opc_tags:<n>`, each holding one or more OPC servers and their tags (see README "OPC Server Configuration" for the JSON shape). A "slot" is the unit of lease/assignment; each ingestor pod leases some number of slots via `ResourceManager` and only subscribes to the tags in slots it holds.
+
+**Main loop** (`Ingestor.loop()` in `ingestor.py`) each cycle:
+1. `declare_active()` + heartbeat (keeps this pod visible to peers via Redis TTL keys — `LEASE_TTL`/`HEARTBEAT_TTL`)
+2. Reads current active-ingestor count, lease count, slot count
+3. `manage_no_slots()` — grabs one slot if this pod is idle and slots exist
+4. `manage_leases()` — rebalances: acquires slots if other pods are lacking, drops extras (keeps at most 1 slot per pod when supply is sufficient) — this is the load-balancing algorithm
+5. `update_slot_config()` + `check_opc_servers_integrity()` — refresh OPC configs, detect stalled connections
+6. `update_ingestor_manager()` — diffs current vs previous managed tags, subscribes new, resubscribes changed, unsubscribes removed
+
+Any change to slot/lease counts should be traced through `manage_leases`/`manage_no_slots` — that's where the rebalancing math lives, not in `ResourceManager` itself (which just does Redis primitives).
+
+**OpcManager** owns one OPC UA `Client` connection (asyncua), handles security policy setup (cert/key-based, `SecurityPolicyBasic256`), subscriptions per tag group, and `datachange_notification` callbacks that push into `data_queue` for `DataManager` to consume. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles`).
+
+**Metrics** (`ingestor/metrics.py`): all Prometheus metrics defined here, always labeled by `pod_id`. Emitted via `SientiaMonitoring.emit_metric(...)` (from `sientia_do`), not the prometheus client directly — new metrics should follow that pattern for consistency with logging/notifications.
+
+**Error handling convention**: broad `except Exception` + `traceback.print_exc()` + notification-handler alert is intentional throughout (see `ruff.lint.ignore` for `BLE001` in `pyproject.toml`) — this is an always-on service where a single tag/server failure must not kill the pod; failures are surfaced via notifications/metrics instead of raised.
+
+## Config
+
+Env vars documented in README ("Configuration" section) and `.env.example`. Notable ones affecting behavior: `LEASE_TTL`/`HEARTBEAT_TTL` (failover speed vs stability), `POLL_INTERVAL` (loop cadence), `EXPORT_TO_KAFKA`.
+
+## Testing notes
+
+- `tests/unit/managers/` mirrors `ingestor/managers/` 1:1.
+- `tests/functional/` exercises more end-to-end paths against `conftest.py` fixtures.
+- Coverage config (`pyproject.toml` / `.coveragerc`) excludes `ingestor/app.py` (thin entrypoint, signal wiring) — don't chase coverage there.
+- `ruff` complexity cap is `max-complexity = 15` (mccabe) — factor out branches in manager methods before that.

sonar-project.properties

@@ -7,5 +7,4 @@ sonar.qualitygate.timeout=300
 sonar.python.coverage.reportPaths=coverage.xml
 sonar.python.xunit.reportPath=pytest.xml
 sonar.python.version=3.11
-sonar.projectVersion=1.2.0
 sonar.coverage.exclusions=ingestor/app.py

validate.sh

@@ -1,124 +0,0 @@
-#!/bin/bash
-# Model Manager Code Validation Script
-# This script runs all code quality checks before committing or deploying
-
-set -e  # Exit on any error
-
-# Colors for output
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m' # No Color
-
-# Args
-FIX_MODE=false
-while [[ $# -gt 0 ]]; do
-    case "$1" in
-        --fix)
-            FIX_MODE=true
-            shift
-            ;;
-        -h|--help)
-            echo "Usage: $0 [--fix]"
-            echo "  --fix   Apply Ruff auto-fixes (format and lint fixes)."
-            exit 0
-            ;;
-        *)
-            echo -e "${RED}Unknown option: $1${NC}"
-            echo "Usage: $0 [--fix]"
-            exit 2
-            ;;
-    esac
-done
-
-echo -e "${BLUE}╔════════════════════════════════════════════════════════╗${NC}"
-echo -e "${BLUE}║  Model Manager - Code Validation Suite                ║${NC}"
-echo -e "${BLUE}╚════════════════════════════════════════════════════════╝${NC}"
-echo ""
-
-# Check if virtual environment is activated
-if [[ -z "${VIRTUAL_ENV}" ]] && [[ -z "${CONDA_DEFAULT_ENV}" ]]; then
-    echo -e "${YELLOW}⚠️  Warning: No virtual environment detected${NC}"
-    echo -e "${YELLOW}   Consider activating your venv/conda environment${NC}"
-    echo ""
-fi
-
-# Function to run a validation step
-run_step() {
-    local step_name=$1
-    local step_command=$2
-    
-    echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
-    echo -e "${BLUE}▶ ${step_name}${NC}"
-    echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
-    
-    if eval "$step_command"; then
-        echo -e "${GREEN}✅ ${step_name} - PASSED${NC}"
-        echo ""
-        return 0
-    else
-        echo -e "${RED}❌ ${step_name} - FAILED${NC}"
-        echo ""
-        return 1
-    fi
-}
-
-# Track failures
-FAILED_STEPS=()
-
-# Step 1: Code Formatting Check (Ruff)
-# - default: check only
-# - --fix: write changes
-if ! run_step "1. Code Formatting (Ruff)" "if \$FIX_MODE; then ruff format ingestor/ tests/; else ruff format --check ingestor/ tests/; fi"; then
-    FAILED_STEPS+=("Code Formatting")
-fi
-
-# Step 2: Linting (Ruff)
-# - default: check only
-# - --fix: apply autofixes
-if ! run_step "2. Code Linting (Ruff)" "if \$FIX_MODE; then ruff check --fix ingestor/ tests/; else ruff check ingestor/ tests/; fi"; then
-    FAILED_STEPS+=("Linting")
-fi
-
-# Step 3: Type Checking (mypy)
-if ! run_step "3. Type Checking (mypy)" "mypy ingestor/"; then
-    FAILED_STEPS+=("Type Checking")
-fi
-
-# Step 4: Security Analysis (Bandit)
-if ! run_step "4. Security Analysis (Bandit)" "bandit -r ingestor/ -ll -q"; then
-    FAILED_STEPS+=("Security Analysis")
-fi
-
-# Step 5: Unit Tests (pytest)
-if ! run_step "5. Unit Tests (pytest)" "pytest tests/ --cov=ingestor --cov-report=term-missing --cov-report=xml --cov-report=html --cov-fail-under=80 -q"; then
-    FAILED_STEPS+=("Unit Tests")
-fi
-
-# Summary
-echo -e "${BLUE}╔════════════════════════════════════════════════════════╗${NC}"
-echo -e "${BLUE}║  Validation Summary                                    ║${NC}"
-echo -e "${BLUE}╚════════════════════════════════════════════════════════╝${NC}"
-echo ""
-
-if [ ${#FAILED_STEPS[@]} -eq 0 ]; then
-    echo -e "${GREEN}✅ All validation checks passed!${NC}"
-    echo -e "${GREEN}   Your code is ready for commit/deployment.${NC}"
-    echo ""
-    exit 0
-else
-    echo -e "${RED}❌ Validation failed for the following steps:${NC}"
-    for step in "${FAILED_STEPS[@]}"; do
-        echo -e "${RED}   • ${step}${NC}"
-    done
-    echo ""
-    echo -e "${YELLOW}💡 Tips:${NC}"
-    echo -e "${YELLOW}   • Run 'ruff format ingestor/ tests/' to auto-fix formatting${NC}"
-    echo -e "${YELLOW}   • Run 'ruff check --fix ingestor/ tests/' to auto-fix linting issues${NC}"
-    echo -e "${YELLOW}   • Review mypy errors and add type hints where needed${NC}"
-    echo -e "${YELLOW}   • Check bandit warnings for security issues${NC}"
-    echo -e "${YELLOW}   • Fix failing tests or improve test coverage${NC}"
-    echo ""
-    exit 1
-fi