From 7352b1c1dfe02dffaa251587224d036d90773db0 Mon Sep 17 00:00:00 2001 From: Bruno Domingues Date: Wed, 5 Aug 2026 15:52:11 -0300 Subject: [PATCH] chore(devops): Update CI quality gate workflow and add Claude AI guidance --- .github/workflows/quality-gate.yml | 16 +++++++- CLAUDE.md | 65 ++++++++++++++++++++++++++++++ sonar-project.properties | 1 - 3 files changed, 80 insertions(+), 2 deletions(-) create mode 100644 CLAUDE.md diff --git a/.github/workflows/quality-gate.yml b/.github/workflows/quality-gate.yml index 5224d77..e023bd6 100644 --- a/.github/workflows/quality-gate.yml +++ b/.github/workflows/quality-gate.yml @@ -1,15 +1,29 @@ name: Quality gate on: + push: + branches: + - main + - 'release/**' + - 'feature/**' pull_request: branches: - main + - 'release/**' + - 'feature/**' types: [ opened, synchronize, reopened ] +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: quality-gate: uses: Aignosi/github_workflow_templates/.github/workflows/dataops-module-quality-gate.yml@main - permissions: write-all + permissions: + contents: read + pull-requests: write + issues: write with: project_name: 'orchestrator' repositories: 'sientia-dataops-library' diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..66e7092 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,65 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project + +SIENTIA DataOps Orchestrator — a Temporal.io workflow application for pipeline orchestration, real-time alerting, and scheduled reporting on the SIENTIA platform. Python 3.11+, package name `orchestrator`. + +## Commands + +Install deps: +```bash +pip install -r requirements.txt +pip install -r requirements-dev.txt # ruff, mypy, bandit, pytest, pytest-cov, pytest-asyncio +``` + +Run the worker locally (activates venv, loads `.env`, runs `python -m orchestrator.worker.worker`): +```bash +./run_local.sh +``` + +Tests: +```bash +pytest tests/ --cov=orchestrator --cov-report=html # full suite with coverage +pytest tests/orchestrator/activities/test_mongo_db.py # single file +pytest tests/orchestrator/activities/test_mongo_db.py::TestClass::test_name # single test +./run_coverage.sh # runs pytest w/ coverage, opens htmlcov/index.html +``` + +Lint/format/type/security (run individually — `validate.sh` referenced by README no longer exists in the repo, so don't rely on it): +```bash +ruff format --check orchestrator/ tests/ +ruff check orchestrator/ tests/ +mypy orchestrator/ +bandit -r orchestrator/ -ll + +# autofix +ruff format orchestrator/ tests/ +ruff check --fix orchestrator/ tests/ +``` + +CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable workflow `Aignosi/github_workflow_templates/.github/workflows/dataops-module-quality-gate.yml` on PRs to `main` — it's the source of truth for what must pass, not a local script. + +## Architecture + +### Layering +- **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`). +- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` vs all levels). +- **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections. +- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `converters.py` handles Temporal schedule frequency parsing. + +### Temporal-specific conventions (all workflow/activity files follow this) +- Every workflow and activity-consuming module wraps non-deterministic/side-effecting imports in `with workflow.unsafe.imports_passed_through(): ...` right after `from temporalio import workflow`, because Temporal's sandbox restricts what workflow code can import directly. When adding a new workflow file, follow the existing pattern (see `orchestrator/workflows/orchestrator.py`, `orchestrator/activities/activities.py`) rather than importing at module top level. +- Workflows are registered with `@workflow.defn(name='...')`; the `run` method is `@workflow.run`. Activity calls from within a workflow go through `workflow.start_local_activity_method` / `workflow.execute_activity_method` referencing `Activities.` (the class method, not an instance) with a `retry_policy` from `sientia_do.temporal.policies`. +- Task queue isolation is deliberate: don't move activities between the orchestrator/alerts/reports worker activity lists without checking which queue actually needs them — each list in `worker.py` is scoped to what that workflow (+ its subworkflows) calls. +- Namespaces: Temporal schedules span two namespaces, `scouter` and `laborious` (`TEMPORAL_SCOUTER_NAMESPACE` / `TEMPORAL_LABORIOUS_NAMESPACE`), handled inside `TemporalManager`. + +### External dependency +`sientia-dataops-library` (pinned via `requirements.txt` as a git dependency, currently `@1.8.2`) supplies `sientia_do.observability.logger`, `sientia_do.observability.metrics_controller`, `sientia_do.notifications.handlers`, `sientia_do.temporal.policies`, and `sientia_do.temporal.activities.postgres.Postgres`. It is a separate Aignosi repo — its version bump is a deliberate, explicit change, not incidental. + +### Tests +`tests/` mirrors `orchestrator/` (`tests/orchestrator/activities/`, `tests/orchestrator/utils/`, `tests/orchestrator/workflows/...`). Async tests use `pytest-asyncio`; coverage targets the `orchestrator` package. When testing Temporal workflows, follow the existing pattern in `tests/orchestrator/workflows/test_orchestrator.py` / `test_alerts.py` / `test_reports.py` (using Temporal's test env / activity mocking) rather than hitting real Temporal/Mongo/Redis/Postgres. + +### Ruff config notes (`pyproject.toml`) +Single quotes enforced by formatter. Notably ignored lint rules with reasons already documented inline: `B023` (closures over loop vars needed for schedule-action assembly), `BLE001` (blind `except` needed so activities can always notify on any error), `N802`/`N806` (Temporal decorators/variables don't follow standard naming), `S105`/`S106` (false-positive hardcoded-password matches). diff --git a/sonar-project.properties b/sonar-project.properties index 5a21e68..30cd868 100644 --- a/sonar-project.properties +++ b/sonar-project.properties @@ -7,5 +7,4 @@ sonar.qualitygate.timeout=300 sonar.python.coverage.reportPaths=coverage.xml sonar.python.xunit.reportPath=pytest.xml sonar.python.version=3.11 -sonar.projectVersion=1.0.0 sonar.coverage.exclusions=orchestrator/worker/*