SIENTIAPDE-1222

Enhance README and activities with advanced notification filtering features

- Updated README to include new intelligent notification filtering capabilities, including TTL-based duplicate prevention and ignore lists.
- Enhanced `Formatters` activity to support notification filtering for scheduled reports and added detailed filtering logic.
- Improved `SlotManager` to implement advanced notification filtering with persistent alert detection and group-based filtering.
- Updated `Alerts` workflow to incorporate intelligent filtering for real-time error notifications, ensuring efficient alert delivery.

README.md

@@ -13,10 +13,15 @@ A high-performance, scalable workflow orchestration system built on Temporal.io
 
 ### Advanced Capabilities
 - **Incremental Processing**: Timestamp-based data loading to avoid reprocessing
-- **Configurable Filtering**: User group-based notification filtering with custom policies
+- **Intelligent Notification Filtering**: Advanced filtering system with:
+  - User group-based notification filtering with custom policies
+  - TTL-based duplicate prevention for alerts
+  - Ignore lists for specific notifications
+  - Persistent alert detection for ongoing issues
 - **Auto-scaling Workers**: Multiple worker instances with task queue isolation
 - **Comprehensive Logging**: Structured logging with PostgreSQL audit trails
 - **Prometheus Metrics**: Real-time monitoring and alerting integration
+- **Template-based Email Generation**: Jinja2-powered HTML email templates
 
 ## Architecture
 
@@ -252,6 +257,12 @@ flowchart LR
 - `notification_package` (list[dict]): Retrieved notifications
 - `sending_configs` (list[dict]): Active receiver group configurations
 
+**Key Activities**:
+- `get_last_data_timestamp`: Retrieves last processed timestamp from Redis
+- `find_documents_in_mongodb`: Loads receiver group configurations
+- `load_latest_data`: Loads notifications with timestamp filtering
+- `put_last_data_timestamp`: Updates last processed timestamp
+
 **Architecture**:
 
 ```mermaid
@@ -277,10 +288,11 @@ flowchart LR
 **Purpose**: Handles email generation, delivery, and audit logging for notification workflows.
 
 **Key Features**:
-- **HTML Generation**: Creates formatted email content for each receiver group
+- **HTML Generation**: Creates formatted email content for each receiver group using Jinja2 templates
 - **Email Delivery**: Sends emails with attachment support and error handling
 - **Audit Logging**: Records delivery status and metrics in PostgreSQL
 - **Error Recovery**: Handles SMTP failures with detailed error reporting
+- **Template Support**: Uses customizable HTML templates for different email types
 
 **Input Parameters**:
 ```json
@@ -300,6 +312,12 @@ flowchart LR
 **Returns**:
 - `log_report` (list[dict]): Detailed delivery status for each notification
 
+**Key Activities**:
+- `build_email_html`: Generates HTML content using Jinja2 templates
+- `send_email`: Delivers emails to receiver groups with error handling
+- `format_log_report`: Formats delivery results for database storage
+- `export_data_to_postgres`: Stores audit logs in PostgreSQL
+
 **Architecture**:
 
 ```mermaid
@@ -313,6 +331,29 @@ flowchart LR
 
 ```
 
+## 🔔 Notification Filtering System
+
+The orchestrator includes an advanced notification filtering system that prevents alert spam and ensures relevant notifications reach the appropriate user groups.
+
+### **Alert Filtering (`filter_notification_alerts`)**
+- **Purpose**: Filters error-level notifications for immediate alerts
+- **TTL Management**: Prevents duplicate alerts using configurable time-to-live settings
+- **Persistent Detection**: Identifies ongoing issues that require escalation
+- **Group-based Filtering**: Routes notifications to appropriate receiver groups
+- **Ignore Lists**: Supports notification exclusion per group
+
+### **Report Filtering (`filter_notification_reports`)**
+- **Purpose**: Filters all notification levels for comprehensive reports
+- **Comprehensive Coverage**: Includes ERROR, WARNING, INFO, and DEBUG levels
+- **Group Customization**: Applies different content policies per receiver group
+- **Scheduled Processing**: Designed for regular report generation
+
+### **Notification Caching (`store_notification_cache`)**
+- **Purpose**: Manages Redis-based notification cache for TTL enforcement
+- **TTL Support**: Configurable expiration times for different notification types
+- **Duplicate Prevention**: Ensures notifications aren't sent repeatedly within TTL window
+- **Key Management**: Uses structured keys for efficient cache lookups
+
 ### Key Components
 
 #### **Worker (`orchestrator/worker/worker.py`)**
@@ -332,19 +373,23 @@ flowchart LR
 #### **Activities (`orchestrator/activities/`)**
 - **Activities**: Main activity orchestrator combining all operations
 - **TemporalManager**: Temporal schedule CRUD operations across namespaces
-- **SlotManager**: Redis-based OPC slot and cache management
+- **SlotManager**: Redis-based OPC slot and cache management with notification filtering
 - **MongoDB**: Document operations, aggregations, and TTL management
 - **Email**: SMTP operations with HTML generation and attachment support
-- **Formatters**: Configuration processing and slot distribution algorithms
+- **Formatters**: Configuration processing, slot distribution algorithms, and notification filtering for reports
+- **Postgres**: PostgreSQL operations for audit logging and data export (via sientia-dataops-library)
+- **Couchbase**: Database operations (currently unused but maintained for future use)
 
 #### **Utilities (`orchestrator/utils/`)**
 - **Connectors Configuration**: Database and service configuration management
-- **Email Builder**: HTML email template generation and formatting
-- **Orchestrator Functions**: Pipeline configuration transformation utilities
-- **Converters**: Data type conversion and validation utilities
+- **Email Builder**: HTML email template generation and formatting using Jinja2 templates
+- **Orchestrator Functions**: Pipeline configuration transformation utilities for scouter, predictions_batch, and minimal_retrain workflows
+- **Converters**: Data type conversion and validation utilities including frequency parsing
+- **Templates**: HTML email templates for alerts and reports
 
 ## 📋 Prerequisites
 
+### System Requirements
 - Python 3.11+
 - Temporal server/cluster
 - Redis server
@@ -461,6 +506,12 @@ python -m orchestrator.worker.worker
 | `EMAIL_SMTP_PORT` | SMTP port | `587` | No |
 | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
 | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
+| `POSTGRES_MIN_CONNECTIONS` | PostgreSQL minimum connections | `10` | No |
+| `POSTGRES_MAX_CONNECTIONS` | PostgreSQL maximum connections | `40` | No |
+| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index expiration in hours | `1` | No |
+| `PROJECT_NAME` | Project name for metrics and logging | `sientia-orchestrator` | No |
+| `LOG_LEVEL` | Application logging level | `INFO` | No |
+| `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
 
 ### Workflow Configuration
 
@@ -560,11 +611,28 @@ The Orchestrator system exposes comprehensive Prometheus metrics:
 ```
 tests/
 ├── orchestrator/           # Orchestrator workflow tests
+│   ├── test_activities.py
+│   ├── test_couchbase.py
+│   ├── test_email.py
+│   ├── test_formatters.py
+│   ├── test_mongo_db.py
+│   ├── test_slot_manager.py
+│   ├── test_temporal_manager.py
+│   └── test_workflows.py
 ├── activities/            # Activity implementation tests
 ├── utils/                 # Utility function tests
 └── integration/           # End-to-end workflow tests
 ```
 
+### Test Coverage
+The project maintains comprehensive test coverage including:
+- **Activity Tests**: Unit tests for all activity classes
+- **Workflow Tests**: Integration tests for workflow orchestration
+- **Utility Tests**: Tests for configuration builders and converters
+- **Database Tests**: Tests for MongoDB, Redis, and PostgreSQL operations
+- **Email Tests**: Tests for email generation and delivery
+- **Notification Tests**: Tests for filtering and caching logic
+
 ### Test Execution
 ```bash
 # Install test dependencies
@@ -586,21 +654,28 @@ orchestrator/
 ├── activities/              # Temporal activity implementations
 │   ├── activities.py       # Main activities orchestrator
 │   ├── temporal_manager.py # Temporal schedule operations
-│   ├── slot_manager.py     # Redis slot management
+│   ├── slot_manager.py     # Redis slot management and notification filtering
 │   ├── mongo_db.py         # MongoDB operations
 │   ├── email.py            # Email service operations
-│   └── formatters.py       # Configuration formatting
+│   ├── formatters.py       # Configuration formatting and report filtering
+│   └── couchbase.py        # Couchbase operations (currently unused)
 ├── workflows/               # Temporal workflow definitions
 │   ├── orchestrator.py     # Main orchestration workflow
 │   ├── alerts.py           # Error alert workflow
 │   ├── reports.py          # Scheduled report workflow
 │   └── subworkflows/       # Sub-workflow implementations
+│       ├── load_notification_package.py  # Notification data loading
+│       └── process_notifications.py      # Email processing and delivery
 ├── worker/                  # Worker implementation
 │   └── worker.py           # Main worker orchestrator
 ├── utils/                   # Utility functions
 │   ├── connectors_config.py # Database configuration
 │   ├── email_builder.py    # Email template generation
-│   └── orchestrator_functions.py # Pipeline utilities
+│   ├── orchestrator_functions.py # Pipeline utilities
+│   ├── converters.py       # Data type conversion utilities
+│   └── templates/          # Email HTML templates
+│       ├── email_template.html    # Report email template
+│       └── general_template.html  # General email template
 └── metrics.py               # Prometheus metrics definitions
 ```