chore: bump version to 0.6.7

SIENTIAPDE-2149: model_ids allow-list filter for receiver groups

CLAUDE.md

@@ -45,7 +45,7 @@ CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable
 
 ### Layering
 - **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`).
-- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further.
+- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further. Receiver groups also support a `model_ids` allow-list (SIENTIAPDE-2149), independent of the pre-existing `ignore_models_id` deny-list — both apply together (AND) when a group configures both, and `model_ids` follows the same union-at-query-time / precise-per-group-slice split as `levels`. `load_latest_data`'s Mongo query is capped at `NOTIFICATION_QUEUE_BATCH_SIZE` documents per run (default `0`, meaning unlimited — unchanged from pre-SIENTIAPDE-2013 behavior unless a deployment opts in), sorted by `timestamp` ascending so a capped run always returns the oldest pending batch first (SIENTIAPDE-2013) — a hard cap with no drain loop, since the incremental Redis cursor picks up any remainder on the next run.
 - **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections.
 - **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `converters.py` handles Temporal schedule frequency parsing.
 

e2e/conftest.py

@@ -330,6 +330,7 @@ async def test_activities(
             'connection_string': f'mongodb://localhost:{mongo_port}',
             'database_name': E2E_DATABASE,
             'ttl_index_seconds': 3600,
+            'notification_queue_batch_size': 0,
         },
         email_config={
             'sender_email': 'e2e@example.com',

e2e/helpers.py

@@ -347,6 +347,7 @@ def default_receiver_group(
     levels: list[str] | None = None,
     contents: list[str] | None = None,
     ignore_models_id: list[str] | None = None,
+    model_ids: list[str] | None = None,
 ) -> dict[str, Any]:
     """Minimal active receiver_groups document."""
     return {
@@ -356,6 +357,7 @@ def default_receiver_group(
         'levels': levels or ['ERROR', 'WARNING', 'INFO'],
         'contents': contents or ['core_alerts', 'persistent_alerts', 'reports'],
         'ignore_models_id': ignore_models_id or [],
+        'model_ids': model_ids or [],
     }
 
 

e2e/scenarios.md

@@ -68,6 +68,12 @@ the query-time union/intersection across active groups (SIENTIAPDE-2011, see
 `levels`/`ignore_models_id`, even when the query-time filter widened the fetch
 to satisfy another active group's broader configuration.
 
+### A.2.2 Model_id allow-list
+
+Receiver group `model_ids` (SIENTIAPDE-2149) honored: a group configuring it only
+receives notifications from those models, even when another active group with
+no such restriction widens the query-time fetch to include every model.
+
 ### A.3.1 Empty queue
 
 No notifications → no SMTP, no Postgres row.
@@ -99,7 +105,10 @@ No notifications → no SMTP, no Postgres row.
 - No prior Redis timestamp → all matching notifications returned, max timestamp stored.
 - Prior timestamp → only newer notifications returned.
 - Query-time filter (SIENTIAPDE-2011) → union of active groups' `levels`, intersection of their `ignore_models_id`; a model ignored by only one of several active groups is not excluded from the fetch.
+- Model_id allow-list union at query time (SIENTIAPDE-2149) → when every active group sets `model_ids`, the fetch is restricted to the union of every group's allow-list; a model outside that union is never fetched at all, even for a group that would otherwise want it via a different axis.
 - Empty Mongo → no Redis timestamp write.
+- Batch size cap (SIENTIAPDE-2013) → with 5 pending notifications and a batch size of 2, each run returns exactly the next-oldest 2 (sorted ascending by timestamp), and the Redis cursor advances only to what was actually returned, so three successive runs drain the backlog in order instead of skipping anything.
+- Batch size 0 means unlimited (SIENTIAPDE-2013) → the configured default (`0`) does not cap the query; 12 pending notifications all come back in one run, verified against a real MongoDB instance rather than assumed from pymongo's documented convention.
 
 ### ProcessNotifications — `e2e/test_subworkflow_process_notifications.py`
 

e2e/test_alerts_main_workflow.py

@@ -182,6 +182,73 @@ async def test_scenario_a_2_1_per_group_levels_and_ignore_models_id(
     assert 'model-x' not in combined
 
 
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_a_2_2_model_ids_allow_list(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    smtp_server,
+):
+    """
+    A.2.2 (SIENTIAPDE-2149): a group configuring `model_ids` only receives
+    notifications from those models, on top of the existing query-time/per-group
+    filtering (SIENTIAPDE-2011/2012).
+
+    group-allow (model_ids=['model-a']) coexists with group-everything (no
+    model_ids restriction). Because the query-time filter only restricts to an
+    allow-list union when EVERY active group has one, group-everything's lack
+    of a model_ids restriction means the query itself fetches both models --
+    the per-group filter added by this ticket is what narrows group-allow back
+    down to model-a only.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(
+                group_name='group-allow', contents=['core_alerts'], model_ids=['model-a']
+            ),
+            default_receiver_group(group_name='group-everything', contents=['core_alerts']),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id='err-a', level='ERROR', model_name='model-a', model_id='model-a'
+            ),
+            default_notification(
+                notification_id='err-b',
+                level='ERROR',
+                model_name='model-b',
+                model_id='model-b',
+                timestamp=datetime(2024, 6, 1, 12, 1, 0, tzinfo=UTC),
+            ),
+        ],
+    )
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Alerts.run,
+        load_scenario_input('alerts_happy_path.json'),
+        make_workflow_id('alerts-2149'),
+    )
+
+    # One email per group -- group-allow got only err-a, group-everything got both.
+    assert len(smtp_server.messages) == 2
+    bodies = [extract_html_body(message) for message in smtp_server.messages]
+
+    # Distinguish the two emails by content rather than order: only
+    # group-everything's email mentions model-b.
+    allow_body = next(b for b in bodies if 'model-b' not in b)
+    everything_body = next(b for b in bodies if 'model-b' in b)
+    assert 'model-a' in allow_body
+    assert 'model-a' in everything_body
+
+
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_scenario_a_3_1_empty_queue(

e2e/test_subworkflow_load_notification_package.py

@@ -205,6 +205,57 @@ async def test_load_package_query_time_filter_union_and_intersection(
     assert ids == {'error-included', 'warning-included', 'modely-included'}
 
 
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_query_time_filter_model_ids_union(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+):
+    """
+    SIENTIAPDE-2149: query-time filter unions active groups' `model_ids`
+    allow-list when every active group sets one, excluding any model outside
+    that union from the fetch entirely -- not just from a single group's
+    per-group slice.
+
+    group_a wants only model-a, group_b wants only model-b -- both restrict
+    levels to ERROR so only the model_ids axis is under test. model-c is
+    outside the union of both allow-lists and must never be fetched.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(group_name='group_a', levels=['ERROR'], model_ids=['model-a']),
+            default_receiver_group(group_name='group_b', levels=['ERROR'], model_ids=['model-b']),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(notification_id='a-included', level='ERROR', model_id='model-a'),
+            default_notification(notification_id='b-included', level='ERROR', model_id='model-b'),
+            default_notification(notification_id='c-excluded', level='ERROR', model_id='model-c'),
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-model-ids-union'),
+    )
+
+    ids = {n['notification_id'] for n in result['notification_package']}
+    assert ids == {'a-included', 'b-included'}
+
+
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_load_package_empty_mongo_no_redis_write(
@@ -228,3 +279,114 @@ async def test_load_package_empty_mongo_no_redis_write(
     )
 
     assert redis_client.get('notification_last_timestamp:Alerts') is None
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_batch_size_caps_and_returns_oldest_first(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+    test_activities,
+):
+    """
+    SIENTIAPDE-2013: a capped run returns the oldest pending documents first,
+    and the incremental cursor picks up the remainder on the next run.
+
+    Five notifications exist, but the batch size is capped at 2. Each run must
+    return exactly the next-oldest 2 (never an arbitrary subset), and the
+    Redis cursor must advance only to the max timestamp of what was actually
+    returned -- not the true max across all 5 -- or the skipped documents
+    would never be fetched again.
+    """
+    test_activities.notification_queue_batch_size = 2
+
+    seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id=f'n{i}',
+                timestamp=datetime(2024, 6, 1, 12, i, 0, tzinfo=UTC),
+            )
+            for i in range(1, 6)
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    first_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-1'),
+    )
+    assert [n['notification_id'] for n in first_run['notification_package']] == ['n1', 'n2']
+
+    second_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-2'),
+    )
+    assert [n['notification_id'] for n in second_run['notification_package']] == ['n3', 'n4']
+
+    third_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-3'),
+    )
+    assert [n['notification_id'] for n in third_run['notification_package']] == ['n5']
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_batch_size_zero_means_unlimited(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    test_activities,
+):
+    """
+    SIENTIAPDE-2013: notification_queue_batch_size=0 (the config default) must
+    NOT cap the query. This exercises pymongo's own documented convention --
+    `Collection.find(..., limit=0)` is equivalent to no limit at all -- against
+    a real MongoDB instance rather than trusting the convention on faith.
+
+    12 notifications is deliberately more than any other test in this file
+    seeds, so a regression that silently caps at some smaller default (e.g. if
+    `limit=0` were ever misread as "return nothing", or pymongo's own behavior
+    changed) would fail loudly here instead of passing by coincidence.
+    """
+    test_activities.notification_queue_batch_size = 0
+
+    seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id=f'n{i}',
+                timestamp=datetime(2024, 6, 1, 12, i, 0, tzinfo=UTC),
+            )
+            for i in range(1, 13)
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-unlimited'),
+    )
+
+    assert len(result['notification_package']) == 12

orchestrator/activities/activities.py

@@ -84,6 +84,7 @@ class Activities(TemporalManager, SlotManager, Formatters, MongoDB, Email, Postg
             connection_string=mongodb_config['connection_string'],
             database_name=mongodb_config['database_name'],
             ttl_index_seconds=mongodb_config['ttl_index_seconds'],
+            notification_queue_batch_size=mongodb_config['notification_queue_batch_size'],
             logger=logger,
             notification_handler=notification_handler,
             metrics_controller=metrics_controller,

orchestrator/activities/formatters.py

@@ -826,6 +826,7 @@ class Formatters(SientiaMonitoring):
                     - ignore (list, optional): Notification IDs to exclude
                     - levels (list, optional): Notification levels to include
                     - ignore_models_id (list, optional): Model IDs to exclude
+                    - model_ids (list, optional): Model IDs to allow (SIENTIAPDE-2149)
 
         Returns:
             dict[str, Any]: Filtered receiver groups with their notifications, keyed by group_name.
@@ -851,6 +852,7 @@ class Formatters(SientiaMonitoring):
             ignore_list = receiver_group.get('ignore', [])
             levels_list = receiver_group.get('levels')
             ignore_models_id_list = receiver_group.get('ignore_models_id') or []
+            model_ids_list = receiver_group.get('model_ids')
 
             for notification in notification_package:
                 alert_type = 'reports'
@@ -868,6 +870,7 @@ class Formatters(SientiaMonitoring):
                         not ignore_models_id_list
                         or notification.get('model_id') not in ignore_models_id_list
                     )
+                    and (not model_ids_list or notification.get('model_id') in model_ids_list)
                 ):
                     receiver_groups[group_name]['notifications'].append(notification)
                     already_added_keys.append(key)

orchestrator/activities/mongo_db.py

@@ -31,6 +31,8 @@ class MongoDB(SientiaMonitoring):
         ttl_index_seconds (int): TTL index duration in seconds
         logger (Logger): Application logger instance
         notification_handler (NotificationHandler): Notification management handler
+        notification_queue_batch_size (int): Hard cap on documents fetched per
+            `load_latest_data` run (SIENTIAPDE-2013). `0` means no limit.
     """
 
     def __init__(
@@ -41,6 +43,7 @@ class MongoDB(SientiaMonitoring):
         logger: Logger,
         notification_handler: NotificationHandler,
         metrics_controller: MetricsController,
+        notification_queue_batch_size: int = 0,
     ):
         SientiaMonitoring.__init__(
             self,
@@ -61,6 +64,7 @@ class MongoDB(SientiaMonitoring):
         )
 
         self.ttl_index_seconds = ttl_index_seconds
+        self.notification_queue_batch_size = notification_queue_batch_size
 
         # Initialize MongoDB client here (omitted for brevity)
         logger.info('MongoDB connection initialized')
@@ -447,6 +451,10 @@ class MongoDB(SientiaMonitoring):
         filtering by timestamp to enable incremental data processing. It
         handles connection management and provides comprehensive error reporting.
 
+        The query is capped at ``notification_queue_batch_size`` documents and sorted by
+        ``timestamp`` ascending, so a capped run returns the oldest pending documents first --
+        the caller advances its cursor only to what was actually returned, so nothing is skipped.
+
         Args:
             input_data (dict[str, Any]): Activity input parameters.
                 Required fields:
@@ -488,7 +496,13 @@ class MongoDB(SientiaMonitoring):
 
             self.debug(f'Data filter: {data_filter}', metadata=metadata)
 
-            data = self.mongo_db_repository.find(collection_name, data_filter, metadata)
+            data = self.mongo_db_repository.find(
+                collection_name,
+                data_filter,
+                metadata,
+                limit=self.notification_queue_batch_size,
+                sort=[('timestamp', 1)],
+            )
 
             self.debug(f'Collected: {data}', metadata=metadata)
 

orchestrator/activities/slot_manager.py

@@ -398,6 +398,7 @@ class SlotManager(SientiaMonitoring):
                     - ignore (list, optional): Notification IDs to exclude
                     - levels (list, optional): Notification levels to include
                     - ignore_models_id (list, optional): Model IDs to exclude
+                    - model_ids (list, optional): Model IDs to allow (SIENTIAPDE-2149)
                 - notification_ttl (int): Seconds before considering notification persistent
 
         Returns:
@@ -450,6 +451,7 @@ class SlotManager(SientiaMonitoring):
         ignore_list = receiver_group.get('ignore', [])
         levels_list = receiver_group.get('levels')
         ignore_models_id_list = receiver_group.get('ignore_models_id') or []
+        model_ids_list = receiver_group.get('model_ids')
 
         notifications = []
         for notification in notification_package:
@@ -462,8 +464,8 @@ class SlotManager(SientiaMonitoring):
                 and self._check_keys_and_notif_id(
                     notification_id, ignore_list, key, already_added_keys
                 )
-                and self._check_levels_and_ignore_models_id(
-                    notification, levels_list, ignore_models_id_list
+                and self._check_levels_and_model_filters(
+                    notification, levels_list, ignore_models_id_list, model_ids_list
                 )
             ):
                 notifications.append(notification)
@@ -493,23 +495,29 @@ class SlotManager(SientiaMonitoring):
         return 'do_nothing'
 
     @staticmethod
-    def _check_levels_and_ignore_models_id(notification, levels_list, ignore_models_id_list):
+    def _check_levels_and_model_filters(
+        notification, levels_list, ignore_models_id_list, model_ids_list
+    ):
         """
-        Check if the notification's level is in the levels_list and its model_id is not in the ignore_models_id_list.
+        Check the notification's level and model_id against a group's filters.
 
         Args:
             notification (dict): The notification to check.
             levels_list (list, optional): List of levels to include. If None, all levels are included.
             ignore_models_id_list (list, optional): List of model_ids to exclude. If None, no models are excluded.
+            model_ids_list (list, optional): List of model_ids to allow (SIENTIAPDE-2149).
+                If None/empty, all models are allowed. Independent of ignore_models_id_list --
+                both apply together when both are configured.
 
         Returns:
             bool: True if the notification passes the checks, False otherwise.
         """
         level_check = not levels_list or notification.get('level') in levels_list
-        model_id_check = (
+        ignore_models_id_check = (
             not ignore_models_id_list or notification.get('model_id') not in ignore_models_id_list
         )
-        return level_check and model_id_check
+        model_ids_check = not model_ids_list or notification.get('model_id') in model_ids_list
+        return level_check and ignore_models_id_check and model_ids_check
 
     @staticmethod
     def _check_keys_and_notif_id(

orchestrator/utils/connectors_config.py

@@ -21,7 +21,8 @@ def build_mongodb_config():
     Build MongoDB configuration from environment variables.
 
     Returns:
-        dict: MongoDB configuration with connection string, database name, and TTL index seconds.
+        dict: MongoDB configuration with connection string, database name, TTL index seconds,
+            and the notification queue's per-run batch size cap.
     """
     username = getenv('MONGODB_USERNAME', 'root')
     password = getenv('MONGODB_PASSWORD', 'wKZDbMNU1c')
@@ -32,6 +33,7 @@ def build_mongodb_config():
         'connection_string': connection_string,
         'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
         'ttl_index_seconds': int(getenv('MONGODB_TTL_INDEX_HOURS', '1')) * 3600,
+        'notification_queue_batch_size': int(getenv('NOTIFICATION_QUEUE_BATCH_SIZE', '0')),
     }
 
 

orchestrator/workflows/subworkflows/load_notification_package.py

@@ -137,18 +137,21 @@ class LoadNotificationPackage:
           is only safe to drop from the fetch if EVERY active group ignores it.
           A group with no `ignore_models_id` is treated as "ignores nothing" (empty
           set), which naturally zeroes out the intersection via set math.
+        - model_ids (SIENTIAPDE-2149): union of every group's `model_ids` allow-list,
+          mirroring `levels`' own union-or-nothing logic. If ANY active group has no
+          restriction (missing/empty `model_ids`), that group wants every model, so
+          the query can't restrict to an allow-list at all. `model_ids` and
+          `ignore_models_id` are independent axes and can both narrow the same
+          `model_id` field at once (Mongo supports multiple operators on one field).
         """
         if not sending_configs:
             return default_filter
 
-        sending_configs_levels = [group.get('levels') for group in sending_configs]
-
-        if all(sending_configs_levels):
-            levels_union: set[str] = set()
-            for levels in sending_configs_levels:
-                if levels:
-                    levels_union.update(levels)
-            query_filter = {**default_filter, 'level': {'$in': sorted(levels_union)}}
+        levels_union = self._union_when_all_restrict(
+            [group.get('levels') for group in sending_configs]
+        )
+        if levels_union is not None:
+            query_filter = {**default_filter, 'level': {'$in': levels_union}}
         else:
             query_filter = dict(default_filter)
 
@@ -160,7 +163,40 @@ class LoadNotificationPackage:
         ]
         models_intersection = set.intersection(*sending_configs_ignore_models_id)
 
+        model_ids_union = self._union_when_all_restrict(
+            [group.get('model_ids') for group in sending_configs]
+        )
+
+        model_id_filter: dict[str, Any] = {}
+        if model_ids_union is not None:
+            model_id_filter['$in'] = model_ids_union
         if models_intersection:
-            query_filter['model_id'] = {'$nin': sorted(models_intersection)}
+            model_id_filter['$nin'] = sorted(models_intersection)
+        if model_id_filter:
+            query_filter['model_id'] = model_id_filter
 
         return query_filter
+
+    @staticmethod
+    def _union_when_all_restrict(per_group_values: list[list[str] | None]) -> list[str] | None:
+        """
+        Union a list of per-group optional restrictions, but only when every group has one.
+
+        Shared by `levels` and `model_ids` (SIENTIAPDE-2149) in `_compute_query_time_filter`,
+        which both follow the same rule: if ANY active group is unrestricted on that axis
+        (missing/empty value), that group wants everything, so the union can't restrict the
+        query either — the caller falls back to no filter on that axis in that case (`None`).
+
+        Args:
+            per_group_values: Each active group's raw value for the axis being unioned
+                (e.g. `group.get('levels')` for every group).
+
+        Returns:
+            Sorted union of every group's values, or None if any group was unrestricted.
+        """
+        if not all(per_group_values):
+            return None
+        union: set[str] = set()
+        for values in per_group_values:
+            union.update(values or [])
+        return sorted(union)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "orchestrator"
-version = "0.6.5"
+version = "0.6.7"
 description = "Sientia DataOps Orchestrator - ML Model Orchestration System"
 readme = "README.md"
 requires-python = ">=3.11"

requirements-local.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.2
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.3
 prometheus-client

requirements.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-sientia_do>=1.12.2
+sientia_do>=1.12.3
 prometheus-client

tests/orchestrator/activities/test_activities.py

@@ -27,6 +27,7 @@ def test___init__(
         'connection_string': 'mongodb://localhost:27017',
         'database_name': 'test_db',
         'ttl_index_seconds': 3600,
+        'notification_queue_batch_size': 5000,
     }
 
     redis_config = {'host': 'localhost', 'port': 6379, 'username': 'admin', 'password': 'password'}
@@ -89,6 +90,7 @@ def test___init__(
         connection_string='mongodb://localhost:27017',
         database_name='test_db',
         ttl_index_seconds=3600,
+        notification_queue_batch_size=5000,
         logger=logger,
         notification_handler=notification_handler,
         metrics_controller=mock_metrics_controller.return_value,

tests/orchestrator/activities/test_formatters.py

@@ -852,6 +852,123 @@ def test_filter_notification_reports_ignore_models_id_restriction(formatters):
     }
 
 
+def test_filter_notification_reports_model_ids_allow_list_restriction(formatters):
+    # SIENTIAPDE-2149: a group with `model_ids` set only receives notifications
+    # from those models, unlike `ignore_models_id` which is a deny-list.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['reports'],
+                'model_ids': ['model_x'],
+            },
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['reports'],
+            'model_ids': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'model_id': 'model_x',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_reports_model_ids_and_ignore_models_id_together(formatters):
+    # SIENTIAPDE-2149: the allow-list and deny-list apply together (AND), not
+    # as alternatives -- a model must pass both checks.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+            {
+                'trigger': 'test_trigger_3',
+                'notification_id': 'test_notification_id_3',
+                'model_id': 'model_z',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['reports'],
+                'model_ids': ['model_x', 'model_y'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['reports'],
+            'model_ids': ['model_x', 'model_y'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_reports_model_ids_explicit_none_does_not_raise(formatters):
+    # Same defensive concern as ignore_models_id: receiver_groups has no schema
+    # validation, so model_ids can be explicitly null instead of omitted.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['reports'], 'model_ids': None},
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
 def test_filter_notification_reports_no_levels_or_ignore_models_id_passes_everything(
     formatters,
 ):

tests/orchestrator/activities/test_mongo_db.py

@@ -31,13 +31,14 @@ def test___init__(mongo_mock):
     notification_handler = MagicMock()
     metrics_controller = AsyncMock()
 
-    MongoDB(
+    mongo = MongoDB(
         connection_string='mongodb://localhost:27017',
         database_name='test_db',
         ttl_index_seconds=3600,
         logger=logger,
         notification_handler=notification_handler,
         metrics_controller=metrics_controller,
+        notification_queue_batch_size=5000,
     )
 
     mongo_mock.assert_called_once_with(
@@ -47,6 +48,22 @@ def test___init__(mongo_mock):
         notification_handler=notification_handler,
         metrics_controller=metrics_controller,
     )
+    assert mongo.notification_queue_batch_size == 5000
+
+
+@patch('orchestrator.activities.mongo_db.MongoDBRepository')
+def test___init___notification_queue_batch_size_defaults_to_zero(mongo_mock):
+    """0 means no limit, matching the underlying find()'s own convention."""
+    mongo = MongoDB(
+        connection_string='mongodb://localhost:27017',
+        database_name='test_db',
+        ttl_index_seconds=3600,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+
+    assert mongo.notification_queue_batch_size == 0
 
 
 @patch('orchestrator.activities.mongo_db.SientiaMonitoring')
@@ -480,6 +497,8 @@ def test_load_latest_data_none_last_data_timestamp(mongo_db):
         'test_collection',
         {'level': 'ERROR'},
         {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=0,
+        sort=[('timestamp', 1)],
     )
 
     assert result == [
@@ -487,6 +506,29 @@ def test_load_latest_data_none_last_data_timestamp(mongo_db):
     ]
 
 
+def test_load_latest_data_passes_configured_batch_size_as_limit(mongo_db):
+    """SIENTIAPDE-2013: the configured notification_queue_batch_size caps the query."""
+    mongo_db.notification_queue_batch_size = 100
+    mongo_db.mongo_db_repository.find = MagicMock(return_value=[])
+
+    mongo_db.load_latest_data(
+        {
+            'metadata': {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+            'collection_name': 'test_collection',
+            'last_data_timestamp': None,
+            'base_data_filter': {'level': 'ERROR'},
+        }
+    )
+
+    mongo_db.mongo_db_repository.find.assert_called_once_with(
+        'test_collection',
+        {'level': 'ERROR'},
+        {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=100,
+        sort=[('timestamp', 1)],
+    )
+
+
 def test_load_latest_data_not_none_last_data_timestamp(mongo_db):
     """Test load_latest_data compares native datetime against native datetime in $gt"""
 
@@ -516,6 +558,8 @@ def test_load_latest_data_not_none_last_data_timestamp(mongo_db):
             'timestamp': {'$gt': datetime(2023, 1, 1, 12, 0, 0, tzinfo=UTC)},
         },
         {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=0,
+        sort=[('timestamp', 1)],
     )
 
     assert result == [
@@ -529,7 +573,7 @@ def test_load_latest_data_run_boundary_no_skip_or_duplicate(mongo_db):
 
     boundary_timestamp = datetime(2023, 1, 1, 12, 0, 0, tzinfo=UTC)
 
-    def fake_find(collection_name, data_filter, metadata):
+    def fake_find(collection_name, data_filter, metadata, limit=0, sort=None):
         gt = data_filter.get('timestamp', {}).get('$gt')
         all_docs = [
             {

tests/orchestrator/activities/test_slot_manager.py

@@ -481,6 +481,137 @@ def test_filter_notification_alerts_ignore_models_id_restriction(slot_manager):
     }
 
 
+def test_filter_notification_alerts_model_ids_allow_list_restriction(slot_manager):
+    # SIENTIAPDE-2149: a group with `model_ids` set only receives notifications
+    # from those models, unlike `ignore_models_id` which is a deny-list.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'model_ids': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'model_ids': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'model_id': 'model_x',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_and_ignore_models_id_together(slot_manager):
+    # SIENTIAPDE-2149: the allow-list and deny-list apply together (AND), not
+    # as alternatives -- a model must pass both checks.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+            {
+                'trigger': 'test_trigger_3',
+                'notification_id': 'test_notification_id_3',
+                'model_id': 'model_z',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'model_ids': ['model_x', 'model_y'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    # model_x: allowed but ignored -> excluded. model_y: allowed and not
+    # ignored -> included. model_z: not in the allow-list -> excluded.
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'model_ids': ['model_x', 'model_y'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_explicit_none_does_not_raise(slot_manager):
+    # Same defensive concern as ignore_models_id: receiver_groups has no schema
+    # validation, so model_ids can be explicitly null instead of omitted.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'model_ids': None},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
 def test_filter_notification_alerts_no_levels_or_ignore_models_id_passes_everything(
     slot_manager,
 ):

tests/orchestrator/utils/test_connectors_config.py

@@ -64,10 +64,12 @@ def test_build_mongo_db_config_with_env_vars():
     environ['MONGODB_URL'] = 'localhost:27018'
     environ['MONGODB_DATABASE_NAME'] = 'test_db'
     environ['MONGODB_TTL_INDEX_HOURS'] = '2'
+    environ['NOTIFICATION_QUEUE_BATCH_SIZE'] = '2000'
     assert build_mongodb_config() == {
         'connection_string': 'mongodb://sientia1:sientia1@localhost:27018',
         'database_name': 'test_db',
         'ttl_index_seconds': 7200,
+        'notification_queue_batch_size': 2000,
     }
 
 
@@ -77,10 +79,12 @@ def test_build_mongo_db_config_with_defaults():
     environ.pop('MONGODB_DATABASE_NAME', None)
     environ.pop('MONGODB_URL', None)
     environ.pop('MONGODB_TTL_INDEX_HOURS', None)
+    environ.pop('NOTIFICATION_QUEUE_BATCH_SIZE', None)
     assert build_mongodb_config() == {
         'connection_string': 'mongodb://root:wKZDbMNU1c@localhost:27018',
         'database_name': 'sientia',
         'ttl_index_seconds': 3600,
+        'notification_queue_batch_size': 0,
     }
 
 

tests/orchestrator/workflows/subworkflows/test_load_notification_package.py

@@ -237,6 +237,65 @@ def test_compute_query_time_filter_model_ignored_by_only_one_group_not_excluded(
     assert 'model_id' not in result
 
 
+def test_compute_query_time_filter_model_ids_union_when_all_groups_restrict(
+    load_notification_package,
+):
+    # SIENTIAPDE-2149: every active group sets a model_ids allow-list -> the
+    # union applies at query time, mirroring how `levels` already behaves.
+    sending_configs = [
+        {'group_name': 'group_a', 'model_ids': ['model_a']},
+        {'group_name': 'group_b', 'model_ids': ['model_b']},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'model_id': {'$in': ['model_a', 'model_b']}}
+
+
+def test_compute_query_time_filter_model_ids_no_restriction_when_any_group_missing(
+    load_notification_package,
+):
+    # One active group has no model_ids restriction -> it wants every model,
+    # so the query can't restrict to an allow-list at all.
+    sending_configs = [
+        {'group_name': 'group_a', 'model_ids': ['model_a']},
+        {'group_name': 'group_b'},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert 'model_id' not in result
+
+
+def test_compute_query_time_filter_model_ids_union_and_ignore_models_id_intersection_together(
+    load_notification_package,
+):
+    # SIENTIAPDE-2149: the allow-list union ($in) and the ignore-list
+    # intersection ($nin) are independent axes and can both narrow the same
+    # model_id field at once.
+    sending_configs = [
+        {
+            'group_name': 'group_a',
+            'model_ids': ['model_a', 'model_b'],
+            'ignore_models_id': ['model_a'],
+        },
+        {
+            'group_name': 'group_b',
+            'model_ids': ['model_b', 'model_c'],
+            'ignore_models_id': ['model_a'],
+        },
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {
+        'model_id': {'$in': ['model_a', 'model_b', 'model_c'], '$nin': ['model_a']},
+    }
+
+
 def test_compute_query_time_filter_union_levels_and_intersection_models(
     load_notification_package,
 ):