Files
sientia-dataops-orchestrato…/orchestrator/utils
Eduardo Rios 9c153041ad SIENTIAPDE-2013: cap load_latest_data's Mongo fetch at a configurable batch size
A cold start or notification storm had no backstop: load_latest_data
fetched the entire matching backlog in one query. Confirmed live during
SIENTIAPDE-2012's cluster verification -- an empty Redis cursor plus a
real recurring error let a single run's query exceed Temporal's 4MB
message limit and get stuck retrying forever.

Threads a new notification_queue_batch_size config (env var
NOTIFICATION_QUEUE_BATCH_SIZE, default 5000) through to the repository's
find(), hard cap per run, no drain loop -- the incremental
last_data_timestamp cursor picks up any remainder next run, per the
parent ticket's design decision. Sorts by timestamp ascending so a
capped run always returns the oldest pending documents first; without
that, a capped batch has no guarantee of hitting the oldest documents,
and the cursor would advance past whatever it skipped.

Requires sientia_do>=1.13.0 (adds limit/sort params to
MongoDBRepository.find in both the sync and async client) -- bumped the
pin here per the parent ticket's decision to do it in the same PR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 09:33:46 -03:00
..
2025-07-29 10:25:31 -03:00
2025-05-30 16:24:52 -03:00
2025-10-15 17:20:52 -03:00
2025-11-17 16:30:29 -03:00
2026-05-22 15:02:16 -03:00