Files
sientia-dataops-orchestrato…/requirements.txt
Eduardo Rios 9c153041ad SIENTIAPDE-2013: cap load_latest_data's Mongo fetch at a configurable batch size
A cold start or notification storm had no backstop: load_latest_data
fetched the entire matching backlog in one query. Confirmed live during
SIENTIAPDE-2012's cluster verification -- an empty Redis cursor plus a
real recurring error let a single run's query exceed Temporal's 4MB
message limit and get stuck retrying forever.

Threads a new notification_queue_batch_size config (env var
NOTIFICATION_QUEUE_BATCH_SIZE, default 5000) through to the repository's
find(), hard cap per run, no drain loop -- the incremental
last_data_timestamp cursor picks up any remainder next run, per the
parent ticket's design decision. Sorts by timestamp ascending so a
capped run always returns the oldest pending documents first; without
that, a capped batch has no guarantee of hitting the oldest documents,
and the cursor would advance past whatever it skipped.

Requires sientia_do>=1.13.0 (adds limit/sort params to
MongoDBRepository.find in both the sync and async client) -- bumped the
pin here per the parent ticket's decision to do it in the same PR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 09:33:46 -03:00

9 lines
96 B
Plaintext

temporalio
psycopg2-binary
sqlalchemy
redis
pymongo
jinja2
sientia_do>=1.13.0
prometheus-client