41 lines
2.5 KiB
Markdown
41 lines
2.5 KiB
Markdown
# Two independent MinIO expiry mechanisms
|
|
|
|
This platform expires objects in MinIO through two mechanisms that were built independently, for
|
|
different features, and remain independent today. This note exists so a reader who finds one does
|
|
not assume it is the whole story.
|
|
|
|
## `ModelImport.ensure_import_bundle_retention`
|
|
|
|
`laborious/activities/model_import.py`. Applies a **bucket lifecycle rule** (a MinIO/S3-native
|
|
`Expiration` rule, `ENABLED`, keyed by a fixed `rule_id`) to the import bundle prefix
|
|
(`import_config['prefix']`), expiring objects after `import_config['retention_days']` days. The
|
|
rule is declarative: once set, MinIO itself removes the object when it ages out — the workflow does
|
|
not scan for or delete anything. The activity's job is only to ensure the rule exists and matches
|
|
the configured `days`/`prefix`, which is what makes "lifecycle active" a criterion the code can
|
|
verify rather than an assumption about a Helm value.
|
|
|
|
## `Storage.cleanup_minio_objects_expired`
|
|
|
|
`laborious/activities/storage.py`. An older, unrelated activity that **scans and deletes**: given a
|
|
list of key prefixes and a retention window (`retention_hours`), it lists objects under each prefix,
|
|
compares their age against `now() - retention_hours`, and deletes the ones that have aged out. There
|
|
is no bucket-side rule — the deletion happens in this code, on whatever schedule calls the activity.
|
|
|
|
## Why they are not unified today
|
|
|
|
They expire different prefix families, on different units (days vs. hours), through different
|
|
mechanisms (declarative rule vs. active scan), and were written for different features at different
|
|
times. Neither mechanism depends on or interferes with the other; a bucket can carry an
|
|
`ensure_import_bundle_retention` lifecycle rule for one prefix and still have
|
|
`cleanup_minio_objects_expired` scanning a different prefix on its own cadence.
|
|
|
|
## What unifying them would require
|
|
|
|
Either retiring the scan-and-delete path in favor of a lifecycle rule for every prefix it currently
|
|
handles (only possible if every one of its callers can tolerate a fixed number of retention *days*,
|
|
since S3-style lifecycle rules do not support hour-granularity windows), or teaching the
|
|
lifecycle-rule path to manage a per-prefix table of rules with arbitrary retention windows instead of
|
|
the one fixed rule it manages today. Either direction touches callers and configuration this change
|
|
does not, and neither is a refactor — this is tracked as a separate, larger change, not a task of
|
|
this one.
|