SIENTIAPDE-1712
SIENTIAPDE-1712 Add last_timestamp parameter to MLFlow and Gates activities for enhanced tracking - Introduced last_timestamp parameter in the MLFlow and Gates classes to improve tracking of data processing times. - Updated MinioDataFramePayload to handle last_timestamp, ensuring it defaults to the maximum timestamp from the dataframe if not provided.
This commit is contained in:
@@ -173,6 +173,7 @@ class MinioDataFramePayload:
|
||||
operation: OperationKind,
|
||||
status: dict[str, Any] | None = None,
|
||||
workflow_metadata: dict | None = None,
|
||||
last_timestamp: str | None = None,
|
||||
) -> 'MinioDataFramePayload':
|
||||
"""
|
||||
Evaluate the DataFrame size, then either inline dict or upload parquet to MinIO.
|
||||
@@ -199,7 +200,9 @@ class MinioDataFramePayload:
|
||||
data=None, last_timestamp=now().strftime(DATETIME_FORMAT_WITH_TZ), status=status
|
||||
)
|
||||
|
||||
last_timestamp = max(dataframe['timestamp'].values.tolist())
|
||||
|
||||
if last_timestamp is None:
|
||||
last_timestamp = max(dataframe['timestamp'].values.tolist())
|
||||
|
||||
if cls.estimate_size_bytes(dataframe) <= OFFLOAD_THRESHOLD_BYTES:
|
||||
return cls(data=dataframe.to_dict(), last_timestamp=last_timestamp)
|
||||
|
||||
Reference in New Issue
Block a user