gitea_admin/sientia-dataops-orchestrator_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1222

Browse Source 
Refactor orchestration and email handling

- Updated the orchestration queries to streamline model lookups and added conditions for active models.
- Changed the entry point in the local run script to use the worker module.
- Removed the deprecated samples.json file.
- Enhanced the Email activity to skip sending if the SMTP server is not configured.
- Added timestamp field handling in MongoDB activities for better data management.
- Updated connectors configuration to allow for a None SMTP server.
- Improved the common configuration function to include model configuration details.
This commit is contained in:
vitor-aignosi
2025-09-12 14:39:42 -03:00
parent 5db411e5b5
commit 137367c963
11 changed files with 92 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
init_orchestration.ipynb
View File

@@ -23,7 +23,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"id": "d9d2a242",
"metadata": {},
"outputs": [
@@ -75,40 +75,28 @@
" \"pipelines_query\": {\n",
" \"collection\": \"pipelines\",\n",
" \"aggregation\": [\n",
" {\n",
" \"$lookup\": {\n",
" \"from\": \"models\",\n",
" \"localField\": \"model_id\",\n",
" \"foreignField\": \"id\",\n",
" \"as\": \"model_docs\"\n",
" }\n",
" },\n",
" {\n",
" \"$match\": {\n",
" \"active\": True\n",
" }\n",
" },\n",
" {\n",
" \"$addFields\": {\n",
" \"models\": {\n",
" \"$arrayElemAt\": [\n",
" \"$model_docs\",\n",
" 0\n",
" ]\n",
" {\n",
" \"$lookup\": {\n",
" \"from\": \"models\",\n",
" \"localField\": \"model_id\",\n",
" \"foreignField\": \"id\",\n",
" \"as\": \"model\"\n",
" }\n",
" },\n",
" {\n",
" \"$unwind\": \"$model\"\n",
" },\n",
" {\n",
" \"$match\": {\n",
" \"active\": True\n",
" }\n",
" },\n",
" {\n",
" \"$match\": {\n",
" \"model.active\": True\n",
" }\n",
" }\n",
" }\n",
" },\n",
" {\n",
" \"$match\": {\n",
" \"models.active\": True\n",
" }\n",
" },\n",
" {\n",
" \"$project\": {\n",
" \"model_docs\": 0\n",
" }\n",
" }\n",
" ]\n",
" ]\n",
" },\n",
" \"opc_servers_query\": {\n",
" \"collection\": \"opc-servers\",\n",