12 Commits

Author SHA1 Message Date
dc9fd871bb fix(ci): restaura pull-requests:read no deploy.yml, exigido pelo release-tarball
reusable-deploy-dataops.yml encadeia release-tarball -> dataops-module-release.yml,
que pede pull-requests:read; GitHub valida a interseção de permissions em todo o
workflow_call encadeado, independente do if: que gate o job. Removido por engano
em 586a5b9, achando que só existia por causa do trigger pull_request antigo.

[skip ci]
2026-10-21 14:10:25 -03:00
242566f9b1 fix(ci): grant contents:write permission to version-bump workflow
Caller workflow had no permissions block, so GitHub's permission-
intersection rule capped the nested reusable-version-bump.yml call
to read-only, breaking dispatch ("requesting 'contents: write', but
is only allowed 'contents: read'").

[skip ci]
2026-10-21 11:44:53 -03:00
d0b4644f63 chore(ci): add version-bump.yml, decoupled from deploy
New workflow bumps pyproject.toml's version on PR merge, calling the
shared reusable-version-bump.yml (SIENTIAPDE-2217 version-bump reform) —
zero build/image/chart/deploy side effects. deploy.yml (workflow_dispatch-
only) will read this committed value instead of calculating one.

[skip ci]
2026-10-21 00:18:51 -03:00
ced847bbe7 chore(ci): switch deploy.yml to manual workflow_dispatch with required chart_ref
Removes auto-deploy-on-PR-merge trigger and the old PR-merge gate. Deploy now
triggers only via workflow_dispatch, with chart_ref required (no default) as
the literal sientia-helm-chart branch bumped and deployed — hard error if it
doesn't exist, no silent fallback to development/main.

- .github/workflows/deploy.yml: drop pull_request trigger, add chart_ref
  input, branch_name now github.ref_name, drop pull-requests:read permission
- Delete .github/workflows/require-chart-ref-label.yml
- README.md: update Deploy section to describe manual trigger

[skip ci]
2026-10-20 17:03:33 -03:00
49c8942a87 chore(ci): remove feature/** do push trigger (evita run duplicado) [skip ci] 2026-10-19 15:38:47 -03:00
github-actions[bot]
3b90812c12 chore: bump version to 2.0.3 2026-10-16 15:10:26 +00:00
0b673b35a1 Merge pull request #33 from Aignosi/fix/SIENTIAPDE-2201
SIENTIAPDE-2201: Keep the timestamp column in train/test data CSVs
2026-10-16 12:10:01 -03:00
PedroHMCosme
37f16f34ff SIENTIAPDE-2201: fix: keep the timestamp column in train/test data CSVs
generate_report wrote train_data.csv and test_data.csv with index=False, which
dropped the UTC time index. laborious drift reads test_data.csv from MLflow as
its reference and needs a timestamp column; without it, every model trained here
drifted against the first 30% of its own target window. Both CSVs now write the
index as timestamp. If a column already has that name, the CSV is written as
before and a warning is logged, so training never fails over drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 10:09:06 -03:00
dd981b8027 fix(ci): Update env_list_path to array format in deploy workflow
This commit changes the env_list_path parameter in the deploy GitHub Actions workflow to an array format, aligning with expected input types for the action.
2026-09-11 00:41:42 -03:00
github-actions[bot]
5641734c3c chore: bump version to 2.0.2 2026-09-10 13:48:02 +00:00
Bruno Domingues
c51466f304 Merge pull request #32 from Aignosi/fix/SIENTIAPDE-2068
Configure deploy workflow for automated version updates
2026-09-10 10:47:41 -03:00
44a4ffe6d0 SIENTIAPDE-2068: Configure deploy workflow to update version in pyproject.toml. 2026-09-10 10:39:49 -03:00
8 changed files with 113 additions and 27 deletions

View File

@@ -1,10 +1,15 @@
name: Deploy DataOps Worker
on:
pull_request:
types: [closed]
branches: [main, 'release/**', 'feature/**']
workflow_dispatch:
inputs:
chart_ref:
description: >
sientia-helm-chart branch to bump GITHUB_BRANCH on AND deploy — the literal
deploy target. Validated via git ls-remote; a branch that does not exist in
sientia-helm-chart is a hard error, not a silent fallback to main.
required: true
type: string
permissions:
contents: write
@@ -14,13 +19,14 @@ permissions:
jobs:
deploy:
name: Deploy
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.merged == true
uses: Aignosi/github_workflow_templates/.github/workflows/reusable-deploy-dataops.yml@main
with:
branch_name: ${{ github.event.pull_request.head.ref }}
branch_name: ${{ github.ref_name }}
chart_ref: ${{ inputs.chart_ref }}
project_name: 'model-manager'
chart_name: 'model-manager'
env_list_path: '.global.env'
env_list_path: '[".global.env"]'
update_version_in: '["pyproject"]'
create_tarball: ${{ github.event_name == 'pull_request' }}
mirror: true
bump_chart: true

View File

@@ -5,7 +5,6 @@ on:
branches:
- main
- 'release/**'
- 'feature/**'
pull_request:
branches:
- main

View File

@@ -1,16 +0,0 @@
name: Require chart-ref label
on:
pull_request:
types: [opened, synchronize, reopened, labeled, unlabeled]
branches:
- main
- 'release/**'
- 'feature/**'
permissions:
pull-requests: write
jobs:
require-chart-ref-label:
uses: Aignosi/github_workflow_templates/.github/workflows/require-chart-ref-label.yml@main

17
.github/workflows/version-bump.yml vendored Normal file
View File

@@ -0,0 +1,17 @@
name: Version Bump
permissions:
contents: write
on:
pull_request:
types: [closed]
branches: [main, 'release/**', 'feature/**']
jobs:
bump:
if: github.event.pull_request.merged == true
uses: Aignosi/github_workflow_templates/.github/workflows/reusable-version-bump.yml@main
with:
update_version_in: '["pyproject"]'
secrets: inherit

View File

@@ -813,7 +813,9 @@ Runs automatically on each Pull Request to `main`:
- ✅ Automatic version calculation based on branch pattern
#### Deploy (`.github/workflows/deploy.yml`)
Runs automatically when a PR is merged to `main`:
Triggered manually via `workflow_dispatch`, with a required `chart_ref` input (no
default — the sientia-helm-chart branch to bump and deploy; a branch that doesn't
exist is a hard error, not a silent fallback):
- ✅ Builds and pushes Docker image to Azure Container Registry
- ✅ Creates GitHub release with calculated version
- ✅ Deploys to Kubernetes using Helm

View File

@@ -527,6 +527,32 @@ class DataManagerRepository(SientiaMonitoring):
self.error(error_msg, metadata)
raise OSError(error_msg) from e
def _save_data_csv(
self, frame: pd.DataFrame, file_path: str, metadata: dict[str, Any] | None = None
) -> None:
"""
Write a train/test frame to CSV with its time index as a ``timestamp`` column.
laborious drift reads ``test_data.csv`` from MLflow as its reference and needs
``timestamp`` (SIENTIAPDE-2201). If a column already has that name, the frame is
written without the index and that model's drift keeps falling back to the target
window: training never fails over drift.
Args:
frame: Frame indexed by the UTC training timestamps.
file_path: Destination CSV path.
metadata: Workflow metadata for logging.
"""
if 'timestamp' in frame.columns:
self.warning(
f"Column 'timestamp' already exists; {path.basename(file_path)} saved without "
'the time column (drift reference falls back to the target window)',
metadata,
)
frame.to_csv(file_path, index=False)
return
frame.to_csv(file_path, index_label='timestamp')
def generate_report(
self, data: TrainModelResult, metadata: dict[str, Any] | None = None
) -> TrainModelResult:
@@ -598,11 +624,11 @@ class DataManagerRepository(SientiaMonitoring):
# Save training / validation CSVs using the same frames as the report (includes
# literal `target` alias for Evidently, plus predictions and float-cast features).
data.train_data_path = path.join(data.run_dir, 'train_data.csv')
reference_data_float.to_csv(data.train_data_path, index=False)
self._save_data_csv(reference_data_float, data.train_data_path, metadata)
# Save test data CSV
data.test_data_path = path.join(data.run_dir, 'test_data.csv')
current_data_float.to_csv(data.test_data_path, index=False)
self._save_data_csv(current_data_float, data.test_data_path, metadata)
# Save equation as JSON
if data.equation is not None and data.params.model_type == 'linear_regression':

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "model-manager"
version = "2.0.1"
version = "2.0.3"
description = "Sientia DataOps Model Manager - ML Model Orchestration System"
readme = "README.md"
requires-python = ">=3.11"

View File

@@ -514,6 +514,58 @@ def test_generate_report_adds_target_alias_for_reports(tmp_path):
assert test_csv['target'].equals(test_csv['t'])
_UTC_INDEX = pd.DatetimeIndex(['2024-01-01 02:00', '2024-01-01 01:00'], tz='UTC')
def _dated_result(**train_columns: list[float]) -> TrainModelResult:
"""Frames indexed by UTC time, like prepare_training_data + predict produce them."""
columns = train_columns or {'v1': [1.0, 2.0], 't': [1.0, 2.0]}
return TrainModelResult(
params=TrainModelParams.from_dict(_minimal_dict_for_prepare()),
train_data=pd.DataFrame(columns, index=_UTC_INDEX),
val_data=pd.DataFrame(columns, index=_UTC_INDEX),
y_train_pred=pd.DataFrame({'t': [1.0, 2.0]}, index=_UTC_INDEX),
y_pred=pd.DataFrame({'t': [1.0, 2.0]}, index=_UTC_INDEX),
run_name='testrun',
)
def _generate_report(repo, tmr: TrainModelResult, tmp_path) -> TrainModelResult:
with (
patch.object(repo, '_get_reports_directory', return_value=str(tmp_path)),
patch('model_manager.utils.repository.data_manager_repository.Reports'),
):
return repo.generate_report(tmr, {})
def test_generate_report_csvs_keep_the_time_index_as_timestamp(tmp_path):
"""laborious drift reads test_data.csv as its reference and needs ``timestamp``."""
repo = dmr.DataManagerRepository(MagicMock())
out = _generate_report(repo, _dated_result(), tmp_path)
for csv_path in (out.train_data_path, out.test_data_path):
csv = pd.read_csv(csv_path)
assert list(csv.columns) == ['timestamp', 'prediction', 'v1', 't', 'target']
assert pd.to_datetime(csv['timestamp']).tolist() == list(_UTC_INDEX)
def test_generate_report_csvs_skip_timestamp_when_a_column_already_has_the_name(tmp_path):
"""A variable literally named ``timestamp`` wins; training must not fail over drift."""
repo = dmr.DataManagerRepository(MagicMock())
tmr = _dated_result(timestamp=[5.0, 6.0], v1=[1.0, 2.0], t=[1.0, 2.0])
with patch.object(repo, 'warning') as warning:
out = _generate_report(repo, tmr, tmp_path)
for csv_path in (out.train_data_path, out.test_data_path):
csv = pd.read_csv(csv_path)
assert list(csv.columns) == ['prediction', 'timestamp', 'v1', 't', 'target']
assert csv['timestamp'].tolist() == [5.0, 6.0]
assert warning.call_count == 2
assert "Column 'timestamp' already exists" in warning.call_args.args[0]
def test_generate_report_skips_equation_file_when_not_linear(tmp_path):
repo = dmr.DataManagerRepository(MagicMock())
p = TrainModelParams.from_dict(_minimal_dict_for_prepare())