Code import - branch release/SIENTIAPDE-1645
This commit is contained in:
33
docs/DB_CV022_WIT230 _double_date_column.csv
Normal file
33
docs/DB_CV022_WIT230 _double_date_column.csv
Normal file
@@ -0,0 +1,33 @@
|
||||
DATA,DATE2,03CV022/CORRENTE_N_M1_PV(Value),303-WIT-230(Value)
|
||||
01/05/2022 00:00:00,07-01-2022 01:00:00,170,33
|
||||
01/05/2022 00:00:10,07-01-2022 01:00:10,169,605
|
||||
01/05/2022 00:00:20,07-01-2022 01:00:20,169,178
|
||||
01/05/2022 00:00:30,07-01-2022 01:00:30,166,468
|
||||
01/05/2022 00:00:40,07-01-2022 01:00:40,162,136
|
||||
01/05/2022 00:00:50,07-01-2022 01:00:50,157,804
|
||||
01/05/2022 00:01:00,07-01-2022 01:01:00,155,883
|
||||
01/05/2022 00:01:10,07-01-2022 01:01:10,155,684
|
||||
01/05/2022 00:01:20,07-01-2022 01:01:20,155,484
|
||||
01/05/2022 00:01:30,07-01-2022 01:01:30,155,284
|
||||
01/05/2022 00:01:40,07-01-2022 01:01:40,155,85
|
||||
01/05/2022 00:01:50,07-01-2022 01:01:50,154,885
|
||||
01/05/2022 00:02:00,09-01-2022 02:02:00,154,685
|
||||
01/05/2022 00:02:10,09-01-2022 02:02:10,154,486
|
||||
01/05/2022 00:02:20,09-01-2022 02:02:20,154,286
|
||||
01/05/2022 00:02:30,09-01-2022 02:02:30,154,86
|
||||
01/05/2022 00:02:40,09-01-2022 02:02:40,152,866
|
||||
01/05/2022 00:02:50,09-01-2022 02:02:50,150,87
|
||||
01/05/2022 00:03:00,09-01-2022 02:03:00,148,874
|
||||
01/05/2022 00:03:10,09-01-2022 02:03:10,148,2134
|
||||
01/05/2022 00:03:20,09-01-2022 02:03:20,148,2068
|
||||
01/05/2022 00:03:30,09-01-2022 02:03:30,148,2022
|
||||
01/05/2022 00:03:40,09-01-2022 02:03:40,148,1976
|
||||
01/05/2022 00:03:50,09-01-2022 02:03:50,154,139
|
||||
01/05/2022 00:04:00,11-01-2022 03:04:00,165,112
|
||||
01/05/2022 00:04:10,11-01-2022 03:04:10,170,42
|
||||
01/05/2022 00:04:20,11-01-2022 03:04:20,170,116
|
||||
01/05/2022 00:04:30,11-01-2022 03:04:30,170,191
|
||||
01/05/2022 00:04:40,11-01-2022 03:04:40,170,266
|
||||
01/05/2022 00:04:50,11-01-2022 03:04:50,170,341
|
||||
01/05/2022 00:05:00,11-01-2022 03:05:00,170,416
|
||||
01/05/2022 00:05:10,11-01-2022 03:05:10,170,491
|
||||
|
647
docs/DB_CV022_WIT230.csv
Normal file
647
docs/DB_CV022_WIT230.csv
Normal file
@@ -0,0 +1,647 @@
|
||||
DATA,03CV022/CORRENTE_N_M1_PV(Value),303-WIT-230(Value)
|
||||
01/05/2022 00:00:00,170,33
|
||||
01/05/2022 00:00:10,169,605
|
||||
01/05/2022 00:00:20,169,178
|
||||
01/05/2022 00:00:30,166,468
|
||||
01/05/2022 00:00:40,162,136
|
||||
01/05/2022 00:00:50,157,804
|
||||
01/05/2022 00:01:00,155,883
|
||||
01/05/2022 00:01:10,155,684
|
||||
01/05/2022 00:01:20,155,484
|
||||
01/05/2022 00:01:30,155,284
|
||||
01/05/2022 00:01:40,155,85
|
||||
01/05/2022 00:01:50,154,885
|
||||
01/05/2022 00:02:00,154,685
|
||||
01/05/2022 00:02:10,154,486
|
||||
01/05/2022 00:02:20,154,286
|
||||
01/05/2022 00:02:30,154,86
|
||||
01/05/2022 00:02:40,152,866
|
||||
01/05/2022 00:02:50,150,87
|
||||
01/05/2022 00:03:00,148,874
|
||||
01/05/2022 00:03:10,148,2134
|
||||
01/05/2022 00:03:20,148,2068
|
||||
01/05/2022 00:03:30,148,2022
|
||||
01/05/2022 00:03:40,148,1976
|
||||
01/05/2022 00:03:50,154,139
|
||||
01/05/2022 00:04:00,165,112
|
||||
01/05/2022 00:04:10,170,42
|
||||
01/05/2022 00:04:20,170,116
|
||||
01/05/2022 00:04:30,170,191
|
||||
01/05/2022 00:04:40,170,266
|
||||
01/05/2022 00:04:50,170,341
|
||||
01/05/2022 00:05:00,170,416
|
||||
01/05/2022 00:05:10,170,491
|
||||
01/05/2022 00:05:20,170,566
|
||||
01/05/2022 00:05:30,170,641
|
||||
01/05/2022 00:05:40,170,716
|
||||
01/05/2022 00:05:50,170,79
|
||||
01/05/2022 00:06:00,170,865
|
||||
01/05/2022 00:06:10,170,94
|
||||
01/05/2022 00:06:20,171,15
|
||||
01/05/2022 00:06:30,171,9
|
||||
01/05/2022 00:06:40,171,165
|
||||
01/05/2022 00:06:50,171,24
|
||||
01/05/2022 00:07:00,171,315
|
||||
01/05/2022 00:07:10,171,39
|
||||
01/05/2022 00:07:20,171,465
|
||||
01/05/2022 00:07:30,171,539
|
||||
01/05/2022 00:07:40,171,614
|
||||
01/05/2022 00:07:50,171,689
|
||||
01/05/2022 00:08:00,171,764
|
||||
01/05/2022 00:08:10,171,839
|
||||
01/05/2022 00:08:20,171,914
|
||||
01/05/2022 00:08:30,171,989
|
||||
01/05/2022 00:08:40,172,64
|
||||
01/05/2022 00:08:50,172,139
|
||||
01/05/2022 00:09:00,172,214
|
||||
01/05/2022 00:09:10,172,288
|
||||
01/05/2022 00:09:20,172,363
|
||||
01/05/2022 00:09:30,172,438
|
||||
01/05/2022 00:09:40,172,513
|
||||
01/05/2022 00:09:50,172,588
|
||||
01/05/2022 00:10:00,172,663
|
||||
01/05/2022 00:10:10,172,738
|
||||
01/05/2022 00:10:20,172,813
|
||||
01/05/2022 00:10:30,172,888
|
||||
01/05/2022 00:10:40,172,962
|
||||
01/05/2022 00:10:50,172,981
|
||||
01/05/2022 00:11:00,172,943
|
||||
01/05/2022 00:11:10,172,905
|
||||
01/05/2022 00:11:20,172,867
|
||||
01/05/2022 00:11:30,172,829
|
||||
01/05/2022 00:11:40,172,79
|
||||
01/05/2022 00:11:50,172,752
|
||||
01/05/2022 00:12:00,172,714
|
||||
01/05/2022 00:12:10,172,676
|
||||
01/05/2022 00:12:20,172,638
|
||||
01/05/2022 00:12:30,172,6
|
||||
01/05/2022 00:12:40,172,562
|
||||
01/05/2022 00:12:50,172,524
|
||||
01/05/2022 00:13:00,172,485
|
||||
01/05/2022 00:13:10,172,447
|
||||
01/05/2022 00:13:20,172,409
|
||||
01/05/2022 00:13:30,172,371
|
||||
01/05/2022 00:13:40,172,333
|
||||
01/05/2022 00:13:50,172,295
|
||||
01/05/2022 00:14:00,172,257
|
||||
01/05/2022 00:14:10,172,219
|
||||
01/05/2022 00:14:20,172,18
|
||||
01/05/2022 00:14:30,172,142
|
||||
01/05/2022 00:14:40,172,104
|
||||
01/05/2022 00:14:50,172,66
|
||||
01/05/2022 00:15:00,172,28
|
||||
01/05/2022 00:15:10,171,99
|
||||
01/05/2022 00:15:20,171,952
|
||||
01/05/2022 00:15:30,171,914
|
||||
01/05/2022 00:15:40,171,876
|
||||
01/05/2022 00:15:50,171,837
|
||||
01/05/2022 00:16:00,171,799
|
||||
01/05/2022 00:16:10,171,761
|
||||
01/05/2022 00:16:20,171,723
|
||||
01/05/2022 00:16:30,171,685
|
||||
01/05/2022 00:16:40,171,647
|
||||
01/05/2022 00:16:50,171,609
|
||||
01/05/2022 00:17:00,171,571
|
||||
01/05/2022 00:17:10,171,533
|
||||
01/05/2022 00:17:20,171,494
|
||||
01/05/2022 00:17:30,171,456
|
||||
01/05/2022 00:17:40,171,418
|
||||
01/05/2022 00:17:50,171,38
|
||||
01/05/2022 00:18:00,171,342
|
||||
01/05/2022 00:18:10,171,304
|
||||
01/05/2022 00:18:20,171,266
|
||||
01/05/2022 00:18:30,171,228
|
||||
01/05/2022 00:18:40,171,189
|
||||
01/05/2022 00:18:50,171,151
|
||||
01/05/2022 00:19:00,171,113
|
||||
01/05/2022 00:19:10,171,75
|
||||
01/05/2022 00:19:20,171,37
|
||||
01/05/2022 00:19:30,170,999
|
||||
01/05/2022 00:19:40,170,961
|
||||
01/05/2022 00:19:50,170,923
|
||||
01/05/2022 00:20:00,170,884
|
||||
01/05/2022 00:20:10,170,846
|
||||
01/05/2022 00:20:20,170,808
|
||||
01/05/2022 00:20:30,170,77
|
||||
01/05/2022 00:20:40,170,732
|
||||
01/05/2022 00:20:50,170,694
|
||||
01/05/2022 00:21:00,170,656
|
||||
01/05/2022 00:21:10,170,618
|
||||
01/05/2022 00:21:20,170,58
|
||||
01/05/2022 00:21:30,170,541
|
||||
01/05/2022 00:21:40,170,503
|
||||
01/05/2022 00:21:50,170,465
|
||||
01/05/2022 00:22:00,170,427
|
||||
01/05/2022 00:22:10,170,389
|
||||
01/05/2022 00:22:20,170,351
|
||||
01/05/2022 00:22:30,170,313
|
||||
01/05/2022 00:22:40,170,275
|
||||
01/05/2022 00:22:50,170,236
|
||||
01/05/2022 00:23:00,170,198
|
||||
01/05/2022 00:23:10,170,16
|
||||
01/05/2022 00:23:20,170,122
|
||||
01/05/2022 00:23:30,170,84
|
||||
01/05/2022 00:23:40,170,46
|
||||
01/05/2022 00:23:50,170,8
|
||||
01/05/2022 00:24:00,169,97
|
||||
01/05/2022 00:24:10,169,932
|
||||
01/05/2022 00:24:20,169,893
|
||||
01/05/2022 00:24:30,169,855
|
||||
01/05/2022 00:24:40,169,817
|
||||
01/05/2022 00:24:50,169,779
|
||||
01/05/2022 00:25:00,169,741
|
||||
01/05/2022 00:25:10,169,703
|
||||
01/05/2022 00:25:20,169,665
|
||||
01/05/2022 00:25:30,169,627
|
||||
01/05/2022 00:25:40,169,588
|
||||
01/05/2022 00:25:50,169,55
|
||||
01/05/2022 00:26:00,169,512
|
||||
01/05/2022 00:26:10,169,474
|
||||
01/05/2022 00:26:20,169,436
|
||||
01/05/2022 00:26:30,169,398
|
||||
01/05/2022 00:26:40,169,36
|
||||
01/05/2022 00:26:50,169,322
|
||||
01/05/2022 00:27:00,169,284
|
||||
01/05/2022 00:27:10,169,245
|
||||
01/05/2022 00:27:20,169,207
|
||||
01/05/2022 00:27:30,169,169
|
||||
01/05/2022 00:27:40,169,131
|
||||
01/05/2022 00:27:50,169,93
|
||||
01/05/2022 00:28:00,169,55
|
||||
01/05/2022 00:28:10,169,17
|
||||
01/05/2022 00:28:20,168,979
|
||||
01/05/2022 00:28:30,168,94
|
||||
01/05/2022 00:28:40,168,902
|
||||
01/05/2022 00:28:50,168,864
|
||||
01/05/2022 00:29:00,168,826
|
||||
01/05/2022 00:29:10,168,788
|
||||
01/05/2022 00:29:20,168,75
|
||||
01/05/2022 00:29:30,168,712
|
||||
01/05/2022 00:29:40,168,674
|
||||
01/05/2022 00:29:50,168,636
|
||||
01/05/2022 00:30:00,168,597
|
||||
01/05/2022 00:30:10,168,559
|
||||
01/05/2022 00:30:20,168,521
|
||||
01/05/2022 00:30:30,168,483
|
||||
01/05/2022 00:30:40,168,445
|
||||
01/05/2022 00:30:50,168,407
|
||||
01/05/2022 00:31:00,168,369
|
||||
01/05/2022 00:31:10,168,331
|
||||
01/05/2022 00:31:20,168,292
|
||||
01/05/2022 00:31:30,168,254
|
||||
01/05/2022 00:31:40,168,216
|
||||
01/05/2022 00:31:50,168,178
|
||||
01/05/2022 00:32:00,168,14
|
||||
01/05/2022 00:32:10,168,102
|
||||
01/05/2022 00:32:20,168,64
|
||||
01/05/2022 00:32:30,168,26
|
||||
01/05/2022 00:32:40,168,26
|
||||
01/05/2022 00:32:50,168,105
|
||||
01/05/2022 00:33:00,168,183
|
||||
01/05/2022 00:33:10,168,262
|
||||
01/05/2022 00:33:20,168,341
|
||||
01/05/2022 00:33:30,168,42
|
||||
01/05/2022 00:33:40,168,499
|
||||
01/05/2022 00:33:50,168,578
|
||||
01/05/2022 00:34:00,168,657
|
||||
01/05/2022 00:34:10,168,735
|
||||
01/05/2022 00:34:20,168,814
|
||||
01/05/2022 00:34:30,168,893
|
||||
01/05/2022 00:34:40,168,972
|
||||
01/05/2022 00:34:50,169,51
|
||||
01/05/2022 00:35:00,169,13
|
||||
01/05/2022 00:35:10,169,209
|
||||
01/05/2022 00:35:20,169,287
|
||||
01/05/2022 00:35:30,169,366
|
||||
01/05/2022 00:35:40,169,445
|
||||
01/05/2022 00:35:50,169,524
|
||||
01/05/2022 00:36:00,169,603
|
||||
01/05/2022 00:36:10,169,682
|
||||
01/05/2022 00:36:20,169,761
|
||||
01/05/2022 00:36:30,169,839
|
||||
01/05/2022 00:36:40,169,918
|
||||
01/05/2022 00:36:50,169,997
|
||||
01/05/2022 00:37:00,170,76
|
||||
01/05/2022 00:37:10,170,155
|
||||
01/05/2022 00:37:20,170,234
|
||||
01/05/2022 00:37:30,170,312
|
||||
01/05/2022 00:37:40,170,391
|
||||
01/05/2022 00:37:50,170,47
|
||||
01/05/2022 00:38:00,170,549
|
||||
01/05/2022 00:38:10,170,628
|
||||
01/05/2022 00:38:20,170,707
|
||||
01/05/2022 00:38:30,170,786
|
||||
01/05/2022 00:38:40,170,864
|
||||
01/05/2022 00:38:50,170,943
|
||||
01/05/2022 00:39:00,170,993
|
||||
01/05/2022 00:39:10,170,967
|
||||
01/05/2022 00:39:20,170,942
|
||||
01/05/2022 00:39:30,170,917
|
||||
01/05/2022 00:39:40,170,891
|
||||
01/05/2022 00:39:50,170,866
|
||||
01/05/2022 00:40:00,170,841
|
||||
01/05/2022 00:40:10,170,815
|
||||
01/05/2022 00:40:20,170,79
|
||||
01/05/2022 00:40:30,170,764
|
||||
01/05/2022 00:40:40,170,739
|
||||
01/05/2022 00:40:50,170,714
|
||||
01/05/2022 00:41:00,170,688
|
||||
01/05/2022 00:41:10,170,663
|
||||
01/05/2022 00:41:20,170,637
|
||||
01/05/2022 00:41:30,170,612
|
||||
01/05/2022 00:41:40,170,587
|
||||
01/05/2022 00:41:50,170,561
|
||||
01/05/2022 00:42:00,170,536
|
||||
01/05/2022 00:42:10,170,51
|
||||
01/05/2022 00:42:20,170,485
|
||||
01/05/2022 00:42:30,170,46
|
||||
01/05/2022 00:42:40,170,434
|
||||
01/05/2022 00:42:50,170,409
|
||||
01/05/2022 00:43:00,170,384
|
||||
01/05/2022 00:43:10,170,358
|
||||
01/05/2022 00:43:20,170,333
|
||||
01/05/2022 00:43:30,170,307
|
||||
01/05/2022 00:43:40,170,282
|
||||
01/05/2022 00:43:50,170,257
|
||||
01/05/2022 00:44:00,170,231
|
||||
01/05/2022 00:44:10,170,206
|
||||
01/05/2022 00:44:20,170,18
|
||||
01/05/2022 00:44:30,170,155
|
||||
01/05/2022 00:44:40,170,13
|
||||
01/05/2022 00:44:50,170,104
|
||||
01/05/2022 00:45:00,170,79
|
||||
01/05/2022 00:45:10,170,54
|
||||
01/05/2022 00:45:20,170,28
|
||||
01/05/2022 00:45:30,170,3
|
||||
01/05/2022 00:45:40,169,977
|
||||
01/05/2022 00:45:50,169,952
|
||||
01/05/2022 00:46:00,169,927
|
||||
01/05/2022 00:46:10,169,901
|
||||
01/05/2022 00:46:20,169,876
|
||||
01/05/2022 00:46:30,169,85
|
||||
01/05/2022 00:46:40,169,825
|
||||
01/05/2022 00:46:50,169,8
|
||||
01/05/2022 00:47:00,169,774
|
||||
01/05/2022 00:47:10,169,749
|
||||
01/05/2022 00:47:20,169,723
|
||||
01/05/2022 00:47:30,169,698
|
||||
01/05/2022 00:47:40,169,673
|
||||
01/05/2022 00:47:50,169,647
|
||||
01/05/2022 00:48:00,169,622
|
||||
01/05/2022 00:48:10,169,597
|
||||
01/05/2022 00:48:20,169,571
|
||||
01/05/2022 00:48:30,169,546
|
||||
01/05/2022 00:48:40,169,52
|
||||
01/05/2022 00:48:50,169,495
|
||||
01/05/2022 00:49:00,169,47
|
||||
01/05/2022 00:49:10,169,444
|
||||
01/05/2022 00:49:20,169,419
|
||||
01/05/2022 00:49:30,169,393
|
||||
01/05/2022 00:49:40,169,368
|
||||
01/05/2022 00:49:50,169,343
|
||||
01/05/2022 00:50:00,169,317
|
||||
01/05/2022 00:50:10,169,292
|
||||
01/05/2022 00:50:20,169,266
|
||||
01/05/2022 00:50:30,169,241
|
||||
01/05/2022 00:50:40,169,216
|
||||
01/05/2022 00:50:50,169,19
|
||||
01/05/2022 00:51:00,169,165
|
||||
01/05/2022 00:51:10,169,14
|
||||
01/05/2022 00:51:20,169,114
|
||||
01/05/2022 00:51:30,169,89
|
||||
01/05/2022 00:51:40,169,63
|
||||
01/05/2022 00:51:50,169,38
|
||||
01/05/2022 00:52:00,169,13
|
||||
01/05/2022 00:52:10,168,987
|
||||
01/05/2022 00:52:20,168,962
|
||||
01/05/2022 00:52:30,168,936
|
||||
01/05/2022 00:52:40,168,911
|
||||
01/05/2022 00:52:50,168,886
|
||||
01/05/2022 00:53:00,168,86
|
||||
01/05/2022 00:53:10,168,835
|
||||
01/05/2022 00:53:20,168,809
|
||||
01/05/2022 00:53:30,168,784
|
||||
01/05/2022 00:53:40,168,759
|
||||
01/05/2022 00:53:50,168,733
|
||||
01/05/2022 00:54:00,168,708
|
||||
01/05/2022 00:54:10,168,683
|
||||
01/05/2022 00:54:20,168,657
|
||||
01/05/2022 00:54:30,168,632
|
||||
01/05/2022 00:54:40,168,606
|
||||
01/05/2022 00:54:50,168,581
|
||||
01/05/2022 00:55:00,168,556
|
||||
01/05/2022 00:55:10,168,53
|
||||
01/05/2022 00:55:20,168,505
|
||||
01/05/2022 00:55:30,168,479
|
||||
01/05/2022 00:55:40,168,454
|
||||
01/05/2022 00:55:50,168,429
|
||||
01/05/2022 00:56:00,168,403
|
||||
01/05/2022 00:56:10,168,378
|
||||
01/05/2022 00:56:20,168,352
|
||||
01/05/2022 00:56:30,168,327
|
||||
01/05/2022 00:56:40,168,302
|
||||
01/05/2022 00:56:50,168,276
|
||||
01/05/2022 00:57:00,168,251
|
||||
01/05/2022 00:57:10,168,226
|
||||
01/05/2022 00:57:20,168,2
|
||||
01/05/2022 00:57:30,168,175
|
||||
01/05/2022 00:57:40,168,149
|
||||
01/05/2022 00:57:50,168,124
|
||||
01/05/2022 00:58:00,168,99
|
||||
01/05/2022 00:58:10,168,73
|
||||
01/05/2022 00:58:20,168,48
|
||||
01/05/2022 00:58:30,168,22
|
||||
01/05/2022 00:58:40,167,76
|
||||
01/05/2022 00:58:50,160,151
|
||||
01/05/2022 00:59:00,161,484
|
||||
01/05/2022 00:59:10,162,817
|
||||
01/05/2022 00:59:20,164,281
|
||||
01/05/2022 00:59:30,166,777
|
||||
01/05/2022 00:59:40,167,148
|
||||
01/05/2022 00:59:50,148,45
|
||||
01/05/2022 01:00:00,117,525
|
||||
01/05/2022 01:00:10,105,3
|
||||
01/05/2022 01:00:20,105,315
|
||||
01/05/2022 01:00:30,105,6
|
||||
01/05/2022 01:00:40,105,886
|
||||
01/05/2022 01:00:50,106,171
|
||||
01/05/2022 01:01:00,106,456
|
||||
01/05/2022 01:01:10,106,742
|
||||
01/05/2022 01:01:20,107,698
|
||||
01/05/2022 01:01:30,115,81
|
||||
01/05/2022 01:01:40,122,464
|
||||
01/05/2022 01:01:50,129,847
|
||||
01/05/2022 01:02:00,137,231
|
||||
01/05/2022 01:02:10,144,138
|
||||
01/05/2022 01:02:20,145,801
|
||||
01/05/2022 01:02:30,147,464
|
||||
01/05/2022 01:02:40,149,835
|
||||
01/05/2022 01:02:50,160,808
|
||||
01/05/2022 01:03:00,171,2
|
||||
01/05/2022 01:03:10,171,36
|
||||
01/05/2022 01:03:20,171,7
|
||||
01/05/2022 01:03:30,171,103
|
||||
01/05/2022 01:03:40,171,137
|
||||
01/05/2022 01:03:50,171,171
|
||||
01/05/2022 01:04:00,171,204
|
||||
01/05/2022 01:04:10,171,238
|
||||
01/05/2022 01:04:20,171,272
|
||||
01/05/2022 01:04:30,171,305
|
||||
01/05/2022 01:04:40,171,339
|
||||
01/05/2022 01:04:50,171,372
|
||||
01/05/2022 01:05:00,171,406
|
||||
01/05/2022 01:05:10,171,44
|
||||
01/05/2022 01:05:20,171,473
|
||||
01/05/2022 01:05:30,171,507
|
||||
01/05/2022 01:05:40,171,541
|
||||
01/05/2022 01:05:50,171,574
|
||||
01/05/2022 01:06:00,171,608
|
||||
01/05/2022 01:06:10,171,642
|
||||
01/05/2022 01:06:20,171,675
|
||||
01/05/2022 01:06:30,171,709
|
||||
01/05/2022 01:06:40,171,743
|
||||
01/05/2022 01:06:50,171,776
|
||||
01/05/2022 01:07:00,171,81
|
||||
01/05/2022 01:07:10,171,844
|
||||
01/05/2022 01:07:20,171,877
|
||||
01/05/2022 01:07:30,171,911
|
||||
01/05/2022 01:07:40,171,944
|
||||
01/05/2022 01:07:50,171,978
|
||||
01/05/2022 01:08:00,172,12
|
||||
01/05/2022 01:08:10,172,45
|
||||
01/05/2022 01:08:20,172,79
|
||||
01/05/2022 01:08:30,172,113
|
||||
01/05/2022 01:08:40,172,146
|
||||
01/05/2022 01:08:50,172,18
|
||||
01/05/2022 01:09:00,172,214
|
||||
01/05/2022 01:09:10,172,247
|
||||
01/05/2022 01:09:20,172,281
|
||||
01/05/2022 01:09:30,172,315
|
||||
01/05/2022 01:09:40,172,348
|
||||
01/05/2022 01:09:50,172,382
|
||||
01/05/2022 01:10:00,172,415
|
||||
01/05/2022 01:10:10,172,449
|
||||
01/05/2022 01:10:20,172,483
|
||||
01/05/2022 01:10:30,172,516
|
||||
01/05/2022 01:10:40,172,55
|
||||
01/05/2022 01:10:50,172,584
|
||||
01/05/2022 01:11:00,172,617
|
||||
01/05/2022 01:11:10,172,651
|
||||
01/05/2022 01:11:20,172,685
|
||||
01/05/2022 01:11:30,172,718
|
||||
01/05/2022 01:11:40,172,752
|
||||
01/05/2022 01:11:50,172,786
|
||||
01/05/2022 01:12:00,172,819
|
||||
01/05/2022 01:12:10,172,853
|
||||
01/05/2022 01:12:20,172,887
|
||||
01/05/2022 01:12:30,172,92
|
||||
01/05/2022 01:12:40,172,954
|
||||
01/05/2022 01:12:50,172,987
|
||||
01/05/2022 01:13:00,173,21
|
||||
01/05/2022 01:13:10,173,55
|
||||
01/05/2022 01:13:20,173,88
|
||||
01/05/2022 01:13:30,173,122
|
||||
01/05/2022 01:13:40,173,156
|
||||
01/05/2022 01:13:50,173,189
|
||||
01/05/2022 01:14:00,173,223
|
||||
01/05/2022 01:14:10,173,257
|
||||
01/05/2022 01:14:20,173,29
|
||||
01/05/2022 01:14:30,173,324
|
||||
01/05/2022 01:14:40,173,358
|
||||
01/05/2022 01:14:50,173,391
|
||||
01/05/2022 01:15:00,173,425
|
||||
01/05/2022 01:15:10,173,458
|
||||
01/05/2022 01:15:20,173,492
|
||||
01/05/2022 01:15:30,173,526
|
||||
01/05/2022 01:15:40,173,559
|
||||
01/05/2022 01:15:50,173,593
|
||||
01/05/2022 01:16:00,173,627
|
||||
01/05/2022 01:16:10,173,66
|
||||
01/05/2022 01:16:20,173,694
|
||||
01/05/2022 01:16:30,173,728
|
||||
01/05/2022 01:16:40,173,761
|
||||
01/05/2022 01:16:50,173,795
|
||||
01/05/2022 01:17:00,173,829
|
||||
01/05/2022 01:17:10,173,862
|
||||
01/05/2022 01:17:20,173,896
|
||||
01/05/2022 01:17:30,173,93
|
||||
01/05/2022 01:17:40,173,963
|
||||
01/05/2022 01:17:50,173,997
|
||||
01/05/2022 01:18:00,174,103
|
||||
01/05/2022 01:18:10,174,218
|
||||
01/05/2022 01:18:20,174,332
|
||||
01/05/2022 01:18:30,174,446
|
||||
01/05/2022 01:18:40,174,56
|
||||
01/05/2022 01:18:50,174,674
|
||||
01/05/2022 01:19:00,174,788
|
||||
01/05/2022 01:19:10,174,902
|
||||
01/05/2022 01:19:20,175,17
|
||||
01/05/2022 01:19:30,175,131
|
||||
01/05/2022 01:19:40,175,245
|
||||
01/05/2022 01:19:50,175,359
|
||||
01/05/2022 01:20:00,175,473
|
||||
01/05/2022 01:20:10,175,587
|
||||
01/05/2022 01:20:20,175,701
|
||||
01/05/2022 01:20:30,175,816
|
||||
01/05/2022 01:20:40,175,93
|
||||
01/05/2022 01:20:50,176,44
|
||||
01/05/2022 01:21:00,176,158
|
||||
01/05/2022 01:21:10,176,272
|
||||
01/05/2022 01:21:20,176,386
|
||||
01/05/2022 01:21:30,176,501
|
||||
01/05/2022 01:21:40,176,615
|
||||
01/05/2022 01:21:50,176,729
|
||||
01/05/2022 01:22:00,176,843
|
||||
01/05/2022 01:22:10,176,957
|
||||
01/05/2022 01:22:20,177,71
|
||||
01/05/2022 01:22:30,177,185
|
||||
01/05/2022 01:22:40,177,3
|
||||
01/05/2022 01:22:50,177,414
|
||||
01/05/2022 01:23:00,177,528
|
||||
01/05/2022 01:23:10,177,642
|
||||
01/05/2022 01:23:20,177,756
|
||||
01/05/2022 01:23:30,177,87
|
||||
01/05/2022 01:23:40,177,984
|
||||
01/05/2022 01:23:50,178,7
|
||||
01/05/2022 01:24:00,178,15
|
||||
01/05/2022 01:24:10,178,24
|
||||
01/05/2022 01:24:20,178,32
|
||||
01/05/2022 01:24:30,178,4
|
||||
01/05/2022 01:24:40,178,48
|
||||
01/05/2022 01:24:50,178,57
|
||||
01/05/2022 01:25:00,178,65
|
||||
01/05/2022 01:25:10,178,73
|
||||
01/05/2022 01:25:20,178,81
|
||||
01/05/2022 01:25:30,178,9
|
||||
01/05/2022 01:25:40,178,98
|
||||
01/05/2022 01:25:50,178,106
|
||||
01/05/2022 01:26:00,178,114
|
||||
01/05/2022 01:26:10,178,123
|
||||
01/05/2022 01:26:20,178,131
|
||||
01/05/2022 01:26:30,178,139
|
||||
01/05/2022 01:26:40,178,147
|
||||
01/05/2022 01:26:50,178,156
|
||||
01/05/2022 01:27:00,178,164
|
||||
01/05/2022 01:27:10,178,172
|
||||
01/05/2022 01:27:20,178,18
|
||||
01/05/2022 01:27:30,178,189
|
||||
01/05/2022 01:27:40,178,197
|
||||
01/05/2022 01:27:50,178,205
|
||||
01/05/2022 01:28:00,178,213
|
||||
01/05/2022 01:28:10,178,222
|
||||
01/05/2022 01:28:20,178,23
|
||||
01/05/2022 01:28:30,178,238
|
||||
01/05/2022 01:28:40,178,246
|
||||
01/05/2022 01:28:50,178,255
|
||||
01/05/2022 01:29:00,178,263
|
||||
01/05/2022 01:29:10,178,271
|
||||
01/05/2022 01:29:20,178,279
|
||||
01/05/2022 01:29:30,178,288
|
||||
01/05/2022 01:29:40,178,296
|
||||
01/05/2022 01:29:50,178,304
|
||||
01/05/2022 01:30:00,178,312
|
||||
01/05/2022 01:30:10,178,321
|
||||
01/05/2022 01:30:20,178,329
|
||||
01/05/2022 01:30:30,178,337
|
||||
01/05/2022 01:30:40,178,345
|
||||
01/05/2022 01:30:50,178,354
|
||||
01/05/2022 01:31:00,178,362
|
||||
01/05/2022 01:31:10,178,37
|
||||
01/05/2022 01:31:20,178,378
|
||||
01/05/2022 01:31:30,178,387
|
||||
01/05/2022 01:31:40,178,395
|
||||
01/05/2022 01:31:50,178,403
|
||||
01/05/2022 01:32:00,178,411
|
||||
01/05/2022 01:32:10,178,42
|
||||
01/05/2022 01:32:20,178,428
|
||||
01/05/2022 01:32:30,178,436
|
||||
01/05/2022 01:32:40,178,444
|
||||
01/05/2022 01:32:50,178,453
|
||||
01/05/2022 01:33:00,178,461
|
||||
01/05/2022 01:33:10,178,469
|
||||
01/05/2022 01:33:20,178,477
|
||||
01/05/2022 01:33:30,178,486
|
||||
01/05/2022 01:33:40,178,494
|
||||
01/05/2022 01:33:50,178,502
|
||||
01/05/2022 01:34:00,178,51
|
||||
01/05/2022 01:34:10,178,519
|
||||
01/05/2022 01:34:20,178,527
|
||||
01/05/2022 01:34:30,178,535
|
||||
01/05/2022 01:34:40,178,543
|
||||
01/05/2022 01:34:50,178,552
|
||||
01/05/2022 01:35:00,178,56
|
||||
01/05/2022 01:35:10,178,568
|
||||
01/05/2022 01:35:20,178,576
|
||||
01/05/2022 01:35:30,178,585
|
||||
01/05/2022 01:35:40,178,593
|
||||
01/05/2022 01:35:50,178,601
|
||||
01/05/2022 01:36:00,178,609
|
||||
01/05/2022 01:36:10,178,618
|
||||
01/05/2022 01:36:20,178,626
|
||||
01/05/2022 01:36:30,178,634
|
||||
01/05/2022 01:36:40,178,642
|
||||
01/05/2022 01:36:50,178,651
|
||||
01/05/2022 01:37:00,178,659
|
||||
01/05/2022 01:37:10,178,667
|
||||
01/05/2022 01:37:20,178,675
|
||||
01/05/2022 01:37:30,178,684
|
||||
01/05/2022 01:37:40,178,692
|
||||
01/05/2022 01:37:50,178,7
|
||||
01/05/2022 01:38:00,178,708
|
||||
01/05/2022 01:38:10,178,717
|
||||
01/05/2022 01:38:20,178,725
|
||||
01/05/2022 01:38:30,178,733
|
||||
01/05/2022 01:38:40,178,741
|
||||
01/05/2022 01:38:50,178,75
|
||||
01/05/2022 01:39:00,178,758
|
||||
01/05/2022 01:39:10,178,766
|
||||
01/05/2022 01:39:20,178,774
|
||||
01/05/2022 01:39:30,178,783
|
||||
01/05/2022 01:39:40,178,791
|
||||
01/05/2022 01:39:50,178,799
|
||||
01/05/2022 01:40:00,178,807
|
||||
01/05/2022 01:40:10,178,816
|
||||
01/05/2022 01:40:20,178,824
|
||||
01/05/2022 01:40:30,178,832
|
||||
01/05/2022 01:40:40,178,84
|
||||
01/05/2022 01:40:50,178,849
|
||||
01/05/2022 01:41:00,178,857
|
||||
01/05/2022 01:41:10,178,865
|
||||
01/05/2022 01:41:20,178,873
|
||||
01/05/2022 01:41:30,178,882
|
||||
01/05/2022 01:41:40,178,89
|
||||
01/05/2022 01:41:50,178,898
|
||||
01/05/2022 01:42:00,178,906
|
||||
01/05/2022 01:42:10,178,915
|
||||
01/05/2022 01:42:20,178,923
|
||||
01/05/2022 01:42:30,178,931
|
||||
01/05/2022 01:42:40,178,939
|
||||
01/05/2022 01:42:50,178,948
|
||||
01/05/2022 01:43:00,178,956
|
||||
01/05/2022 01:43:10,178,964
|
||||
01/05/2022 01:43:20,178,972
|
||||
01/05/2022 01:43:30,178,981
|
||||
01/05/2022 01:43:40,178,989
|
||||
01/05/2022 01:43:50,178,997
|
||||
01/05/2022 01:44:00,169,147
|
||||
01/05/2022 01:44:10,148,934
|
||||
01/05/2022 01:44:20,119,458
|
||||
01/05/2022 01:44:30,108,326
|
||||
01/05/2022 01:44:40,108,825
|
||||
01/05/2022 01:44:50,109,324
|
||||
01/05/2022 01:45:00,109,824
|
||||
01/05/2022 01:45:10,110,323
|
||||
01/05/2022 01:45:20,110,822
|
||||
01/05/2022 01:45:30,123,3
|
||||
01/05/2022 01:45:40,141,628
|
||||
01/05/2022 01:45:50,160,252
|
||||
01/05/2022 01:46:00,169,863
|
||||
01/05/2022 01:46:10,174,353
|
||||
01/05/2022 01:46:20,176,8
|
||||
01/05/2022 01:46:30,176,19
|
||||
01/05/2022 01:46:40,176,31
|
||||
01/05/2022 01:46:50,176,43
|
||||
01/05/2022 01:47:00,176,55
|
||||
01/05/2022 01:47:10,176,67
|
||||
01/05/2022 01:47:20,176,79
|
||||
01/05/2022 01:47:30,176,91
|
||||
|
37097
docs/data-1749222138290.csv
Normal file
37097
docs/data-1749222138290.csv
Normal file
File diff suppressed because it is too large
Load Diff
66
docs/scenarios.md
Normal file
66
docs/scenarios.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# E2E Test Scenarios
|
||||
|
||||
This document maps the workflow scenarios tested in the E2E suite to their corresponding JSON input files and expected behaviors.
|
||||
|
||||
## Infrastructure (second-pass review)
|
||||
|
||||
- **Containers:** PostgreSQL, MinIO, MongoDB, and Gitea via testcontainers; real clients and `Activities` code paths.
|
||||
- **MLflow:** `file://` tracking URI (real SDK, no remote server).
|
||||
- **Temporal:** `WorkflowEnvironment.start_time_skipping()` — official temporalio test runtime; workflows and activities are not stubbed.
|
||||
- **Logging/metrics:** `get_logger` + `MetricsController` (sientia_do); no `unittest.mock` for observability in `e2e/conftest.py`.
|
||||
- **Unit tests** under `tests/` may still use mocks where appropriate; that policy is separate from this E2E suite.
|
||||
|
||||
## 1. TrainModel Workflow (`test_train_model_workflow.py`)
|
||||
|
||||
### 1.1 Happy Paths (Successful execution)
|
||||
|
||||
| Test Function | Input JSON | Expected Status | Description |
|
||||
|---|---|---|---|
|
||||
| `test_scenario_1_1_1_linear_regression_basic` | `01-linear-regression-basic.json` | `TRAINING_SUCCESS` | Basic linear regression without scaler. Verifies end-to-end pipeline. |
|
||||
| `test_scenario_1_1_2_linear_regression_with_scaler` | `02-linear-regression-with-scaler.json` | `TRAINING_SUCCESS` | Linear regression with `Standard Scaler`. |
|
||||
| `test_scenario_1_1_3_polynomial_regression_degree2_with_scaler` | `03-polynomial-regression-degree2.json` | `TRAINING_SUCCESS` | Polynomial regression (degree 2) with Standard Scaler. |
|
||||
| `test_scenario_1_1_4_polynomial_regression_degree3_with_scaler` | `04-polynomial-regression-degree3.json` | `TRAINING_SUCCESS` | Polynomial regression (degree 3) with Standard Scaler. |
|
||||
| `test_scenario_1_1_5_linear_regression_with_lags` | `05-linear-regression-with-lags.json` | `TRAINING_SUCCESS` | Linear regression with `lag_train`/`lag_val` per variable. |
|
||||
| `test_scenario_1_1_6_linear_regression_nan_interpolation` | `06-linear-regression-nan-interpolation.json` | `TRAINING_SUCCESS` | Linear regression with `nan_treatment='linear interpolation'`. |
|
||||
| `test_scenario_1_1_7_linear_regression_static_window_removal` | `07-linear-regression-static-window-removal.json` | `TRAINING_SUCCESS` | `rem_static_win=true` with default `static_threshold`. |
|
||||
| `test_scenario_1_1_8_linear_regression_with_limits` | `08-linear-regression-with-limits.json` | `TRAINING_SUCCESS` | `support_filters` with `min`/`max` per variable. |
|
||||
| `test_scenario_1_1_9_polynomial_degree2_scaler_and_lags` | `09-polynomial-degree2-with-scaler-and-lags.json` | `TRAINING_SUCCESS` | Polynomial (degree 2), Standard Scaler, and lags. |
|
||||
| `test_scenario_1_1_10_linear_regression_with_ar_opt_params` | `10-linear-regression-with-ar.json` | `TRAINING_SUCCESS` | `opt_params.include_ar=true` (placeholder for future AR behavior). |
|
||||
| `test_scenario_1_1_11_linear_regression_static_threshold_custom` | `11-linear-regression-static-threshold-custom.json` | `TRAINING_SUCCESS` | `rem_static_win=true` with custom `static_threshold`. |
|
||||
| `test_scenario_1_1_12_alternate_date_format_dd_mm_yyyy` | `12-angular-test-date-format.json` | `TRAINING_SUCCESS` | `date_column=DATA`, `dd/MM/yyyy` format, object `training_data_dd_mm_yyyy.csv`. |
|
||||
| `test_scenario_1_1_13_alternate_csv_narrow_date_window` | `13-angular-test-double-date-column.json` | `TRAINING_SUCCESS` | Same alternate CSV with a bounded `start_date`/`end_date` window. |
|
||||
| `test_scenario_1_1_14_polynomial_with_support_filters` | `14-angular-test-polynomial-support-filters.json` | `TRAINING_SUCCESS` | Polynomial (degree 4), scaler, `upper_line`/`lower_line` support filters. |
|
||||
| `test_scenario_1_1_15_linear_regression_custom_target_column_name` | `15-linear-regression-custom-target-column.json` | `TRAINING_SUCCESS` | Custom `target_variable` column name (not literal ``target``); Evidently/report columns must match. |
|
||||
| `test_scenario_1_1_16_naive_timestamp_header_column` | `16-linear-regression-naive-timestamp-header.json` | `TRAINING_SUCCESS` | `date_column`=`Timestamp`, naive CSV `training_data_timestamp_naive.csv`. |
|
||||
| `test_scenario_1_1_17_linear_regression_blank_timestamp_row_dropped` | `17-linear-regression-blank-timestamp-row.json` | `TRAINING_SUCCESS` | One empty timestamp cell; row dropped before index. |
|
||||
|
||||
### 1.2 Error Paths
|
||||
|
||||
| Test Function | Input JSON | Expected Status | Description |
|
||||
|---|---|---|---|
|
||||
| `test_scenario_1_2_1_minio_file_not_found` | `01-linear-regression-basic.json` | `TRAINING_ERROR` | MinIO file does not exist. Workflow fails during file download. |
|
||||
| `test_scenario_1_2_2_experiment_run_id_not_in_db` | `01-linear-regression-basic.json` | N/A (raises Exception) | `experiment_run_id` does not exist in DB. Workflow fails immediately on status update attempt. |
|
||||
|
||||
## 2. Parameter Validation (`test_train_model_validation.py`)
|
||||
|
||||
These scenarios test the business rule validations inside `validate_train_params`. All are expected to terminate with `ORCHESTRATOR_VALIDATION_ERROR`.
|
||||
|
||||
| Test Function | Modification | Expected Error Substring |
|
||||
|---|---|---|
|
||||
| `test_scenario_2_1_1_train_size_out_of_range` | `train_size = 5` | `'train_size'` |
|
||||
| `test_scenario_2_1_2_empty_variable_columns` | `variable_columns = []` | `'variable_columns'` |
|
||||
| `test_scenario_2_1_3_invalid_date_format` | `date_format = 'INVALID'` | `'date_format'` |
|
||||
| `test_scenario_2_1_4_whitespace_only_model_name` | `model_name = ' '` | `'model_name'` |
|
||||
| `test_scenario_2_1_5_unknown_model_type` | `model_type = 'totally_unknown_model'` | `'totally_unknown_model'` |
|
||||
| `test_scenario_2_1_6_missing_target_variable` | `target_variable = ''` | `'target_variable'` |
|
||||
| `test_scenario_2_1_7_missing_experiment_run_id` | Missing `experiment_run_id` | N/A (raises ValueError immediately) |
|
||||
| `test_scenario_2_1_8_missing_date_column` | Missing `date_column` | N/A (raises ValueError immediately) |
|
||||
| `test_scenario_2_1_9_whitespace_date_column` | `date_column = ' '` | `'date_column'` |
|
||||
|
||||
## 3. CleanupFiles Workflow (`test_cleanup_files_workflow.py`)
|
||||
|
||||
| Test Function | Description |
|
||||
|---|---|
|
||||
| `test_scenario_3_1_1_cleanup_with_no_temp_dirs` | Temp directory is empty. Activity completes without error. |
|
||||
| `test_scenario_3_1_2_cleanup_removes_old_temp_dirs` | Two stale directories matching `name_YYYYMMDD_HHMMSS_microseconds` are removed when older than retention. |
|
||||
| `test_scenario_3_1_3_cleanup_nonexistent_temp_path` | Target path does not exist. Handled gracefully without error. |
|
||||
37097
docs/test-model-data.csv
Executable file
37097
docs/test-model-data.csv
Executable file
File diff suppressed because it is too large
Load Diff
40
docs/test-scenarios/01-linear-regression-basic.json
Normal file
40
docs/test-scenarios/01-linear-regression-basic.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Cenário básico de regressão linear sem scaler",
|
||||
"experiment_run_id": 1001,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
40
docs/test-scenarios/02-linear-regression-with-scaler.json
Normal file
40
docs/test-scenarios/02-linear-regression-with-scaler.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão linear com Standard Scaler habilitado",
|
||||
"experiment_run_id": 1002,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "Standard Scaler"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
40
docs/test-scenarios/03-polynomial-regression-degree2.json
Normal file
40
docs/test-scenarios/03-polynomial-regression-degree2.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão polinomial de grau 2 com scaler (obrigatório para evitar overflow)",
|
||||
"experiment_run_id": 1003,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Polynomial Regression",
|
||||
"model_type": "polynomial_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 2,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "Standard Scaler"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
40
docs/test-scenarios/04-polynomial-regression-degree3.json
Normal file
40
docs/test-scenarios/04-polynomial-regression-degree3.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão polinomial de grau 3 com scaler",
|
||||
"experiment_run_id": 1004,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Polynomial Regression",
|
||||
"model_type": "polynomial_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 3,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "Standard Scaler"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
40
docs/test-scenarios/05-linear-regression-with-lags.json
Normal file
40
docs/test-scenarios/05-linear-regression-with-lags.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão linear com lags de treino e validação",
|
||||
"experiment_run_id": 1005,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 5
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 3
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão linear com tratamento de NaN por interpolação linear",
|
||||
"experiment_run_id": 1006,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "linear interpolation",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão linear com remoção de janelas estáticas",
|
||||
"experiment_run_id": 1007,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": true,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
45
docs/test-scenarios/08-linear-regression-with-limits.json
Normal file
45
docs/test-scenarios/08-linear-regression-with-limits.json
Normal file
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"_description": "Regressão linear com limites inferior e superior para variáveis",
|
||||
"experiment_run_id": 1008,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {
|
||||
"303-WIT-200(Value)": {
|
||||
"min": 0.0,
|
||||
"max": 1000.0
|
||||
}
|
||||
},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Cenário completo: regressão polinomial grau 2 com scaler e lags",
|
||||
"experiment_run_id": 1009,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Polynomial Regression",
|
||||
"model_type": "polynomial_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 3
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 2
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 2,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "Standard Scaler"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
42
docs/test-scenarios/10-linear-regression-with-ar.json
Normal file
42
docs/test-scenarios/10-linear-regression-with-ar.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"_description": "Linear regression placeholder for autoregressive features; include_ar is reserved for future wrapper support (see opt_params).",
|
||||
"experiment_run_id": 1010,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {
|
||||
"include_ar": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Regressão linear com remoção de janelas estáticas e static_threshold customizado",
|
||||
"experiment_run_id": 1011,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": true,
|
||||
"static_threshold": 100,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
40
docs/test-scenarios/12-angular-test-date-format.json
Normal file
40
docs/test-scenarios/12-angular-test-date-format.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Alternate date column (DATA) and dd/MM/yyyy HH:mm:ss format; uses MinIO object training_data_dd_mm_yyyy.csv from E2E fixtures.",
|
||||
"experiment_run_id": 1012,
|
||||
"variable_columns": [
|
||||
"303-WIT-230(Value)"
|
||||
],
|
||||
"target_variable": "03CV022/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data_dd_mm_yyyy.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "DATA",
|
||||
"date_format": "dd/MM/yyyy HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-230(Value)": 3
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-230(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": "01/05/2022",
|
||||
"end_date": "31/07/2022",
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
40
docs/test-scenarios/13-angular-test-double-date-column.json
Normal file
40
docs/test-scenarios/13-angular-test-double-date-column.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Same alternate CSV as scenario 12 (DATA + dd/MM/yyyy); narrow date window for regression coverage. Not a multi-date-column dataset.",
|
||||
"experiment_run_id": 1013,
|
||||
"variable_columns": [
|
||||
"303-WIT-230(Value)"
|
||||
],
|
||||
"target_variable": "03CV022/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data_dd_mm_yyyy.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "DATA",
|
||||
"date_format": "dd/MM/yyyy HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-230(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-230(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": "01/05/2022 00:00:00",
|
||||
"end_date": "31/05/2022 23:59:59",
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"_description": "Cenário angular-test-01: regressão polinomial degree 4, scaler, support filters em 303-WIT-200",
|
||||
"experiment_run_id": 1014,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Polynomial Regression",
|
||||
"model_type": "polynomial_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": "2025-06-02 00:00:00",
|
||||
"end_date": "2025-06-08 23:59:59",
|
||||
"support_filters": {
|
||||
"303-WIT-200(Value)": {
|
||||
"upper_line": {
|
||||
"intercept": 40.400002,
|
||||
"angle": 0
|
||||
},
|
||||
"lower_line": {
|
||||
"intercept": 30.5,
|
||||
"angle": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 4,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "Standard Scaler"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Target column name is not ``target``; report/Evidently sections must use params.target_variable.",
|
||||
"experiment_run_id": 1015,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "MY_CUSTOM_TARGET_COLUMN",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data_custom_target.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "Naive Timestamp column header; snake_case date_column/date_format and training_data_timestamp_naive.csv.",
|
||||
"experiment_run_id": 1016,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data_timestamp_naive.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "Timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_description": "One CSV row has an empty timestamp; pipeline should drop it and continue training.",
|
||||
"experiment_run_id": 1017,
|
||||
"variable_columns": [
|
||||
"303-WIT-200(Value)"
|
||||
],
|
||||
"target_variable": "03CV020/CORRENTE_N_M1_PV(Value)",
|
||||
"bucket_name": "model-training",
|
||||
"file_name": "training_data_blank_timestamp_row.csv",
|
||||
"line_separator": ",",
|
||||
"decimal_separator": ".",
|
||||
"date_column": "timestamp",
|
||||
"date_format": "yyyy-MM-dd HH:mm:ss",
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"random_state": 42,
|
||||
"model_name": "Linear Regression",
|
||||
"model_type": "linear_regression",
|
||||
"data_model_kwargs": {
|
||||
"lag_train": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"lag_val": {
|
||||
"303-WIT-200(Value)": 0
|
||||
},
|
||||
"nan_treatment": "drop",
|
||||
"rem_static_win": false,
|
||||
"static_threshold": null,
|
||||
"start_date": null,
|
||||
"end_date": null,
|
||||
"support_filters": {},
|
||||
"removed_intervals": []
|
||||
},
|
||||
"model_kwargs": {
|
||||
"degree": 1,
|
||||
"interaction_only": false,
|
||||
"scaler_name": "None"
|
||||
},
|
||||
"opt_params": {}
|
||||
}
|
||||
417
docs/train-model-workflow-io-diff-main-vs-current-branch.md
Normal file
417
docs/train-model-workflow-io-diff-main-vs-current-branch.md
Normal file
@@ -0,0 +1,417 @@
|
||||
# Train Model Workflow IO Diff (`main` vs current branch)
|
||||
|
||||
Base comparison: `git diff main...HEAD`
|
||||
Workflow analyzed: `train_model`
|
||||
|
||||
## 1) Executive overview
|
||||
|
||||
This branch introduces a structural refactor of the training stack and a contract update for workflow input/output.
|
||||
|
||||
Main impacts:
|
||||
|
||||
- The old in-house training stack (`TrainingRepository`, `ModelRepository`, `StorageRepository`, `model_manager.sientia.models`) was replaced by:
|
||||
- `DataManagerRepository` (data prep + metrics + report generation)
|
||||
- `SientiaModel` wrapper from plugin store (`sientia_model`)
|
||||
- `SientiaMLflowRepository` (MLflow integration)
|
||||
- `MinioRepository` (storage integration)
|
||||
- Input contract moved from many fixed legacy ML params to a plugin/wrapper-oriented schema (`model_type`, `*_kwargs`, `model_metadata`, optional `val_file_name`).
|
||||
- Workflow return changed from `None` to a serializable result object (`dict[str, Any] | None`) containing training execution metadata.
|
||||
- Queue naming and worker bootstrap architecture now depend on runtime (`train_model-<runtime>-queue`).
|
||||
|
||||
---
|
||||
|
||||
## 2) Input contract diff (before vs now)
|
||||
|
||||
### 2.1 Previous contract (`main`)
|
||||
|
||||
`TrainModelParams` in `main` required a large set of explicit fields for the old preprocessing/model pipeline, focused only in linear regression model:
|
||||
|
||||
- Core:
|
||||
- `experiment_run_id`, `variable_columns`, `target_variable`
|
||||
- `bucket_name`, `file_name`, `line_separator`, `decimal_separator`
|
||||
- `train_size`, `shuffle`
|
||||
- Legacy preprocessing/model fields focused only in linear regression model (required in `from_dict`):
|
||||
- `lag_train`, `lag_val`
|
||||
- `rem_static_win`, `low_lim`, `upp_lim`, `window`
|
||||
- `use_scaler`, `include_ar`, `scaler_name`
|
||||
- `removed_intervals`, `start_date`, `end_date`, `nan_treatment`
|
||||
- `degree`, `interaction_only`
|
||||
- `experiment_name`, `model_name`
|
||||
- `support_filters` (optional dict), `static_threshold` (optional int)
|
||||
|
||||
Validation was strongly tied to this structure (lag ranges, limits consistency, polynomial/scaler constraints, etc.).
|
||||
|
||||
### 2.2 Current contract (this branch)
|
||||
|
||||
`TrainModelParams` now supports a plugin-driven schema and wrapper kwargs:
|
||||
|
||||
- Kept/mandatory core fields:
|
||||
- `experiment_run_id` (now accepts numeric string too; coerced to int)
|
||||
- `variable_columns`, `target_variable`
|
||||
- `bucket_name`, `file_name`, `line_separator`, `decimal_separator`
|
||||
- `train_size`, `shuffle`
|
||||
- `model_name`
|
||||
- `model_type`
|
||||
- `data_model_kwargs`, `model_kwargs`, `opt_params` (required as dict by current `from_dict`)
|
||||
- New/updated fields:
|
||||
- `random_state` (default `42`)
|
||||
- `val_file_name` (optional explicit validation file)
|
||||
- `model_id` (currently optional, but needs discussion, since the model metadata in MongoDB should be created before the model training)
|
||||
- Removed from required input contract:
|
||||
- `lag_train`, `lag_val`, `rem_static_win`, `low_lim`, `upp_lim`, `window`
|
||||
- `use_scaler`, `include_ar`
|
||||
- `degree`, `interaction_only`, `nan_treatment`
|
||||
- `start_date`, `end_date`, `scaler_name`
|
||||
- `removed_intervals`, `support_filters`, `static_threshold`
|
||||
- Parameters internally derived:
|
||||
- `model_metadata` model type info from plugin store.
|
||||
- `run_name` is internally derived from experiment name and datetime.
|
||||
- `experiment_name` is internally derived from `model_name`.
|
||||
|
||||
### 2.3 Validation behavior changes
|
||||
|
||||
Before:
|
||||
- Validation was mostly hardcoded business checks tied to legacy linear/polynomial stack.
|
||||
|
||||
Now:
|
||||
- Validation still checks core constraints (`train_size`, non-empty strings, etc.), but model-specific validation moved to JSON Schema driven checks, using OpenAPI/JSON Schema definitions from plugin store:
|
||||
- `model_metadata.schemas.components.schemas.data_model`
|
||||
- `model_metadata.schemas.components.schemas.model`
|
||||
- `model_metadata.schemas.components.schemas.opt_params`
|
||||
- `model_metadata` is now a required semantic dependency for `validate_business_rules()`.
|
||||
- Date format validation remains, but allowed formats are defined locally in `train_model_params.py`.
|
||||
|
||||
### 2.4 Input loading pipeline changes in workflow
|
||||
|
||||
Before:
|
||||
- `validate_train_params` directly consumed workflow input.
|
||||
|
||||
Now:
|
||||
1. `load_model_metadata` runs first (fetches model index/schema from plugin store and injects `model_metadata`).
|
||||
2. `validate_train_params` runs with enriched payload.
|
||||
|
||||
This means IO preprocessing now depends on plugin-store metadata resolution before final validation.
|
||||
|
||||
---
|
||||
|
||||
## 3) Output contract diff (before vs now)
|
||||
|
||||
### 3.1 Workflow return (`train_model.run`)
|
||||
|
||||
Before (`main`):
|
||||
- Return type: `None`
|
||||
- Workflow side effects were persisted mainly via DB status updates and MLflow artifacts.
|
||||
|
||||
Now:
|
||||
- Return type: `dict[str, Any] | None`
|
||||
- Workflow returns the training activity summary when successful.
|
||||
|
||||
### 3.2 Activity-level training result payload
|
||||
|
||||
Before (from `Training.train_model` in `main` path):
|
||||
- Returned minimal dict:
|
||||
- `run_name`
|
||||
- `run_dir`
|
||||
|
||||
Now:
|
||||
- Returns extended dict:
|
||||
- `run_name`
|
||||
- `experiment_name`
|
||||
- `run_id`
|
||||
- `run_dir`
|
||||
|
||||
### 3.3 Persistence map by destination (DB, MLflow, MinIO, local filesystem)
|
||||
|
||||
This section maps where each artifact/metadata goes, in which format, and how that changed from `main`.
|
||||
|
||||
#### 3.3.1 PostgreSQL (`experiment_run` table)
|
||||
|
||||
## Before (`main`)
|
||||
|
||||
- Update path: `update_experiment_run` activity with `UpdateType.MODEL_SAVED`.
|
||||
- Persisted on success:
|
||||
- `status` transition to `TRAINING_SUCCESS`
|
||||
- `run_name` (MLflow run identifier used by current implementation)
|
||||
- Persisted on failures:
|
||||
- `status` transition to validation/training error statuses
|
||||
- `error_message`
|
||||
|
||||
## Now (current branch)
|
||||
|
||||
- Same update path and status/error behavior.
|
||||
- Even though train activity now returns more metadata (`run_id`, `experiment_name`), current workflow update for `MODEL_SAVED` still forwards mainly `run_name`.
|
||||
- Practical effect:
|
||||
- DB remains status-centric and run-name-centric
|
||||
- richer identifiers exist in workflow return payload, not fully mirrored to DB columns in current flow
|
||||
|
||||
#### 3.3.2 MLflow (tracking server/artifact store)
|
||||
|
||||
## Before (`main`)
|
||||
|
||||
- Persistence orchestration lived in `ModelRepository.save_model()` + `_save_run()`.
|
||||
- Typical persisted content:
|
||||
- model params (many legacy params such as lags, limits, scaler config, removed intervals)
|
||||
- regression metrics (`MSE`, `R2`, `MAE`)
|
||||
- model objects:
|
||||
- `data_model`
|
||||
- `prediction_model`
|
||||
- artifacts:
|
||||
- `report.html`
|
||||
- `train_data.csv`
|
||||
- `test_data.csv`
|
||||
- optional `model_equation.json`
|
||||
- Run naming:
|
||||
- computed by querying existing runs and appending sequence (`<experiment>-<n>` style)
|
||||
|
||||
## Now (current branch)
|
||||
|
||||
- Persistence orchestrated in `Training._persist_training_artifacts()` and MLflow run context is opened by `SientiaMLflowRepository.start_run(...)`.
|
||||
- Persisted content now:
|
||||
- model wrapper itself via `wrapper.store_model(name=train_params.model_name)`
|
||||
- regression metrics also logged as MLflow params via `mlflow.log_param(...)`:
|
||||
- `mse_val`
|
||||
- `mae_val`
|
||||
- `r2_val`
|
||||
- artifacts explicitly logged with `mlflow.log_artifact(...)`:
|
||||
- `report.html`
|
||||
- `train_data.csv`
|
||||
- `test_data.csv`
|
||||
- metrics are computed before save (`mse_val`, `mae_val`, `r2_val`) and persisted in the run as params
|
||||
- Run identifiers now exposed back to workflow:
|
||||
- `experiment_name`
|
||||
- `run_name`
|
||||
- `run_id`
|
||||
- Notable behavioral change:
|
||||
- `wrapper._input_example` is cleared (`None`) before storing model.
|
||||
|
||||
#### 3.3.3 MinIO object storage
|
||||
|
||||
## Before (`main`)
|
||||
|
||||
- Read path:
|
||||
- single source object downloaded via `StorageRepository.fetch_file(bucket_name, file_name)`
|
||||
- Write path:
|
||||
- training workflow did not write generated outputs to MinIO in this code path
|
||||
- generated artifacts were persisted to MLflow, not uploaded back to MinIO
|
||||
- Location:
|
||||
- source data in input bucket/key provided by workflow input (`bucket_name` + `file_name`)
|
||||
|
||||
## Now (current branch)
|
||||
|
||||
- Read path migrated to `MinioRepository.download_file(...)`.
|
||||
- Supports two input objects:
|
||||
- mandatory training object: `bucket_name` + `file_name`
|
||||
- optional validation object: same `bucket_name` + `val_file_name`
|
||||
- Write path:
|
||||
- still no artifact upload to MinIO in this workflow path
|
||||
- report/CSV outputs continue to flow to MLflow artifacts
|
||||
- Location details:
|
||||
- bucket resolved from payload (`bucket_name`)
|
||||
- object key exactly from payload (`file_name`, optional `val_file_name`)
|
||||
- default bucket in env/config is `MINIO_DEFAULT_BUCKET`, but runtime payload can override via `bucket_name`
|
||||
|
||||
#### 3.3.4 Local filesystem (ephemeral runtime workspace)
|
||||
|
||||
## Before (`main`)
|
||||
|
||||
- Temporary run dir created under reports root using run name + timestamp suffix.
|
||||
- Artifacts generated locally in that directory:
|
||||
- `report.html`
|
||||
- `train_data.csv`
|
||||
- `test_data.csv`
|
||||
- optional `model_equation.json`
|
||||
- After MLflow logging, cleanup activity removed temp directory.
|
||||
|
||||
## Now (current branch)
|
||||
|
||||
- Temporary run dir managed by `DataManagerRepository` under runtime reports root (`.../reports/temp/<run_name>`).
|
||||
- Same artifact family generated locally:
|
||||
- `report.html`
|
||||
- `train_data.csv`
|
||||
- `test_data.csv`
|
||||
- optional `model_equation.json` (for `linear_regression`)
|
||||
- Cleanup behavior is now tolerant:
|
||||
- cleanup runs in guarded `finally`
|
||||
- training success is not reverted if cleanup later fails
|
||||
|
||||
#### 3.3.5 Quick matrix (before vs now)
|
||||
|
||||
- **Postgres**
|
||||
- before: status + run_name + errors
|
||||
- now: same persisted shape; workflow return contains extra IDs
|
||||
- **MLflow**
|
||||
- before: legacy model objects + params/metrics + report/data artifacts
|
||||
- now: wrapper-based model persistence + `mse_val`/`mae_val`/`r2_val` as params + report/data artifacts + run_id exposed
|
||||
- **MinIO**
|
||||
- before: reads 1 CSV input object
|
||||
- now: reads 1 or 2 CSV input objects (train + optional validation), still no output upload
|
||||
- **Local temp**
|
||||
- before: generated artifacts, then cleanup
|
||||
- now: generated artifacts, then best-effort cleanup (non-blocking for success result)
|
||||
|
||||
### 3.4 Cleanup behavior impact on output semantics
|
||||
|
||||
Before:
|
||||
- Cleanup was called directly after training result; failures propagated straightforwardly.
|
||||
|
||||
Now:
|
||||
- Cleanup is in a guarded `finally`.
|
||||
- If training succeeded but cleanup fails, workflow warns and does not rollback success semantics.
|
||||
- Effective output semantics: successful training result can be returned even if temp cleanup fails.
|
||||
|
||||
---
|
||||
|
||||
## 4) Detailed field mapping (old -> new)
|
||||
|
||||
## Kept (or equivalent role)
|
||||
|
||||
- `experiment_run_id` -> kept (broader accepted types: int or numeric string)
|
||||
- `variable_columns` -> kept
|
||||
- `target_variable` -> kept
|
||||
- `bucket_name` -> kept
|
||||
- `file_name` -> kept
|
||||
- `line_separator` -> kept
|
||||
- `decimal_separator` -> kept
|
||||
- `date_column` -> required (snake_case key; must exist in CSV)
|
||||
- `date_format` -> optional in payload; omitted/null/blank resolves to default `yyyy-MM-dd HH:mm:ss`
|
||||
- `train_size` -> kept
|
||||
- `shuffle` -> kept
|
||||
- `model_name` -> kept (now less coupled to legacy model enum)
|
||||
|
||||
## Added
|
||||
|
||||
- `model_type` (primary selector for plugin wrapper/index lookup)
|
||||
- `data_model_kwargs`
|
||||
- `model_kwargs`
|
||||
- `opt_params`
|
||||
- `val_file_name` (optional second dataset input)
|
||||
- `model_id` (optional metadata)
|
||||
- `model_metadata` (loaded/required for schema validation)
|
||||
- `random_state` (explicit split reproducibility control)
|
||||
|
||||
## Removed from new required contract
|
||||
|
||||
- `lag_train`, `lag_val`
|
||||
- `rem_static_win`, `static_threshold`
|
||||
- `low_lim`, `upp_lim`
|
||||
- `window`
|
||||
- `use_scaler`, `include_ar`
|
||||
- `degree`, `interaction_only`
|
||||
- `nan_treatment`
|
||||
- `start_date`, `end_date`
|
||||
- `scaler_name`
|
||||
- `removed_intervals`
|
||||
- `support_filters`
|
||||
- `experiment_name` (no longer required as top-level client input)
|
||||
|
||||
---
|
||||
|
||||
## 5) Internal architecture update notes
|
||||
|
||||
### 5.1 Repository layer redesign
|
||||
|
||||
Removed:
|
||||
- `model_manager/utils/repository/model_repository.py`
|
||||
- `model_manager/utils/repository/training_repository.py`
|
||||
- `model_manager/utils/repository/storage_repository.py`
|
||||
|
||||
Added:
|
||||
- `model_manager/utils/repository/data_manager_repository.py`
|
||||
|
||||
Interpretation:
|
||||
- Data preprocessing/report/metrics responsibilities were consolidated into `DataManagerRepository`.
|
||||
- Training/model persistence shifted to wrapper + plugin store + MLflow repository integrations.
|
||||
|
||||
### 5.2 Model engine abstraction migration
|
||||
|
||||
Before:
|
||||
- Strong coupling to local classes in `model_manager.sientia.models` and custom preprocessing/model objects in `TrainModelResult`.
|
||||
|
||||
Now:
|
||||
- Training uses `SientiaModel` wrapper dynamically obtained by `plugin_store.get_model(model_type=...)`.
|
||||
- Contract is wrapper-driven (`train`, `transform`, `predict`, `store_model`).
|
||||
- The codebase removed `model_manager/sientia/models.py`, `model_serving.py`, and `utils.py`, indicating full migration to externalized model runtime abstraction.
|
||||
|
||||
### 5.3 Worker/runtime architecture changes
|
||||
|
||||
- New `prepare_worker.py` centralizes worker setup and autoscaling parameters.
|
||||
- Queue names are now runtime-derived:
|
||||
- `train_model-<runtime>-queue`
|
||||
- `cleanup_files-<runtime>-queue`
|
||||
- `worker.py` now installs runtime via plugin store (`plugin_store.install_runtime(runtime_name=...)`) before starting workers.
|
||||
- This introduces environment/runtime-aware deployment and model packaging behavior.
|
||||
|
||||
### 5.4 Synchronous activity and tracking adjustments
|
||||
|
||||
- `experiment_tracking` migrated from async postgres helper to sync postgres client path (`postgres_sync`).
|
||||
- Several activities switched to sync method signatures.
|
||||
- Error handling in workflow and DB status update paths is more defensive (secondary failures while persisting error status are logged and do not mask primary failure cause).
|
||||
|
||||
### 5.5 `TrainModelResult` shape update
|
||||
|
||||
Before:
|
||||
- Stored classic split artifacts (`x_train`, `x_test`, `y_train`, `y_test`) + concrete preprocessing/model objects (`process_data`, `regr`, `scaler_dict`).
|
||||
|
||||
Now:
|
||||
- Stores `train_data`, `val_data` and prediction DataFrames, plus tracking identifiers (`experiment_name`, `run_id`).
|
||||
- Result object is less tied to internal estimator classes and more aligned with serializable workflow/model-store integration.
|
||||
|
||||
---
|
||||
|
||||
## 6) Net IO compatibility assessment
|
||||
|
||||
## Input compatibility
|
||||
|
||||
Not backward compatible with old payloads without adaptation.
|
||||
|
||||
Key reasons:
|
||||
- Legacy required fields removed/ignored by new path.
|
||||
- New required fields introduced (`model_type`, `*_kwargs` dicts, runtime metadata flow dependency).
|
||||
- Validation pipeline now expects model metadata semantics.
|
||||
|
||||
## Output compatibility
|
||||
|
||||
Behavior changed:
|
||||
- Workflow now returns a result object (previously `None`).
|
||||
- Training summary includes `experiment_name` and `run_id` in addition to `run_name` and `run_dir`.
|
||||
- DB update still centered on `run_name`; callers relying only on DB may not see all new output info unless workflow return is consumed.
|
||||
|
||||
---
|
||||
|
||||
## 7) Practical migration guidance (client side)
|
||||
|
||||
To call `train_model` in this branch:
|
||||
|
||||
1. Send snake_case payload aligned to new `TrainModelParams`.
|
||||
2. Always provide:
|
||||
- `model_name` slugified model name (ex.: `test_model_name or test-model-name`)
|
||||
- `model_type`
|
||||
- `data_model_kwargs` (dict)
|
||||
- `model_kwargs` (dict)
|
||||
- `opt_params` (dict)
|
||||
3. Keep `experiment_run_id` numeric (int or numeric string).
|
||||
4. Use runtime queue naming consistent with worker runtime:
|
||||
- `train_model-<runtime>-queue`
|
||||
5. If you need explicit validation split file, send `val_file_name`; otherwise split uses `train_size`/`shuffle`/`random_state`.
|
||||
|
||||
---
|
||||
|
||||
## 8) Source references used for this document
|
||||
|
||||
Primary diffs:
|
||||
- `model_manager/workflows/train_model.py`
|
||||
- `model_manager/utils/models/train_model_params.py`
|
||||
- `model_manager/utils/models/train_model_result.py`
|
||||
- `model_manager/activities/training.py`
|
||||
- `model_manager/activities/activities.py`
|
||||
- `model_manager/activities/experiment_tracking.py`
|
||||
- `model_manager/utils/repository/data_manager_repository.py`
|
||||
- `model_manager/utils/repository/model_repository.py` (removed)
|
||||
- `model_manager/utils/repository/training_repository.py` (removed)
|
||||
- `model_manager/utils/repository/storage_repository.py` (removed)
|
||||
- `model_manager/worker/worker.py`
|
||||
- `model_manager/worker/prepare_worker.py`
|
||||
- `README.md`
|
||||
- `input-sample.md`
|
||||
- `scripts/run_training_test.py`
|
||||
|
||||
Reference in New Issue
Block a user