TEP-Plant-10days-AIAO — Step A overall report

+------------------------------------------------------------------------------
| Program Name:         step_a_prepare.py
| Program Version:      2026-09-10 10:49:28
| Date:                 2026-09-11
| Time:                 08:48:52
| Input 1 - Raw CSV:    tep_plant_10days.csv
| Input 2 - Glossary:   tep_schema.csv
| Input 3 - Time stamp: auto-detected  (e.g. "2026-04-01 00:00:00")  -> 3.0 min grid
+------------------------------------------------------------------------------

CTRL Designer LLC

1. Objective and where we are

Objective: turn raw plant history into decisions, in five steps. Each step writes a folder the next one reads, so any result can be traced back to the file it came from.
StepNameWhat it doesProgramOutput folderState
APrepareRaw historian exports become one clean, regular Data table. Merge files, normalise timestamps, replace bad-value strings, put on a regular grid, fill short gaps, apply the repair rules, and record every change.step_a_prepare.py + A-Prepare-GUI-v01.pyA-Data/<project>/you are here
BExploreTrend and cluster the clean data. The engineer chooses which tags, which period, and which window counts as normal, and writes it down as the scope.step_b_explore.py + B-Explore-GUI-v01.pyB-Scope/<project>/next
CAnalyzeOperability (PCA, T2/SPE, events), oscillation and slowness per tag, control-loop metrics, batch phases - all driven by the scope from B.step_c_analyze.py + C-Analyze-GUI-v01.pyC-Analysis/<project>/
DAI reviewAn AI reads the results and the scope and drafts findings, anomalies and questions for the engineer.step_d_review.py + D-Review-GUI-v01.pyD-Review/<project>/
EReportThe engineer edits, decides, and either closes the study or returns to B with a new scope.to buildE-Report/<project>/

2. Input data

FileRoleSizeLocation
tep_plant_10days.csvdata1.8 MB/Users/yahyanazer/Dropbox/__C_2026_Work/CTRL-Designer/B-CDL-Data-Analysis/A-Raw-Data/TEP/tep_plant_10days.csv
tep_schema.csvglossary (TAG - Description - EU)6 kB/Users/yahyanazer/Dropbox/__C_2026_Work/CTRL-Designer/B-CDL-Data-Analysis/A-Raw-Data/TEP/tep_schema.csv

Timestamp formats found in the raw files:

filefirst timestamp as written
tep_plant_10days.csv2026-04-01 00:00:00

Raw data root: /Users/yahyanazer/Dropbox/__C_2026_Work/CTRL-Designer/B-CDL-Data-Analysis/A-Raw-Data

3. What the program did

#OperationResult
1Read the raw files and detect the timestamp columnsingle file; formats seen: 1
2Replace PI bad-value strings (Bad, No Data, Shutdown, …) with unknown15 values
3Put every tag on one regular time grid4,800 samples at 3 min; 40 timestamps were absent
4Fill gaps of at most 5 samples by time interpolation265 values filled
5Score every tag: missing, bad values, update interval, longest flat run, spikes52 tags scored
6Drop tags that are constant or mostly missing0 dropped, 52 kept
7Apply the repair rulesgaps<=5 interpolate; slow tags held; frozen unknown; spikes flag → 7 actions
8Locate every issue in time214 issues: spike 154, missing 52, slow update 6, bad values 1, frozen 1
9Attach descriptions and units from the glossary41 AI, 11 AO tags labelled
10Draw the evidence and write the reportsavailability map, issue timeline, per-tag panels, 4 HTML reports
Repair rules applied: gaps<=5 interpolate; slow tags held; frozen unknown; spikes flag. 7 actions; 2,310 samples left unknown on purpose — long gaps and stuck sensors are never invented.

4. Results

FileWhat it isSizeLocation
prepare_report.htmlwhat Step A did, with the charts29 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/prepare_report.html
bad_tags_report.htmlevery flagged tag: trend + zoom per bad zone58 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/bad_tags_report.html
fixed_tags_report.htmlACTUAL / BAD / FIXED per repaired tag48 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/fixed_tags_report.html
data.parquetthe corrected table Step B reads888 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/data.parquet
data.csvsame table for Excel / Seeq1.8 MB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/data.csv
raw_on_grid.parquetraw values on the same grid (before/after)874 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/raw_on_grid.parquet
repairs.csvevery repair action with its reason1 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/repairs.csv
data_quality.csvper-tag metrics and status3 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/data_quality.csv
issues.csvevery issue located in time12 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/issues.csv
glossary_used.csvTAG - Description - EU as used3 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/glossary_used.csv
manifest.jsoninputs, settings, counts - the reproducibility record1 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/manifest.json
quality_overview.pngtags x time availability map157 kB/Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO/quality_overview.png

Project folder: /Volumes/YN-4T/WIP/CTRL-Designer/B-CDL-Data-Analysis/A-Data/TEP-Plant-10days-AIAO

5. Next steps

  1. Review the 52 flagged tags in the bad-tags and fixed-tags reports. Decide which spikes are process events and which are data faults - that judgement belongs to a person, not the tool.
  2. If the repairs are not right, re-run Step A with different rules (spike flag/clip/remove, frozen unknown/keep).
  3. Step B (Explore): read data.parquet from this folder, trend and cluster the tags, choose the tags, the period and the reference (normal) window, and write scope.json into B-Scope/TEP-Plant-10days-AIAO/.
  4. Then Step C analyses only what the scope names.

© 2026 CTRL Designer LLC. All rights reserved.
Yahya Nazer · info@ctrlDesigner.com