# Data source — IndPenSim, industrial-scale penicillin fermentation

Written 10 September 2026 · folder `A-Raw-Data/IndPenSim/`

## 1. What it is

"Data for: Modern day monitoring and control challenges outlined on an industrial-scale benchmark
fermentation process." Generated by **IndPenSim**, an advanced mathematical simulation of a
**100,000 litre penicillin fermentation system**, and the first simulation to include a realistic
simulated Raman spectroscopy device, built for developing and evaluating advanced control solutions
in biotechnology facilities.

| | |
| --- | --- |
| Contributor | Stephen Goldrick |
| Published | 1 July 2019, version 1 |
| DOI | 10.17632/pdnjz7zz5x.1 |
| Simulator | www.industrialpenicillinsimulation.com |
| Licence | CC BY 4.0 |

**Cite as:**
Goldrick S., Stefan A., Lovett D., Montague G., Lennox B. (2015) *The development of an industrial-scale
fed-batch fermentation simulation.* Journal of Biotechnology, 193:70–82.
and
Goldrick S., Duran-Villalobos C., Jankauskas K., Lovett D., Farid S. S., Lennox B. (2019) *Modern day
control challenges for industrial-scale fermentation processes.* Computers and Chemical Engineering.

## 2. The download

| | |
| --- | --- |
| `100_Batches_IndPenSim_V3.csv` | 2.6 GB, 113,935 rows × **2,237** columns |
| `100_Batches_IndPenSim_Statistics.csv` | 4 kB, one row per batch: penicillin harvested and total yield |
| Batches | 100 |
| Samples per batch | 835 – 1,450 |
| Batch length | 167 – 290 h |
| Sampling | 12 minutes |

**2,200 of the 2,237 columns are Raman spectral channels** (wavenumbers 2400 down to 201). Only the
first 37 are process and metadata — what a plant historian would actually hold.

### The header is broken

The file's own header has **2,239 names for 2,237 fields**. The PAT_ref description
`1- No Raman spec, 1-Raman spec recorded, 2-PAT control(PAT_ref:PAT ref)` contains two commas, so it is
read as three separate names and every name after it is shifted by two. Read positionally, the column
named "Batch reference" holds the fault flag and the column named "Batch ID" holds Raman intensity.
`make_indpensim.py` ignores the file's header and applies its own.

The true layout of the first 37 fields is: 31 process variables, then `fault_ref`, `control_ref`,
`pat_ref`, `batch_ref`, `batch_id` (a duplicate of `batch_ref`), `fault_flag`.

## 3. The four groups of batches

| Group | Batches | Mean length (h) | Mean total yield (kg) |
| --- | --- | --- | --- |
| recipe driven | 30 | 229 | 2,912,750 |
| operator controlled | 30 | 229 | 2,833,113 |
| APC with Raman | 30 | 225 | 3,484,283 |
| faulty | 10 | 230 | 2,600,203 |

Batches 1–30 are recipe driven, 31–60 operator controlled, 61–90 under Advanced Process Control using
the Raman spectrometer, and 91–100 carry faults. **Only batches 91–100 are faulty** — 11,525 samples
flagged in total, about 1,152 per faulty batch.

Worth noting in those numbers: APC with Raman averages **3,484,283 kg** against **2,833,113 kg** for
operator control, a 23% difference. That is the argument the dataset exists to make.

## 4. What we extracted

`B-Engines/make_indpensim.py` drops the Raman block, repairs the header and writes:

| File | Size | What |
| --- | --- | --- |
| `indpensim_process.csv` | 19 MB | 113,935 rows × 37 cols — all 100 batches, process tags plus batch/fault/control keys |
| `indpensim_batches.csv` | 6 kB | one row per batch: samples, length, control strategy, fault, yield |
| `indpensim_schema.csv` | 2 kB | 30 process tags (19 AI, 11 AO) |

**2.6 GB becomes 19 MB.** The Raman spectra are kept only in the archived original on the drive.

### The process tags

| Tag | AI/AO | Description | Unit |
| --- | --- | --- | --- |
| `aeration_rate` | AO | Aeration rate Fg | L/h |
| `agitator_rpm` | AO | Agitator speed | rpm |
| `sugar_feed_rate` | AO | Sugar feed rate Fs | L/h |
| `acid_flow_rate` | AO | Acid flow rate Fa | L/h |
| `base_flow_rate` | AO | Base flow rate Fb | L/h |
| `cool_water_flow` | AO | Heating/cooling water flow Fc | L/h |
| `heat_water_flow` | AO | Heating water flow Fh | L/h |
| `water_injection` | AO | Water for injection/dilution Fw | L/h |
| `head_pressure` | AI | Air head pressure | bar |
| `dumped_broth_flow` | AI | Dumped broth flow Fremoved | L/h |
| `substrate_conc` | AI | Substrate concentration S | g/L |
| `dissolved_o2` | AI | Dissolved oxygen DO2 | mg/L |
| `penicillin_conc` | AI | Penicillin concentration P | g/L |
| `vessel_volume` | AI | Vessel volume | L |
| `vessel_weight` | AI | Vessel weight | kg |
| `ph` | AI | pH | pH |
| `temperature` | AI | Temperature | K |
| `generated_heat` | AI | Generated heat Q | kJ |
| `co2_offgas` | AI | Carbon dioxide in off-gas | % |
| `paa_flow` | AO | PAA flow Fpaa | L/h |
| `paa_conc_offline` | AI | PAA concentration, offline | g/L |
| `oil_flow` | AO | Oil flow Foil | L/h |
| `nh3_conc_offline` | AI | NH3 concentration, offline | g/L |
| `oxygen_uptake` | AI | Oxygen uptake rate OUR | g/min |
| `o2_offgas` | AI | Oxygen in off-gas | % |
| `penicillin_offline` | AI | Penicillin concentration, offline | g/L |
| `biomass_offline` | AI | Biomass concentration, offline | g/L |
| `carbon_evolution` | AI | Carbon evolution rate CER | g/h |
| `nh3_shots` | AO | Ammonia shots | kg |
| `viscosity_offline` | AI | Viscosity, offline | cP |

Values check out against a real fermenter: temperature about 298 K, pH about 6.5, vessel volume
56,000–96,000 L in a 100,000 L vessel, agitator a constant 100 rpm, penicillin rising to 36 g/L.

Two columns are **answer key and must never be analysis inputs**: `fault_ref` (this batch carries a
fault) and `fault_flag` (this sample is inside a fault).

## 5. Why A-E cannot analyse this yet

**This is batch data, and the pipeline is built for continuous process.** The mismatch is structural,
not a matter of file format:

1. **"Normal" is a set of good batches, not a window of time.** Step B chooses a reference *window*
   and Step C measures everything against it. For batch, normal is the trajectory of the good batches,
   and every batch must be compared against that trajectory at the same point in its own progress.
2. **Batches are different lengths.** 835 to 1,450 samples. They must be aligned — by phase, by
   fraction of batch elapsed, or by an indicator variable — before anything can be averaged.
3. **A batch is not steady.** Concentration climbs from 0 to 36 g/L by design. Run continuously,
   Step C would call the entire fermentation an excursion, and every batch changeover a fault.
4. **Sampling is 12 minutes.** Fine for a 200-hour batch, far too slow for the control-loop analysis,
   which needs several samples per oscillation cycle.

Running `indpensim_process.csv` through Step A would work, and everything after it would be wrong.

**What it would take:** batch mode in Steps B and C — reference *batches* instead of a reference
window, batch alignment, and per-phase statistics. That is the ISA-88 gap already named in
`C-Work-Process/01-Review-of-Current-Process.md`. This dataset is the right thing to build it against,
with 90 good batches to learn from and 10 faulty ones to catch.

## 6. What it is good for right now

- **The APC business case.** The yields above are a clean, citable argument that advanced control on a
  Raman measurement beats operator control by roughly a quarter.
- **A target for batch mode.** 100 batches, 3 control strategies, 10 faulty, with a published answer key.
- **Not** a regression test for the continuous pipeline. Use the Tennessee Eastman and the synthetic
  chemical reactor sets for that.