Illusion of Generalization: Synthetic Contamination in Waveform AI

Aleksandr Ivanaiskii, PhD
Founder of Industrial AI & Systems Architect

Evgeny Ivanaiskii, PhD
Domain Expert in Non-Destructive Testing

Sergei Shipilov
AI Architecture Lead, Rivixi LLC

Abstract

This paper investigates the fundamental problem of false adaptation in deep convolutional neural networks (CNNs) applied to non-destructive testing (NDT) of pipelines. Under severe field data shortages, developers often rely on surrogate audio files generated by rapid, heuristic methods (such as LLM-assisted vibe-coding) tailored to the parameters of physical correlation leak detectors. We experimentally demonstrate that injecting as little as 1% of such synthetic data into scratch training of a deep 1D-CNN (Acoustic1DNet) creates a robust illusion of model improvement: on internal validation, accuracy increases from 97.06% to 99.67%, while false positives drop ninefold. However, when tested on an independent, out-of-domain field dataset (Zayed et al., WDN Hong Kong, 140 files), the model exhibits a 7.14% drop in Recall. As the synthetic contamination increases to 20%, a complete collapse of generalization capability occurs—Recall plummets from 70.00% to 11.43% (the model misses 88% of actual leaks). This phenomenon is driven by the network's convolutional filters shifting their focus to simple mathematical patterns of the generator (the shortcut learning phenomenon [Geirhos et al., 2020]) instead of learning the complex, noisy physical wave envelopes.

1. Introduction and Background

Acoustic leak localization is based on recording elastic pressure waves generated by pressurized fluid escaping from a pipe defect. The amplitude of the propagating acoustic signal decays exponentially according to:

[ A(x) = A_0 e^{-\alpha x} ]

where (\alpha) is the attenuation coefficient, which depends on the pipe wall material, diameter, and soil properties. At distances exceeding 50–60 meters, high-frequency components of the target signal are completely absorbed, reducing the signal-to-noise ratio (SNR) to critical levels.

In NDT practice, field data scarcity is compounded by a specific human factor and reporting fraud. With the accessibility of public LLMs, operators have acquired tools to automate the generation of acoustic wave files. When a leak location is visually obvious (e.g., surface water pooling) and the exact coordinate is known, operators are tempted to save time and skip performing arduous physical measurements under harsh field conditions (mud, cold, deep manholes). Instead, they generate a synthetic acoustic file using a simple Python script pre-configured for the target coordinate. Physical correlators easily ingest this file, displaying a sharp, correct correlation peak. The falsified recording is then saved in corporate databases as a "verified field measurement".

The real danger arises when these fake files are mixed into corporate databases alongside genuine signals. Unsuspecting ML engineers later harvest these databases to train leak classification models, believing they contain raw physical data. Our team encountered this exact problem in NDT databases. To study and analyze the impact of such contamination on model training, we obtained the generator script from NDT operators, assigned it the working name Rivixi-FakeWAV v3 (which generates a 15 ms Gaussian envelope burst at 2-3 kHz with a linear Phase Delay TDOA shift based on a steel wave speed of (v \approx 5291) m/s), and evaluated its effects.

Although these generated files pass correlation analysis seamlessly, they act as "shortcut traps" for deep neural networks. This study explores the impact of such files on feature extraction during full model training from scratch.

2. Methodology and Experimental Setup

2.1. The Acoustic1DNet Waveform Classifier in the RIVIXI Platform

The Acoustic1DNet model is a core neural component of the "Diagnost" module of the RIVIXI platform, responsible for the automated analysis and classification of acoustic diagnostics. Within the "Diagnost" pipeline, this model acts as a fast, edge-capable, first-level waveform classifier. It runs in parallel with a two-dimensional spectral classifier (Mel-spectrogram CNN) and a digital physical correlator, providing an independent decision-making channel in the diagnostic ensemble.

This heterogeneous architecture minimizes blind spots: while the spectral classifier is sensitive to stationary frequency patterns, Acoustic1DNet specializes in the real-time temporal envelope structure and cross-channel phase coherence. Integrating machine learning in this manner builds upon classical acoustic emission diagnostics [Fares et al., 2022], serving as an alternative to more complex transformer architectures (e.g., Time-Transformer [Liu et al., 2025]).

The input to the model is a normalized stereo waveform (x \in \mathbb{R}^{2 \times T}) (where (T = 106810) samples, corresponding to a 5-second window at a sampling rate of (SR = 21362) Hz).

Convolutional Stack (1D-CNN): Employs 1D convolutional layers with a wide receptive field on the first layer to extract temporal amplitude envelopes and suppress high-frequency instrument noise.
Recurrent Block (BiLSTM): A bidirectional recurrent block captures bilateral cross-channel time-of-flight delay (TDOA).
Classification Head: A fully connected network with Dropout regularization projects features into three target classes: Normal, Fake, and Leak.

2.2. Dataset Design

To properly evaluate generalization capability, we partitioned the data into three isolated domains:

Clean Training Pool (D_0) (Global_Dataset): 306 genuine field files (128 Leak, 178 Normal) collected from municipal heating networks.
Out-of-Domain Test Pool (Held-out WDN Hong Kong): 140 files (70 Leak, 70 Normal) from Zayed et al. (2022), recorded using hydrophones and MEMS sensors on a municipal water distribution network in Hong Kong. This represents a true out-of-domain test with distinct attenuation and sensor properties.
Synthetic Pool (S) (Rivixi-FakeWAV v3): 60 files generated on Python with varying leak profiles, used to introduce contamination rates of (\varepsilon \in {0%, 1%, 5%, 20%}).

2.3. Synthetic Signal Generation and Physical Divergences

Heuristics of the Rivixi-FakeWAV v3 Generator:

The synthetic data pool was created using a Python script built by NDT engineers via LLM Claude. The utility objective of the generator was to bypass correlation leak detectors:

Noise Floor: Static, colored Gaussian noise is generated with a spectral slope of (-5.7) dB/decade and an RMS level of (-18.18) dBFS.
Leak Insertion: Series of 4–8 transient bursts are superimposed onto the noise. Each burst is a 2–3 kHz sinusoidal pulse modulated by a symmetric, 15 ms Gaussian window (to simulate NDT acoustic clicks).
TDOA Modeling: The inter-channel delay is simulated with sub-sample precision using linear phase shifting in the frequency domain (Phase Delay via FFT) using (t_1 = x/v) and (t_2 = (L-x)/v) with a steel sound propagation speed of (v = 5291) m/s.

Key Physical Differences:

Envelope Symmetry: Synthetic bursts feature mathematically perfect Gaussian symmetry. Physical wave packets disperse, skew, and attenuate irregularly due to soil friction and joint reflections.
Noise Stationarity: Synthetic background noise is perfectly static. Actual field recordings contain highly non-stationary interferences (valves, pumps, clipping, excavation activities).
Dispersion Absence: Synthetic signals shift all frequencies by an identical delay. Physical pipes induce wave dispersion (Lamb waves), causing different frequencies to travel at different velocities, which spreads the wave front.
Burst Regularity: Synthetic bursts occur at highly regular intervals. Physical leak emission is stochastic and chaotic.

3. Experimental Results

All models were trained from scratch (10 epochs, Adam optimizer, learning rate (10^{-4}), random weight initialization).

3.1. Internal Validation Results (Global_Dataset/Train)

At low contamination levels, the model displays an artificial boost across all metrics:

Training Mode	Accuracy	Precision	Recall	F1 Score	False Positives (FP)
B. Clean Scratch Training (0%)	97.06%	93.43%	100.00%	96.60%	9
C. Scratch Training + 1% Synthetic	99.67%	99.22%	100.00%	99.61%	1

Analysis of the "Straight-A Student" Illusion (99.67% Accuracy Deception)

The apparent performance boost (accuracy reaching 99.67% and false positives dropping from 9 to 1 with only 1% synthetic data) is a dangerous mathematical illusion.

Neural networks optimize loss functions via the path of least resistance (shortcut learning). In genuine physical signals, leak features are weak, dispersed, and masked by environmental noise—forcing the network to learn complex, high-dimensional filters.

Introducing clean synthetic signals provides the network with a "cheat code." Because these 1% synthetic files possess a pristine, high-contrast structure (perfect 15 ms Gaussian envelopes without dispersion), the model quickly learns that classifying them is trivial. It shifts its first-layer convolutional filters to detect this simple, clean template (the "perfect bell curve").

Consequently, the model passes its internal validation (the "homework test") with near-perfect scores because it has flattened its decision boundary around the synthetic shape. However, it loses its adaptability. When deployed in the field (the Hong Kong dataset), where acoustic envelopes are distorted by soil and lacking symmetry, the model searches for its clean "synthetic bell curve," fails to find it, and misses the leak. The internal validation metrics rose, but the real-world utility of the diagnostic tool collapsed.

3.2. Out-of-Domain Validation (Held-out WDN Hong Kong)

The performance degradation curve under synthetic contamination is detailed below:

Contamination (\varepsilon)	Accuracy	Precision	Recall	F1-Score	TP	FP	TN	FN
0% (Clean Baseline)	80.00%	87.50%	70.00%	77.78%	49	7	63	21
1% Contam	75.71%	84.62%	62.86%	72.13%	44	8	62	26
5% Contam	75.71%	89.13%	58.57%	70.69%	41	5	65	29
20% Contam	50.71%	53.33%	11.43%	18.82%	8	7	63	62

Fig. 1. Model performance degradation curve (Recall and Accuracy) on the independent out-of-domain test set (WDN Hong Kong) as a function of the synthetic contamination rate (\varepsilon) in the training dataset.

Critical Point and Model Collapse Analysis (20% Contamination)

Increasing the synthetic contamination to (\varepsilon = 5%) steadily degrades out-of-domain Recall to 58.57% (missing 29 physical leaks), as the network begins prioritizing the cleaner synthetic templates over noisy field signals.

However, once contamination hits 20%, a complete collapse of generalization capability occurs (Strong Model Collapse). Recall drops to an unacceptable 11.43%, meaning the neural network fails to detect 62 out of 70 real-world pipeline leaks.

The overall accuracy on the out-of-domain test pool drops to 50.71%. Given that the test set is balanced (70 Leak and 70 Normal), a score of 50.71% is mathematically equivalent to a random coin flip. The model has completely lost its physical predictive capacity. In an operational setting, a detector running this model would be useless, as its weights have overfitted to the artificial features of Rivixi-FakeWAV v3, rendering it blind to actual physical defects.

4. Discussion: Physical Mechanisms of Shortcut Learning

Spectral and time-domain analysis highlights the physical gaps between raw field data and heuristic synthetic models:

Group Velocity Dispersion: In physical pipes, wave speed varies with frequency (Lamb wave dispersion). High-frequency wave components arrive late, skewing the envelope. In the Rivixi-FakeWAV dataset, all frequencies share a static phase delay.
Noise Non-Stationarity: Real field noise contains fluctuating transients, pump harmonics, and sensor clipping. Synthetic noise is a stationary Gaussian process.
Envelope Shape: The perfect Gaussian shape (e^{-t^2/\sigma^2}) with (\sigma = 15) ms presents an excessively strong local minimum for gradient descent.

Under scratch training, the network optimizes for the easiest mathematical features (shortcut learning [Geirhos et al., 2020]). Instead of extracting weak, dispersed, and noisy physical shapes, the first-layer filters adapt exclusively to detect the perfect Gaussian bell curve. Consequently, the model fails on any real-world leak distorted by soil or pipe junctions.

Fig. 2. Time-domain comparison between a raw physical field recording (left) and the synthetic Rivixi-FakeWAV v3 signal (right). The real leak wave packet is dispersed, non-symmetric, and heavily corrupted by non-stationary noise, whereas the generated signal exhibits perfectly symmetric, mathematical Gaussian-windowed bursts over a stationary background.

5. Conclusions and Engineering Recommendations

Isolation of the Real Core (D_0): Simple mathematical wave generators must not be used in the main training dataset of deep wave models.
Mandatory Cross-Domain Validation: Internal validation metrics are deceptive in the presence of synthetic data. Any synthetic injection must be validated on independent out-of-domain datasets (e.g., the Zayed WDN dataset).
Transition to Physics-Informed Models (PINN): If synthetic data is necessary, generators must incorporate numerical integration of the Navier-Stokes equations and Lamb wave dispersion equations to simulate realistic wave fronts under physical constraints [Raissi et al., 2019].
Implementation of Authenticity Screening for WAV Files: Due to the risk of report falsification at the operator level, all incoming audio files must undergo automated authenticity screening. Simple "synthetic vs. real" classifiers should filter out files with abnormally high envelope symmetry, lack of frequency dispersion, and static backgrounds before they reach ML training databases.

Acknowledgments

The authors used AI-assisted tools for language editing and translation. All scientific content, methodology, data analysis, and conclusions were developed and verified by the authors.

References

Zayed, T., et al. (2022). Acoustic Based Data Acquisition for Leak Detection of Water Distribution Networks. Mendeley Data, V1. DOI: 10.17632/hkn8mxcjyz.1
Shumailov, I., et al. (2024). AI model collapse: training on synthetic data makes models forget. Nature. https://doi.org/10.1038/s41586-024-07566-y
Dohmatob, E., et al. (2025). Model Collapse in Language Models: Theoretical Limits and Solutions. arXiv:2404.01413
Ivanaiskii, A., Ivanaiskii, E., & Shipilov, S. (2026). Optimizing Hybrid Ultrasonic Testing Algorithms for District Heating Networks under Incomplete Spatial Metadata [Preprint]. Zenodo. DOI: 10.5281/zenodo.21431110
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665–673. DOI: 10.1038/s42256-020-00257-z
Fares, A., Tijani, I. A., Rui, Z., & Zayed, T. (2022). Leak detection in real water distribution networks based on acoustic emission and machine learning. Environmental Technology, 44(25), 3850–3866. DOI: 10.1080/09593330.2022.2081242
Liu, R., Zayed, T., Xiao, R., & Hu, Q. (2025). Time-transformer for acoustic leak detection in water distribution network. Journal of Civil Structural Health Monitoring, 15, 759–775. DOI: 10.1007/s13349-024-00898-y
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707. DOI: 10.1016/j.jcp.2018.10.045

Citation

@article{ivanaiskii2026illusion,
  title={The Illusion of Generalization: Impact of Ultra-Low Synthetic Contamination on Deep 1D-CNN Scratch Training in Acoustic Leak Detection},
  author={Ivanaiskii, Aleksandr and Ivanaiskii, Evgeny and Shipilov, Sergei},
  journal={Rivixi Research Preprint},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.21431110}
}

The Illusion of Generalization: Impact of Ultra-Low Synthetic Contamination on Deep Waveform AI Scratch Training in Acoustic Leak Detection