Overview
Every AI model is a product of its training data — the decisions about what data was collected, from where, and how it was processed shape everything the model will ever do, and carry security implications that begin long before the model is deployed. Those risks originate in a data supply chain that is usually invisible, poorly documented, and almost entirely unaudited. This note covers that supply chain, the key concepts behind building and optimizing a model, the risks inherited when building on someone else’s pre-trained work, and the fundamental “black box” problem of trained models. See AI & ML Fundamentals for the broader AI/ML/DL/LLM hierarchy and terminology this note builds on.
Terminology
| Term | Definition |
|---|---|
| Data Provenance | The ability to determine where a piece of training data came from, when it was collected, and whether it has been modified since |
| ML-BOM | AI-equivalent of a software bill of materials (SBOM) — a documented inventory of dataset sources, licenses, PII categories, and filtering decisions |
| Pre-Trained Model | A model already trained on a large, general-purpose dataset and made available for others to build on (open weights like LLaMA, or API access like OpenAI’s) |
| Fine-Tuning | The process of continuing to train a pre-trained model on a smaller, task-specific dataset |
| Model Validation | Holding back a portion of training data, never used for training, and testing the model against it during training to check whether it is generalizing or just memorizing |
| Pruning | Removing parameters that contribute little to a model’s predictions, shrinking its size |
| Quantization | Reducing the numerical precision of a model’s weights (e.g., 32-bit to 8-bit floats) to cut memory and compute requirements |
| Federated Learning | Training a model across many decentralized devices or organizations, where each participant trains locally and only sends weight updates — not raw data — to a central server for aggregation |
| Model Card | A structured document accompanying a model that describes what it is, how it was built, and where it falls short |
Core Concepts
The Training Data Supply Chain
Training a large language model requires a staggering amount of text — GPT-3 was trained on 570GB of filtered text, and that is considered modest by current standards. That data is drawn from four general buckets, each with a different trust profile:
| Source | What It Is | Trust Profile |
|---|---|---|
| Web Scraping | Automated crawls of public internet content (news, forums, blogs, social media) | Low — no curator, no version control, content changes after collection |
| Licensed Datasets | Data purchased or agreed upon with platforms (e.g., OpenAI + Reddit, Meta’s own social posts) | Medium — terms are often unclear, and original users rarely consented to AI training use |
| Synthetic Data | AI-generated content used to train further AI systems | Variable — growing quickly; a meaningful and rising share of fine-tuning datasets now contain LLM-generated content |
| Internal Corpora | Company knowledge bases, support transcripts, clinical notes used for fine-tuning | Higher — the organization has direct control, but also direct liability if mishandled |
The most widely used training dataset is Common Crawl — a free, publicly available archive of web crawl data. Most modern models use it as a core layer and build on top of it with trillions of additional tokens across many languages. How that raw data was filtered — and by whom — matters as much as where it came from, since filtering decisions determine what slips through into the final training set.
Data Provenance and the ML-BOM Problem
Data provenance asks three questions about any piece of training data: where did it come from, when was it collected, and has it been modified since? For most major models, the honest answer today is that nobody fully knows — models are frequently trained on datasets built from other datasets, and original attribution is lost, simplified, or never recorded in the first place.
The software industry solved a version of this problem with software bills of materials (SBOMs); the AI equivalent is the ML-BOM — a documented inventory of dataset sources, licenses, PII categories, and filtering decisions. Most organizations do not have anything close to one.
PII in the Training Pipeline
Because large-scale web scraping is rarely audited, personally identifiable information (PII) ends up baked directly into model weights — and once it is there, it is very difficult to remove. With the right prompting, a model trained on PII can sometimes be coaxed into surfacing training content near-verbatim, including credentials.
Epochs, Overfitting, and Model Validation
An epoch is one complete pass of the training algorithm through the entire dataset; models are typically trained over many epochs. More epochs do not always produce a better model — training for too long causes a model to stop learning general patterns and instead memorize the training data itself, a failure mode already defined as overfitting. An overfit model performs well on the data it was trained on but poorly on anything new, and is more likely to memorize and reproduce sensitive specifics from its training set when prompted.
Model validation is the primary defense against this: a portion of the training data is held back and never used for training. At regular intervals during training, the model is tested against this unseen data to confirm its performance is genuinely generalizing rather than just improving on examples it has already memorized — making validation the quality gate of the ML lifecycle.
Post-Training Optimization: Pruning and Quantization
Once trained, a model typically goes through a compression pass before deployment, especially if it needs to run efficiently on limited hardware:
| Technique | What It Does | Security Consideration |
|---|---|---|
| Pruning | Removes parameters that contribute little to predictions, shrinking model size | Changes model behavior post-training; rarely documented in detail |
| Quantization | Reduces the numerical precision of weights to cut memory and compute requirements | Can degrade safety-aligned behavior; backdoor defenses tested on full-precision models may fail to detect threats in quantized versions |
Both steps are usually applied by a third-party team packaging the model for distribution. Quantization in particular can silently degrade the safety mechanisms built into a model — when an organization downloads a quantized model with no documentation of what changed, it inherits unknown behavioral modifications alongside the efficiency gains.
Federated Learning
In federated learning, a model is trained across many decentralized devices or organizations, with each participant training locally on their own data and sending only weight updates — never the raw data — back to a central server for aggregation. This is designed with privacy in mind and can meaningfully reduce data-level privacy risk, such as a hospital training a model on patient data without the data ever leaving the hospital. The tradeoff is that the integrity of the training process is much harder to verify than in a centralized setup — participants can submit poisoned local updates, and these can be difficult to detect at the aggregation stage.
The Inheritance Problem: Building on Someone Else’s Work
Training an LLM from scratch is resource-intensive and expensive enough that it is not realistic for most organizations. Instead, most AI development starts by building on a pre-trained model — one already trained on a large, general-purpose dataset to learn broad skills like language understanding, grammar, facts, and reasoning patterns. Organizations then apply fine-tuning, continuing to train that pre-trained model on a smaller, task-specific dataset (a law firm training on case law, a healthcare company training on clinical documentation). Fine-tuning changes the model’s task-specific behavior, tone, and domain knowledge — it does not change the base model’s underlying weights, which were shaped by pre-training data the fine-tuning organization never saw and probably never audited.
This creates an inheritance problem: after fine-tuning, an organization inherits everything the base model already contains, including things it cannot see and did not choose. This shows up in three main ways:
- Safety alignment erodes, it doesn’t break — a model is trained to follow a safe response path, but as fine-tuning introduces more paths, they can obscure the original one, shifting the probability of an unsafe response upward over time rather than causing an obvious failure.
- Specialization increases attack surface — fine-tuning narrows a model’s focus, which reduces its resilience to unexpected tokens and makes it more susceptible to prompt injection than the base model. For example, a model fine-tuned on financial records gets better at financial reasoning, but becomes more responsive to an attacker who frames a malicious prompt in financial terms.
- Version matters, and it’s rarely tracked — if the base model and its version aren’t tracked, and that base model is later found to have a security risk (a backdoor, or problematic training data), every fine-tuned model downstream of it inherits that same risk.
The Black Box Problem
Traditional software has source code that can be audited, and even compiled binaries can be decompiled, disassembled, and stepped through. A trained model’s weights are just billions of floating-point numbers with no human-readable record of how they were shaped — making the model a black box in the truest sense. Trusting a model means trusting the process that produced it; testing a model’s outputs is only sampling, not auditing, and there is no way to know how it will behave on an input that has never been tried.
Model Cards: The Closest Thing to Transparency
A model card is a structured document accompanying a model that describes what it is, how it was built, and where it falls short — the closest thing the industry has to a standard transparency format:
| Section | What It Should Tell You |
|---|---|
| Training Data | What sources were used, how they were filtered, known gaps or biases |
| Intended Use | What the model was designed for — and explicitly what it wasn’t |
| Evaluation Results | Performance metrics across different conditions and demographics |
| Known Limitations | Conditions under which the model is known to underperform or behave unexpectedly |
| Bias Assessment | Where training data or evaluation may have introduced skew |
| License | What you’re legally permitted to do with the model |
In practice, model cards are frequently incomplete, vague, or absent entirely — there is no regulatory requirement to produce one, and as of now it remains voluntary for most use cases.