Continued Pretraining for Domain Agents: When Regulatory Vocabulary Justifies the Cost

*Series: LLM Training & Tuning for Agentic AI — Part 6 of 8*

*Parts 1–5 covered the decision framework, prompt engineering, RAG, agent memory, and LoRA/QLoRA fine-tuning. This part goes deep on Step 5 — continued pretraining, the most expensive step before full pretraining itself — with a compliance and regulatory review agent as the working example, current as of mid-2026.*

Executive Summary

Continued pretraining — also called continual pretraining (CPT) or domain-adaptive pretraining (DAPT) — resumes unsupervised training on an existing base model using a large corpus of unlabeled domain text. It is the step in the decision framework most often reached for too early and, when the signal is real, most often executed without the forgetting safeguards that determine whether it helps or quietly degrades the model.

It remains a legitimate technique in 2026 precisely because base-model knowledge cutoffs lag and regulated domains move fast, but it should only be reached after the earlier steps in the framework — prompting, RAG, memory, LoRA fine-tuning — have been tried and have left a broad, representational gap that none of them closed.

This article covers what continued pretraining actually changes, the data curation and forgetting-mitigation discipline that separates a useful domain checkpoint from a degraded one, the three-layer evaluation every continued-pretrained model needs, and walks through a compliance and regulatory review agent where this step was genuinely earned rather than assumed.

Introduction

In enterprise environments, continued pretraining gets reached for when a team notices an agent consistently misunderstands domain terminology, acronyms, or document structure — and mistakes that pattern for a fine-tuning problem. The distinction matters: fine-tuning teaches behavior on labeled examples for a specific task, while continued pretraining shifts the model's underlying representations across an entire domain, using unsupervised training on unlabeled text with no task structure at all.

The signal that genuinely points here is broad — the model struggles with domain vocabulary and structure across many different tasks, not just one — and it should only be considered after RAG and LoRA fine-tuning have already been tried and have left that broader gap unaddressed.

What Continued Pretraining Actually Changes

The foundational research behind this technique, introduced as domain-adaptive pretraining, showed that a second phase of pretraining on a target domain's corpus measurably improves a model's downstream performance in that domain, even when task-specific fine-tuning is applied afterward — the domain adaptation and the task adaptation are complementary, not substitutes for each other.

This is why the four-stage pattern common in 2026 enterprise practice — continued pretraining, then LoRA supervised fine-tuning, then preference alignment such as DPO, then rejection-sampled fine-tuning — treats continued pretraining as the foundation the later stages build on, not a replacement for them.

The Catastrophic Forgetting Problem, and What Actually Mitigates It

Further pretraining on a narrow domain corpus can overwrite parameters the model relies on for general language understanding, degrading capabilities that have nothing to do with the target domain. This is not a hypothetical risk — it is the primary failure mode of continued pretraining, and current research and practice converge on a small set of real mitigations rather than one silver bullet.

• Data replay: mixing a portion of general-domain data into the continued pretraining corpus alongside the domain-specific text, rather than training on domain text exclusively.
• Optimal data composition: recent work has shifted from simple replay ratios toward deliberately tuning the mix of domain and general instruction data for the specific model and domain.
• Architecture-level approaches: masking updates to the parameters most responsible for general-domain performance, or adding new layers and training only those, so the original capacity is preserved rather than overwritten — more advanced techniques than most enterprise teams need, but worth knowing exist for cases where standard replay isn't sufficient.

For most enterprise domain-adaptation projects, a well-tuned data replay ratio combined with the three-layer evaluation below is sufficient; the architecture-level techniques are typically reserved for teams pretraining at a scale where a percentage point of retained general capability has outsized value.

Data Preparation for a Regulatory Corpus

Regulatory and compliance text brings its own preparation challenges beyond generic domain adaptation: scanned filings that need OCR cleanup, boilerplate and duplicate language across near-identical filings that needs deduplication, and content that should never enter the training corpus at all — draft language, privileged legal communications, or anything a compliance team explicitly wants the model not to know.

Current practice treats this last category as a compliance gate applied during data preparation, not an afterthought applied after training: content is filtered out before it ever reaches a GPU, alongside standard quality filters such as perplexity-based filtering, embedding-cluster deduplication, and PII redaction.

The Three-Layer Evaluation Gate

A continued-pretrained checkpoint should be evaluated on three distinct layers before it becomes the foundation for anything downstream, and passing on one layer does not substitute for the others.

Tooling and Cost Landscape

As of mid-2026, framework choice generally follows scale: NeMo or Megatron-style frameworks for the largest continued pretraining runs, DeepSpeed for memory-constrained clusters, and Axolotl or Unsloth for more ergonomic, parameter-efficient continued pretraining on smaller domain corpora.

Some enterprise guides describe a full four-stage pipeline — continued pretraining, LoRA supervised fine-tuning, online preference alignment, and rejection-sampled fine-tuning — completed in roughly six to twelve weeks at a cost commonly cited in the tens to low hundreds of thousands of dollars, though this figure varies enormously with corpus size, model scale, and infrastructure choices, and should be validated against current vendor quotes rather than treated as a fixed budget line.

Compliance and Regulatory Review Agent: Where This Step Was Actually Earned

A mid-sized financial services firm built a compliance review agent to flag regulatory risk in customer communications and marketing materials before they went out the door. The team followed the decision framework in order: a well-structured prompt with the five-layer approach from Part 2 handled straightforward cases, and RAG grounded in the firm's actual policy documents from Part 3 handled lookups against known rules.

But a specific pattern kept showing up in review: the agent misread dense, citation-heavy regulatory language — nested cross-references between sections, defined terms that shift meaning across different rule sets, and the compressed syntax common to filings — not on one task, but across every task that touched this kind of text, including tasks where the correct policy document had already been retrieved and was sitting right there in context.

This was the specific signal the decision framework points to: a broad representational gap in domain vocabulary and structure, present across many tasks, that neither better prompting nor better retrieval was closing, because the model wasn't failing to find the right information — it was failing to parse the language it was already looking at correctly.

The team built a continued pretraining corpus from several years of regulatory filings, prior compliance rulings, and internal policy documentation, running it through deduplication, OCR cleanup for older scanned filings, and a compliance gate that excluded privileged legal correspondence and anything the legal team flagged as never to be used as training data.

Training used a data replay ratio mixing general-domain text alongside the regulatory corpus, specifically to protect the model's broader language and reasoning capability rather than letting it narrow entirely around regulatory syntax. The resulting checkpoint went through all three evaluation layers before anyone treated it as production-ready: capability retention held steady against the pre-adaptation baseline on general tasks, domain gain showed a clear, measurable improvement in correctly parsing nested cross-references and defined terms on a held-out set of filings, and downstream usefulness confirmed the improvement actually reduced misclassification on the compliance team's real flagging task, not just on a generic reading-comprehension benchmark.

Only after this checkpoint passed all three layers did the team layer a LoRA adapter on top for the specific structured-flagging output format — continued pretraining solved the comprehension gap, and fine-tuning from Part 5 handled the task-specific behavior on top of it, exactly the division of labor the framework describes.

Enterprise Architecture Perspective

From an architecture perspective, continued pretraining should be understood as raising the floor for every downstream method, not replacing them. A domain-adapted checkpoint still needs RAG for current, specific facts; it still needs prompt engineering to shape task behavior; it still typically needs LoRA fine-tuning for output format and task-specific decisions.

Teams that treat continued pretraining as a one-time investment that eliminates the need for the rest of the stack are solving a comprehension problem while leaving the knowledge-freshness, behavior-shaping, and format-compliance problems exactly where they were.

Best Practices

• Confirm the gap is broad and representational — present across many tasks, not one — before considering continued pretraining.
• Apply compliance gates and PII redaction during data preparation, not as a post-hoc filter after training.
• Use data replay to mix general-domain text into the training corpus as a default forgetting mitigation.
• Evaluate on all three layers — capability retention, domain gain, and downstream usefulness — before treating a checkpoint as production-ready.
• Plan for continued pretraining to be followed by fine-tuning and prompt engineering, not to replace them.

Anti-Patterns

• Reaching for continued pretraining before confirming the gap isn't actually a RAG or LoRA fine-tuning problem in disguise.
• Training exclusively on domain text with no data replay, risking measurable degradation of general capability.
• Treating domain gain on a generic benchmark as sufficient evidence without checking downstream task usefulness.
• Skipping a compliance review of the training corpus itself, risking privileged or excluded content entering the model's weights.

Frequently Asked Questions

How is continued pretraining different from LoRA fine-tuning on domain text?

Continued pretraining is unsupervised — unlabeled text, next-token prediction, no task structure — and shifts the model's underlying representations across a domain. LoRA fine-tuning is supervised, uses labeled input/output examples, and teaches a specific task or behavior. They solve different problems and are commonly used together, with continued pretraining first.

Can continued pretraining replace RAG for a compliance agent?

No. Continued pretraining improves comprehension of domain language and structure; it does not keep the model current on specific, changing facts the way RAG does. A compliance agent still needs RAG grounded in the actual current policy documents even after continued pretraining improves how well it parses them.

What is the minimum evidence needed before committing to continued pretraining?

A documented pattern of comprehension failures on domain vocabulary or structure across multiple distinct tasks, with prompting and RAG already ruled out as the cause on a representative test set. A single failing task is a signal to look at fine-tuning first, not continued pretraining.

Key Takeaways

• Continued pretraining shifts a model's representations across an entire domain through unsupervised training — a different mechanism from fine-tuning's task-specific supervised learning.
• The signal that justifies it is broad: comprehension failures across many tasks, not one — and it should only follow, not precede, ruling out RAG and LoRA fine-tuning.
• Catastrophic forgetting is the primary risk, and data replay is the standard, practical mitigation for most enterprise projects.
• A continued-pretrained checkpoint needs evaluation on capability retention, domain gain, and downstream usefulness — all three, not just one.
• Continued pretraining raises the floor for everything downstream; it doesn't replace RAG, prompting, or fine-tuning.

References for Further Reading

• Gururangan et al., "Don't Stop Pretraining: Adapt Language Models to Domains and Tasks" — the foundational domain-adaptive pretraining paper.
• McCloskey and Cohen, "Catastrophic Interference in Connectionist Networks" — the original catastrophic forgetting research underlying this risk.
• NeMo, Megatron-LM, DeepSpeed, Axolotl, and Unsloth documentation — for current framework specifics, check live documentation, as this tooling continues to evolve.

Coming Next

Part 7 covers preference alignment — DPO versus RLHF — using a tone and citation-behaviour agent as the working example, including why DPO has become the more common enterprise choice.