Fine-Tuning Guide

How to fine-tune pre-trained language models for your specific NLP tasks.

When to Fine-Tune vs. Use Zero-Shot

Fine-tune when: You have labeled data, need low latency, want to avoid API costs, or need domain-specific accuracy that general models cannot match.

Choosing a Base Model

Quick Start: Classification Fine-Tuning

from nlp_starter_kit.fine_tuning import FineTuningConfig, HuggingFaceFineTuner

config = FineTuningConfig(
    model_name="distilbert-base-uncased",
    num_epochs=3,
    batch_size=16,
    learning_rate=2e-5,
)

tuner = HuggingFaceFineTuner(config)
tuner.prepare_data(
    train_texts=["Great product!", "Terrible service", ...],
    train_labels=["positive", "negative", ...],
    eval_texts=["Loved it!", "Would not recommend", ...],
    eval_labels=["positive", "negative", ...],
)

metrics = tuner.train()
print(f"Accuracy: {metrics['eval_accuracy']}")

tuner.save_model("./my-classifier")

LoRA Fine-Tuning (Parameter-Efficient)

LoRA (Low-Rank Adaptation) freezes the base model and injects small trainable matrices into attention layers. Benefits:

• 95%+ parameter reduction — train 1M params instead of 100M
• Fits on consumer GPUs — fine-tune BERT-large on 8GB VRAM
• Fast training — fewer gradients to compute
• Easy model merging — combine multiple LoRA adapters

config = FineTuningConfig(
    model_name="bert-large-uncased",
    use_peft=True,
    lora_r=8,            # Rank — lower = fewer params, less expressive
    lora_alpha=16,        # Scaling factor — typically 2x lora_r
    lora_dropout=0.05,    # Regularization
    lora_target_modules=["query", "value"],  # Which attention matrices
    learning_rate=3e-4,   # LoRA uses higher LR than full fine-tuning
    num_epochs=5,
    batch_size=8,
    gradient_accumulation_steps=4,  # Effective batch = 32
    fp16=True,
)

LoRA Rank Selection

Learning Rate Scheduling

The kit provides two schedulers, both matching Hugging Face Trainer behavior:

Linear with Warmup (Default)

LR
│  /‾‾‾‾‾‾‾‾‾‾‾‾\
│ /                \
│/                  \
└──────────────────── Steps
  warmup    decay

Good for most tasks. The warmup prevents early training instability.

Cosine with Warmup

LR
│  /‾‾‾‾‾\_
│ /         ‾‾\_
│/               ‾‾‾‾
└──────────────────── Steps
  warmup    cosine decay

Better for longer training runs. The gradual decay lets the model explore longer before converging.

Hyperparameter Recommendations

Dataset Size → Hyperparameters

Gradient Accumulation

If your GPU cannot fit batch_size=32, use gradient accumulation:

batch_size=8,
gradient_accumulation_steps=4,  # Effective batch = 8 × 4 = 32

The optimizer updates after accumulating gradients from 4 mini-batches, giving the same training dynamics as a larger batch without the memory cost.

Evaluation During Training

Always evaluate during training to catch overfitting early:

config = FineTuningConfig(
    eval_strategy="epoch",      # Evaluate after each epoch
    save_strategy="epoch",      # Save checkpoint after each epoch
    save_total_limit=3,         # Keep only 3 most recent checkpoints
)

Watch for:

• Training loss drops, eval loss rises → Overfitting. Reduce epochs or increase regularization.
• Eval loss plateaus → Learning rate may be too low, or model has converged.
• Training loss doesn't decrease → Learning rate too low, or data issues.

Common Mistakes

1. Using the wrong casing. If you're doing NER, use a cased model (bert-base-cased). Uncased models lose important entity signals.

2. Too many epochs on small data. With <1000 examples, 3 epochs of full fine-tuning will overfit. Use LoRA or early stopping.

3. Not shuffling training data. If all positive examples come before negative examples, the model learns order, not content. Hugging Face Trainer shuffles by default.

4. Ignoring class imbalance. If 90% of your data is one class, the model will just predict that class. Use weighted loss or oversample the minority class.

5. Not saving the tokenizer. When you save a fine-tuned model, always save the tokenizer too — it defines how text is converted to input IDs.

Text Preprocessing Guide

How to build robust text cleaning and normalization pipelines that handle real-world data.

Why Preprocessing Matters

Raw text from the wild is messy. Web scrapes contain HTML tags, user-generated content has inconsistent casing and Unicode, CSVs from different systems encode the same characters differently. If you skip preprocessing, your model trains on noise.

The NLP Starter Kit provides three preprocessing layers, each solving a distinct problem:

Pipeline Architecture

Raw Text → Cleaner → Normalizer → Tokenizer → Clean Tokens

Each stage is independently configurable. You can use any combination — skip cleaning if your data is already clean, skip tokenization if you just need normalized strings.

Cleaning Recipes

Recipe 1: Web Scrape Cleanup

from nlp_starter_kit.text_processor import TextProcessor, CleaningConfig

processor = TextProcessor(
    cleaning=CleaningConfig(
        strip_html=True,
        remove_urls=True,
        remove_emails=True,
        remove_mentions=False,
        custom_patterns=[
            r"\[(?:ref|cite|note):\S+\]",   # Remove inline references
            r"(?:Advertisement|Sponsored)\s*$",  # Remove ad markers
        ],
    ),
    normalize=True,
    normalizer_kwargs={"lowercase": True, "strip_accents": False},
)

result = processor.process(html_content)
clean_text = result.normalized

Recipe 2: Social Media Text

# For social media, you often WANT mentions and hashtags as features
processor = TextProcessor(
    cleaning=CleaningConfig(
        strip_html=False,
        remove_urls=True,
        remove_mentions=False,     # Keep @mentions as features
        remove_hashtags=False,     # Keep hashtags
        remove_punctuation=False,  # Punctuation carries sentiment
    ),
    normalize=True,
    normalizer_kwargs={"lowercase": True},
    tokenize=True,
    tokenizer_kwargs={"lowercase": True, "keep_punctuation": True},
)

Recipe 3: Search Index Preparation

# For search, strip everything down to normalized word stems
processor = TextProcessor(
    cleaning=CleaningConfig(
        strip_html=True,
        remove_urls=True,
        remove_emails=True,
        remove_punctuation=True,
        remove_numbers=False,      # Numbers can be search terms
        remove_stopwords=True,     # "the", "is", "at" add no search value
        min_token_length=2,        # Single-char tokens are noise
    ),
    normalize=True,
    normalizer_kwargs={
        "lowercase": True,
        "strip_accents": True,     # café = cafe for search
        "unicode_form": "NFKC",
    },
)

Unicode Normalization Deep Dive

Unicode normalization is the most misunderstood preprocessing step. Here is when to use each form:

Always use NFKC for ML. It normalizes ligatures (fi→fi), superscripts (²→2), and fullwidth characters (Ａ→A) that would otherwise create duplicate vocabulary entries.

Stopword Considerations

The built-in stopword list covers ~175 English function words. But stopword removal is not always beneficial:

Remove stopwords when:

• Building TF-IDF vectors (stopwords dominate term frequency)
• Constructing search indexes
• Doing keyword extraction

Keep stopwords when:

• Training transformer models (they learn context from function words)
• Doing sentiment analysis ("not good" loses meaning without "not")
• Training language models (they need natural text distribution)

Batch Processing Performance

For large datasets, always use batch methods:

# Slow — creates a new processor per call
results = [TextProcessor().process(t) for t in texts]

# Fast — reuses compiled patterns and state
processor = TextProcessor(cleaning=CleaningConfig(remove_urls=True))
results = processor.process_batch(texts)

The batch methods avoid recompiling regex patterns on every call. For 100k+ texts, this makes a measurable difference.

Common Pitfalls

1. Cleaning before normalization: Always clean first. HTML entity decoding (part of cleaning) can introduce characters that normalization needs to handle.

2. Over-aggressive cleaning: Removing all punctuation destroys sentence boundaries, which matters for sentence-level tasks. Only remove punctuation if you truly don't need it.

3. Not handling encoding: If your input files mix UTF-8 and Latin-1, normalize encoding _before_ the text pipeline. Python's str type is always Unicode, but reading from files can produce garbage if the encoding is wrong.

4. Forgetting offset tracking: If you clean text and then run NER on it, the entity character offsets will not match the original text. Either run NER before cleaning, or use the tokenizer's offset tracking to map back.