NLP Starter Kit

NLP Starter Kit

End-to-end NLP toolkit: text preprocessing, embeddings, classification, NER, fine-tuning, and RAG — with stdlib reference implementations that run without a GPU.

Go from raw text to trained models and retrieval-augmented generation with a single coherent toolkit. Every component includes both a stdlib baseline (runs anywhere) and a production backend (Hugging Face, sentence-transformers, LoRA).

What You Get

• Text preprocessing pipeline — tokenization with offset tracking, Unicode normalization (NFC/NFKD/NFKC), HTML/URL/email cleaning, stopword filtering, all composable into a single TextProcessor call
• Embedding engine — pluggable backends for TF-IDF (stdlib), sentence-transformers (SBERT), and OpenAI-compatible APIs, with caching, batch optimization, and cosine similarity utilities
• Text classifier — nearest-centroid baseline (stdlib, no sklearn) and Hugging Face transformer backend, with full metric computation (accuracy, precision, recall, F1, confusion matrix)
• Named entity recognition — rule-based extractor with regex patterns and gazetteers, plus transformer-based NER (Hugging Face token-classification), with span merging and deduplication
• Fine-tuning pipeline — Hugging Face Trainer wrapper with LoRA/PEFT support, learning rate schedulers, gradient accumulation, and checkpoint management
• RAG pipeline — document chunking, in-memory vector store, mock LLM client, and prompt assembly — the full retrieval-augmented generation loop runs on stdlib alone
• Comprehensive test suite — 50+ test cases covering all modules, runnable without external dependencies
• Configuration files — YAML configs for pipeline settings and fine-tuning hyperparameters with presets for common scenarios

File Tree

nlp-starter-kit/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   └── nlp_starter_kit/
│       ├── __init__.py              # Package exports
│       ├── text_processor.py        # Tokenization, normalization, cleaning
│       ├── embeddings.py            # Embedding backends and cosine similarity
│       ├── classifier.py            # Text classification with metrics
│       ├── ner.py                   # Named entity recognition
│       ├── fine_tuning.py           # HF fine-tuning with LoRA support
│       └── rag.py                   # RAG pipeline with vector store
├── configs/
│   ├── pipeline_config.yaml         # Processing pipeline settings
│   └── model_config.yaml            # Fine-tuning hyperparameter presets
├── tests/
│   ├── test_text_processor.py       # Tokenizer, normalizer, cleaner tests
│   ├── test_classifier_ner.py       # Classification and NER tests
│   └── test_rag.py                  # RAG pipeline tests
└── guides/
    ├── text-preprocessing.md        # Preprocessing recipes and pitfalls
    └── fine-tuning.md               # Fine-tuning guide with LoRA walkthrough

Quick Start

1. Install Dependencies

pip install -r requirements.txt

Or for stdlib-only usage (no GPU, no external packages):

# Just use the toolkit directly — TF-IDF embeddings, nearest-centroid
# classifier, rule-based NER, and mock LLM all work on stdlib alone.

*... continues with setup instructions, usage examples, and more.*