**End-to-end NLP toolkit: text preprocessing, embeddings, classification, NER, fine-tuning, and RAG — with stdlib reference implementations that run without a GPU.**
Browse the actual product documentation and code examples included in this toolkit.
Key features of NLP Starter Kit
• **Embedding engine** — pluggable backends for TF-IDF (stdlib), sentence-transformers (SBERT), and OpenAI-compatible APIs, with caching, batch optimization, and cosine similarity utilities • **Text classifier** — nearest-centroid baseline (stdlib, no sklearn) and Hugging Face transformer backend, with full metric computation (accuracy, precision, recall, F1, confusion matrix) • **Named entity recognition** — rule-based extractor with regex patterns and gazetteers, plus transformer-based NER (Hugging Face token-classification), with span merging and deduplication • **Fine-tuning pipeline** — Hugging Face Trainer wrapper with LoRA/PEFT support, learning rate schedulers, gradient accumulation, and checkpoint management • **RAG pipeline** — document chunking, in-memory vector store, mock LLM client, and prompt assembly — the full retrieval-augmented generation loop runs on stdlib alone • **Comprehensive test suite** — 50+ test cases covering all modules, runnable without external dependencies
**Embedding engine** — pluggable backends for TF-IDF (stdlib), sentence-transformers (SBERT), and OpenAI-compatible APIs, with caching, batch optimization, and cosine similarity utilities
**Text classifier** — nearest-centroid baseline (stdlib, no sklearn) and Hugging Face transformer backend, with full metric computation (accuracy, precision, recall, F1, confusion matrix)
**Named entity recognition** — rule-based extractor with regex patterns and gazetteers, plus transformer-based NER (Hugging Face token-classification), with span merging and deduplication
**Fine-tuning pipeline** — Hugging Face Trainer wrapper with LoRA/PEFT support, learning rate schedulers, gradient accumulation, and checkpoint management
**RAG pipeline** — document chunking, in-memory vector store, mock LLM client, and prompt assembly — the full retrieval-augmented generation loop runs on stdlib alone
**Comprehensive test suite** — 50+ test cases covering all modules, runnable without external dependencies
Configure NLP Starter Kit parameters to see how the product works.
pip install -r requirements.txt