← Back to all products
$29
Embedding Generator
Python text embedding pipeline with tokenization, vector generation, similarity search, and caching.
MarkdownPython
📄 Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample📁 File Structure 11 files
embedding-generator/
├── LICENSE
├── README.md
├── examples/
│ ├── basic_usage.py
│ └── sample_corpus.txt
├── free-sample.zip
├── guide/
│ ├── 01_features.md
│ ├── 02_quick-start.md
│ ├── 03_configuration.md
│ └── 04_license.md
├── index.html
└── src/
└── embedding_generator.py
📖 Documentation Preview README excerpt
Embedding Generator
Python text embedding pipeline with tokenization, vector generation, similarity search, and caching. Zero dependencies.
Part of the AI Toolkit collection by [CodeVault](https://ai-toolkit.codevault.dev).
Features
- Multiple embedding methods — Hash-trick, Bag-of-Words, and TF-IDF
- Tokenizer — Configurable tokenization with stop-word removal
- Similarity search — Cosine similarity ranking over a corpus
- LRU caching — Automatic embedding cache with configurable size
- Corpus builder — Build vocabulary and IDF from your documents
- CLI interface — Embed text, load corpora, and search from terminal
- Zero dependencies — Python stdlib only
Quick Start
# Embed a single text
python src/embedding_generator.py --text "machine learning is great"
# Load a corpus and search
python src/embedding_generator.py --file examples/sample_corpus.txt --query "neural networks"
# Interactive mode
python src/embedding_generator.py
# Use TF-IDF method
python src/embedding_generator.py --method tfidf --file examples/sample_corpus.txt --query "data science"
Configuration
| Flag | Default | Description |
|---|---|---|
--method | hash | Embedding method: hash, bow, tfidf |
--dim | 128 | Vector dimension (hash method only) |
--top-k | 5 | Number of search results |
--text | — | Single text to embed |
--file | — | Corpus file (one doc per line) |
--query | — | Search query (requires --file) |
License
MIT — use in personal, commercial, or client projects. No attribution required.
📄 Code Sample .py preview
examples/basic_usage.py#!/usr/bin/env python3
"""Basic usage of the Embedding Generator."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
from embedding_generator import EmbeddingEngine, cosine_similarity
def main() -> None:
engine = EmbeddingEngine(method="hash", dim=128)
# Embed two texts and compare
a = engine.embed("Machine learning is a subset of artificial intelligence")
b = engine.embed("AI and ML are closely related fields")
c = engine.embed("The weather today is sunny and warm")
print(f"Similarity (ML vs AI): {cosine_similarity(a.vector, b.vector):.4f}")
print(f"Similarity (ML vs Weather): {cosine_similarity(a.vector, c.vector):.4f}")
# Corpus search
corpus = [
"Python is great for data science",
"JavaScript runs in the browser",
"Machine learning models need training data",
"Web servers handle HTTP requests",
"Neural networks learn from examples",
]
engine.build_corpus(corpus)
results = engine.search("deep learning training", top_k=3)
print("\nSearch: 'deep learning training'")
for text, score in results:
print(f" [{score:.4f}] {text}")
if __name__ == "__main__":
main()