NLP Engineering: From Text to Production

NLP before transformers required careful feature engineering: bag-of-words, TF-IDF, hand-crafted n-gram features. After transformers, the problem became: which pretrained model, how to fine-tune it, and how to serve it efficiently. Both eras taught different things; understanding both makes you a better practitioner.

The Modern NLP Stack

Raw text
    ↓ Preprocessing (normalization, tokenization)
    ↓ Embedding (pretrained model or fine-tuned)
    ↓ Task head (classification, generation, extraction)
    ↓ Inference (batched, quantized, cached)
    ↓ Production API

Tokenization: What Actually Happens

Most engineers treat tokenization as a black box. Understanding it matters:

pythonCopy
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "The quick brown fox jumps over the lazy dog."
tokens = tokenizer(text, return_tensors="pt")

print(tokens.input_ids)       # tensor of token IDs
print(tokenizer.convert_ids_to_tokens(tokens.input_ids[0]))
# ['[CLS]', 'the', 'quick', 'brown', 'fox', 'jump', '##s', 'over', 'the', 'lazy', 'dog', '.', '[SEP]']

BERT uses WordPiece: "jumps" becomes "jump" + "##s". GPT uses BPE (Byte-Pair Encoding). Both create subword vocabularies that handle unknown words by splitting them into known subwords.

Why it matters: your "word count" is not the same as your "token count." Code, technical jargon, non-English text, and numbers often tokenize inefficiently (more tokens per word). This affects cost, context window usage, and model behavior.

Text Classification with Fine-Tuning

pythonCopy
from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer
from datasets import Dataset
import torch

# Prepare data
texts = ["I love this product", "Terrible quality, waste of money", ...]
labels = [1, 0, ...]  # 1=positive, 0=negative

dataset = Dataset.from_dict({"text": texts, "label": labels})

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, padding=True, max_length=128)

tokenized_dataset = dataset.map(tokenize, batched=True)
train_dataset, val_dataset = tokenized_dataset.train_test_split(test_size=0.2).values()

# Fine-tune
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=2
)

training_args = TrainingArguments(
    output_dir="./sentiment_model",
    evaluation_strategy="epoch",
    learning_rate=2e-5,          # typical range: 1e-5 to 5e-5 for fine-tuning
    per_device_train_batch_size=32,
    num_train_epochs=3,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)
trainer.train()

Named Entity Recognition

NER extracts structured information from unstructured text:

pythonCopy
from transformers import pipeline

ner = pipeline("ner", model="dbmdz/bert-large-cased-finetuned-conll03-english",
               aggregation_strategy="simple")

text = "Apple Inc. announced that CEO Tim Cook will meet with President Biden in Washington D.C."
entities = ner(text)

for entity in entities:
    print(f"{entity['word']} → {entity['entity_group']} (score: {entity['score']:.3f})")

# Output:
# Apple Inc. → ORG (score: 0.998)
# Tim Cook → PER (score: 0.997)
# President Biden → PER (score: 0.991)
# Washington D.C. → LOC (score: 0.994)

Semantic Search with Sentence Embeddings

pythonCopy
from sentence_transformers import SentenceTransformer, util
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')  # fast, good quality

# Encode corpus
corpus = [
    "How do I reset my password?",
    "What are your business hours?",
    "I need to return a product",
    "My order has not arrived",
    "How do I update my billing information?"
]
corpus_embeddings = model.encode(corpus, convert_to_tensor=True)

# Semantic search
query = "I forgot my login credentials"
query_embedding = model.encode(query, convert_to_tensor=True)

# Cosine similarity search
hits = util.semantic_search(query_embedding, corpus_embeddings, top_k=3)[0]
for hit in hits:
    print(f"{corpus[hit['corpus_id']]} (score: {hit['score']:.3f})")

# "How do I reset my password?" (score: 0.721)  -  correct!

Efficient Serving: The 10x Cost Problem

Transformer models are large. A BERT-base has 110M parameters. Serving it naively is expensive.

Quantization - fastest to implement:

pythonCopy
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer
import onnxruntime as ort

# Export to ONNX with quantization
model = ORTModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english",
    export=True,
    provider="CPUExecutionProvider"
)

# Dynamic quantization (post-training, no data needed)
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic("model.onnx", "model_quantized.onnx", weight_type=QuantType.QInt8)
# Typically: 2-4x speedup, <1% accuracy loss

Batching - handle multiple requests together:

pythonCopy
from fastapi import FastAPI
import asyncio
from collections import defaultdict

app = FastAPI()
pending_requests = []
batch_size = 32
max_wait_ms = 50

async def process_batch():
    while True:
        await asyncio.sleep(max_wait_ms / 1000)
        if pending_requests:
            batch = pending_requests[:batch_size]
            del pending_requests[:batch_size]
            texts = [req["text"] for req in batch]
            results = model(texts)  # batch inference
            for req, result in zip(batch, results):
                req["future"].set_result(result)

Key NLP Metrics

Task	Primary metric	Why
Classification	F1 (macro for imbalanced)	Balances precision and recall
NER	Entity-level F1	Partial matches do not count
Summarization	ROUGE-L	Longest common subsequence
Translation	BLEU	N-gram precision with brevity penalty
Semantic similarity	Spearman correlation with human judgments	STS benchmark

For production LLM tasks, complement automatic metrics with LLM-as-judge evaluation - it correlates better with human preference than ROUGE or BLEU.

Common Mistakes

Truncating prompts from the wrong end. When a user input exceeds the model's context limit, most naive implementations truncate the end of the string. For tasks like question answering over a long document, this throws away the question - which is often appended last. Truncation strategy should be task-aware: truncate the document portion, not the instruction or question portion.

Using BERT for generation tasks. BERT is an encoder-only model trained with masked language modeling; it has no causal language modeling head and cannot generate text autoregressively. Attempting to use it as a generator produces nonsense. For generation, use a decoder-only model (GPT-2 and descendants) or an encoder-decoder model (T5, BART, mT5).

Not accounting for multilingual text using more tokens than English. Most tokenizers are trained on English-dominant corpora, which means non-Latin scripts and low-resource languages are tokenized into many more subword tokens for the same semantic content. A sentence in Thai or Arabic may use 3-5x more tokens than its English translation. This affects cost, context window utilization, and truncation behavior - always measure token counts across languages for multilingual applications.

What to Practice Next

• Tokenize the same sentence in English and two other languages using a production tokenizer (tiktoken or the HuggingFace tokenizer for your model); measure the token count ratio and reflect on what this means for your context budget.
• Implement input truncation for a QA system that preserves the question and truncates the document context when the combined input exceeds the token limit; verify the question is always present in the model input.
• Fine-tune a BERT model for text classification (not generation) and a GPT-2 model for text generation; compare the training objectives and confirm you could not swap them.