Working with LLMs

For many text tasks, such as summarising, classifying support tickets or extracting fields from documents, you no longer train your own model. You call one from OpenAI, Anthropic, Google or an open-weight model like Llama, Qwen or Mistral, and write a clear prompt.

Your job shifts. Instead of feature engineering, you design prompts, choose the right model size, structure the output (for example as JSON) and handle failures. The engineering habits stay the same: version, test, monitor.

A model only knows what was in its training data. To answer from your company's documents, you use retrieval-augmented generation, or RAG. You split documents into chunks, turn each chunk into an embedding (a list of numbers that captures meaning), and store them in a vector database.

When a question comes in, you find the most similar chunks and paste them into the prompt. The model answers from those sources and can cite them. Most RAG problems are retrieval problems: the right chunk was never found.

You cannot check LLM output with a simple equals sign. So you build an eval set: 50 to 200 real questions with known good answers or clear criteria. Every time you change a prompt or model, you run the set and compare.

Some checks are code, such as 'is it valid JSON?' or 'did it cite a source?'. Others use a second model as a judge, scored against a written rubric. Check the judge against your own ratings on a sample before you trust it.

You pay per token, both in and out. Long prompts and big models add up fast at scale. Try the smallest model that passes your evals, cache repeated prompts, and cap output length. Measure latency too: users notice a slow answer.

Treat model output as untrusted. It can be wrong or be steered by text hidden in documents, called prompt injection. Never let it run actions or reach private data without checks.