Section 5 · Beyond the Basics
Where LLMs and Embeddings Fit
17 min4 min readChoose wisely
Lesson at a glance
Explain what a large language model and an embedding are, compare texts with cosine similarity, and decide when an LLM, embeddings or a classic scikit-learn model is the right tool.
No video here — just the real thing: a written lesson, runnable examples, a quiz and practice.
Large language models have changed what software can do with text, and every data team now gets asked “can’t we just use an LLM for this?”. Sometimes the answer is yes. Often it’s “for part of it”. To answer well, you need an accurate picture of what LLMs and embeddings are, and that picture is built from ideas you’ve used all course.
What an LLM is
A large language model is a neural network, almost always a transformer, trained on an enormous amount of text with one objective: given the text so far, predict the next token (a word or a piece of a word). That’s pretraining. Done at sufficient scale, next-token prediction forces the network to absorb grammar, facts, styles and a surprising amount of reasoning-like behaviour, because all of it helps predict what comes next.
A pretrained model then gets further training to follow instructions and to give helpful, safe answers, usually with human-written examples and human or AI preference feedback. When you send it a prompt, it computes a probability for every possible next token, picks one (with some randomness, controlled by a temperature setting), appends it, and repeats.
Three consequences matter for a data scientist:
- The output is generated text, not a measured probability. If you ask “how likely is this customer to churn?” and it says “70%”, that number hasn’t been calibrated against outcomes the way
predict_probawas in section 3. - It can be confidently wrong. Fluency isn’t accuracy. Anything you build on an LLM needs a labelled test set and a baseline, exactly like a churn model.
- It costs per call in money and latency, and it may send data outside your systems. A logistic regression scores a million customers in under a second on a laptop.
Embeddings: text as coordinates
The more quietly useful idea is the embedding. An embedding model reads a piece of text and returns a fixed-length vector, often several hundred to a few thousand numbers, arranged so that texts with similar meaning get nearby vectors. Cartwheel’s support tickets become rows of numbers, and everything you know about rows of numbers applies.
Closeness is measured with cosine similarity: the cosine of the angle between two vectors, 1 for the same direction, near 0 for unrelated. Compare that with the word-count features scikit-learn can build on its own:
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
tickets = ["My parcel never arrived", "The package did not show up",
"I was charged twice for one order", "Refund the duplicate payment please",
"How do I change my delivery address"]
tfidf = TfidfVectorizer().fit_transform(tickets)
print("TF-IDF similarity of ticket 0 to the others:", cosine_similarity(tfidf)[0].round(2))
# Illustrative 4-number "embeddings" (real ones come from an embedding model and are far longer)
emb = np.array([[0.90, 0.10, 0.00, 0.20], [0.85, 0.15, 0.05, 0.10],
[0.05, 0.90, 0.10, 0.00], [0.10, 0.85, 0.20, 0.05],
[0.30, 0.00, 0.10, 0.90]])
print("embedding similarity of ticket 0 to the others:", cosine_similarity(emb)[0].round(2))TF-IDF counts shared words. “Parcel never arrived” and “package did not show up” share none, so it scores them 0, yet it links ticket 0 to the address question because both contain “my”. Embeddings score the two delivery complaints as near-identical. The vectors here are hand-made to show the idea; a real embedding model produces this behaviour from its training, for texts it has never seen.
Where each tool fits at Cartwheel
| Task | Good fit | Why |
|---|---|---|
| Predict churn from customer facts | scikit-learn on the table | Labelled tabular data, needs calibrated, cheap, testable scores |
| Route support tickets to teams | Embeddings + logistic regression, or an LLM | Text input; embeddings make it a classic classifier |
| Find tickets similar to a new one | Embeddings + nearest neighbours | Similarity search is what embeddings are for |
| Summarise a week of complaints for a manager | An LLM | Free-form text output; a person reviews it |
| Add “complaint topic” as a churn feature | Embeddings → k-means, or an LLM label | Turns text into a column the churn model can test |
The last row is where the two worlds meet. Text becomes a feature; cross-validation decides whether it helps. Nothing about the discipline changes because the feature came from a large model.
Whichever model you ship, it will make decisions about people. Next lesson asks whether those decisions are equally good for everyone, and what to watch after launch.
Key takeaways
- A large language model is a transformer network pretrained to predict the next token on huge amounts of text, then tuned to follow instructions.
- An LLM’s output is sampled text, not a calibrated probability, so it needs the same evaluation discipline as any model: a labelled test set and a baseline.
- An embedding model maps a text to a fixed-length vector so that texts with similar meaning land close together; cosine similarity measures how close.
- Embeddings turn unstructured text into features that classic tools (k-NN, logistic regression, k-means) can use.
- For tabular predictions such as churn, scikit-learn models stay cheaper, faster and easier to validate; LLMs and embeddings earn their place on text.