Embedding

What an Embedding Is

An embedding represents data that computers struggle to handle directly—words, sentences, documents, images—as a high-dimensional vector of real numbers that preserves meaning. OpenAI's embeddings documentation describes an embedding simply as a "vector (list) of floating point numbers" and explains that this vector measures the semantic relatedness between pieces of text. In essence, an embedding can be understood as a way of "turning meaning into coordinates."

Why is such a conversion needed? As Google's Machine Learning Crash Course explains, representing a word with plain one-hot encoding produces a vector as long as the number of items (for example, a menu of 5,000 dishes yields a length-5,000 vector) and, more importantly, captures none of the semantic similarity—it cannot express that "a hot dog is more like shawarma than it is like a salad." An embedding compresses this into a low-dimensional dense vector that carries those semantic relationships, sharply reducing the number of weights a neural network must learn along with the memory and compute burden.

How It Works and Where It Is Used

The governing rule of an embedding space is straightforward: items that are similar in meaning are placed close together as vectors. OpenAI's documentation states that "small distances suggest high relatedness and large distances suggest low relatedness," and it recommends cosine similarity for measuring similarity. Because OpenAI embeddings are normalized to length 1, cosine similarity is efficient to compute and produces the same ranking as Euclidean distance.

Thanks to this property, embeddings serve as the foundation for a wide range of tasks. OpenAI lists their primary uses as search (ranking results by relevance to a query), clustering (grouping similar texts), recommendation, anomaly detection, diversity measurement, and classification. Search in particular enables semantic search, where a match can occur even when keywords do not align exactly, as long as the meanings are close. To borrow Pinecone's phrasing, it is a matter of "matching concepts rather than keywords."

The Retrieval Step of RAG and GEO

Embeddings matter especially in generative AI search because of RAG (retrieval-augmented generation). According to Pinecone's explanation of RAG, you store a knowledge base as embeddings and, at query time, retrieve the chunks closest to the question's embedding and feed them to the LLM as context. The model then answers grounded in your own data, which reduces hallucination and keeps responses current without retraining. The index compares the query vector against the stored chunk vectors and returns semantic proximity as a score.

This structure ties directly to GEO (generative engine optimization). For content to be cited in answers from tools like ChatGPT and Perplexity, it must be represented by a vector close enough to the user's question to be selected as a candidate during retrieval. As a result, content with a cohesive topic and clear core concepts is more likely to sit near the question in the embedding space than writing whose meaning is ambiguous.

Evidence and Examples

The research that most strikingly demonstrated that embeddings preserve meaning is Google's word2vec. The 2013 paper by Mikolov et al., "Efficient Estimation of Word Representations in Vector Space" (arXiv:1301.3781), showed that placing words in a continuous vector space (roughly 20 to 300 dimensions) preserves linear regularities of meaning and grammar, so that vector arithmetic analogies such as king − man + woman ≈ queen hold. The model was efficient enough to learn high-quality word vectors from a 1.6-billion-word dataset within a day.

Modern embedding models are larger and more flexible. Per OpenAI's documentation, text-embedding-3-small defaults to 1,536 dimensions and text-embedding-3-large to 3,072 dimensions, and the dimensions parameter lets you shrink the dimensionality without losing much representational power (for example, 3,072 → 1,024). Google's Gemini Embedding likewise uses 3,072-dimensional vectors. Below is pseudocode illustrating an embedding API call and a similarity comparison.

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(text):
    # text-embedding-3-small -> returns a 1536-dim vector by default
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return np.array(resp.data[0].embedding)

q = embed("What is an embedding")
d = embed("A representation technique that turns words into meaning vectors")

# Vectors are normalized, so the dot product equals cosine similarity
similarity = float(np.dot(q, d))
print(similarity)  # the closer the value is to 1, the closer the meaning

References

• Embeddings — Google Machine Learning Crash Course
• Vector embeddings — OpenAI API Documentation
• Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space (arXiv:1301.3781)
• Retrieval-Augmented Generation (RAG) — Pinecone