Learn how to implement semantic search using OpenAI embeddings, enabling high-quality content matching based on meaning instead of exact keyword matching.
Prerequisites
To build semantic search with OpenAI embeddings, ensure the following:
prerequisites
- A basic understanding of embeddings and cosine similarity.
- A working development environment with Node.js or a similar runtime installed.
- Access to the OpenAI Text Embedding model via API.
Initializing the Semantic Search API
Start by setting up the project structure to handle semantic search requests:
steps
- Create a new folder named
semantic-searchand add aroute.tsfile inside it. - Import necessary functions like
embed,embedMany, andcosineSimilarityfor embedding and similarity computation. - Create a hardcoded list of movie descriptions for initial testing.
- Configure the API route to accept and process movie search requests.
import { embed, embedMany, cosineSimilarity } from './embedding-utils';
// Mock movie dataset
const movies = [
{ id: 1, title: "The Matrix", description: "A sci-fi movie about simulation." },
{ id: 2, title: "Inception", description: "A thriller about dream invasion." },
];
export async function post(request: Request) {
const { query } = await request.json();
// Additional processing follows in the next steps...
}Embedding Movie Descriptions
To enable semantic search, pre-compute embeddings for the movie descriptions:
steps
- Use the
embedManyfunction to generate embeddings for your movie dataset. - Call the OpenAI Text Embedding model to convert descriptions into vector embeddings.
- Store these embeddings for reference during search queries.
const generateEmbeddings = async () => {
const descriptions = movies.map(movie => movie.description);
const embeddings = await embedMany({
model: 'text-embedding-ada-002',
input: descriptions
});
return embeddings;
};Processing Search Queries
When a user submits a search query, compute its embedding and compare it against the movie embeddings:
steps
- Extract the query string from the HTTP request.
- Use the
embedfunction to compute the query's embedding vector. - Calculate the cosine similarity between the query embedding and all movie embeddings.
const processQuery = async (query: string, movieEmbeddings: number[][]) => {
const queryEmbedding = await embed({ model: 'text-embedding-ada-002', input: query });
const similarities = movieEmbeddings.map((embedding, index) => ({
...movies[index],
similarity: cosineSimilarity(queryEmbedding, embedding)
}));
return similarities;
};Refining and Sorting Results
To give the user the best possible search experience, refine and present the top results:
steps
- Sort the results by similarity scores in descending order.
- Return only the top relevant results or apply a similarity threshold to filter less relevant matches.
- Format the results for API responses.
const filterAndSortResults = (movieScores: { similarity: number }[], threshold = 0.4) => {
return movieScores
.filter(movie => movie.similarity >= threshold)
.sort((a, b) => b.similarity - a.similarity)
.slice(0, 3);
};Deploying and Testing the API
Now put your implementation to the test:
steps
- Deploy the API locally or to a server.
- Use tools like Thunderclient or Postman to send requests with a query string.
- Validate the results for accuracy and semantic relevance.
curl -X POST http://localhost:3000/api/semantic-search \
-H "Content-Type: application/json" \
-d '{"query": "A sci-fi movie about space"}'
# Expected output: JSON results with relevant movies, sorted by similarity.Scaling with Vector Databases
Embedding computations are computationally intensive at scale. To handle larger datasets effectively:
- Store embeddings in a vector database: Save precomputed embeddings to a database such as Pinecone, Weaviate, or FAISS.
- Use indexing for efficiency: Vector databases index embeddings to allow fast similarity searches across millions of items.
- Optimize for real-time applications: Avoid computing embeddings at runtime for datasets; focus on embedding queries alone.
Takeaway
Building semantic search with OpenAI embeddings allows you to match content based on meaning rather than keywords. While the basic implementation handles smaller datasets, scaling to larger databases benefits from vectorized storage and indexing. This approach enriches user experiences by providing relevant and meaningful search results.
FAQ
What is semantic search?
Semantic search is a method of finding content related by meaning rather than exact keyword matches. By using OpenAI embeddings, you can implement similarity-based matching to provide more relevant results.
Why use cosine similarity for semantic search?
Cosine similarity measures the angle between two vectors in high-dimensional space, making it ideal for assessing how similar two embeddings are without being affected by their magnitude.
How do vector databases optimize semantic search?
Vector databases store embeddings in a way that supports efficient similarity lookups. They rely on indexing methods to quickly identify near matches without comparing every embedding.
Can I store embeddings in a traditional database?
While traditional databases can store embeddings as JSON or binary data, they lack the indexing capabilities of vector databases, making large-scale searches slower and less efficient.
Official reference: OpenAI API documentation.