Claude API Vector Database Retrieval: A Working Example

Retrieval-augmented generation (RAG) with the Claude API means storing your documents as embeddings in a vector database, retrieving the most relevant chunks for a user's question, and passing those chunks to Claude as context before it generates an answer. This pattern lets Claude answer questions about your private data — internal docs, support tickets, product manuals — without fine-tuning and without exceeding context limits.

Below is a complete, working example: embed documents with an embedding model, store vectors in a lightweight vector database, retrieve the top matches for a query, and send them to Claude for a grounded answer. The same architecture works whether you call Claude directly through Anthropic or through a proxy like SubToAPI that gives you a standard HTTPS API key for the same models.

How Claude API retrieval actually works

Claude has no built-in vector search. Retrieval is something you build around the API:

1. Chunk your source documents into passages (typically 200–800 tokens each).
2. Embed each chunk with an embedding model (OpenAI's text-embedding-3-small, Voyage AI, or any open-source model).
3. Store the embeddings plus the original text in a vector database (Pinecone, Chroma, Qdrant, pgvector, Weaviate).
4. Retrieve the top-k chunks most similar to the user's question at query time.
5. Generate — send those chunks as context in the Claude messages request, and let Claude answer using only that grounded information.

Claude does the final reasoning step. Everything else is plumbing you control, which means you can swap embedding models or vector stores without touching your Claude integration.

Step 1: Chunk and embed your documents

import { OpenAI } from "openai";
import { ChromaClient } from "chromadb";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const chroma = new ChromaClient();
const collection = await chroma.getOrCreateCollection({ name: "docs" });

async function indexDocument(id, text) {
  const chunks = chunkText(text, 500); // split into ~500 token pieces

  for (let i = 0; i < chunks.length; i++) {
    const embedding = await openai.embeddings.create({
      model: "text-embedding-3-small",
      input: chunks[i],
    });

    await collection.add({
      ids: [`${id}-${i}`],
      embeddings: [embedding.data[0].embedding],
      documents: [chunks[i]],
    });
  }
}

function chunkText(text, size) {
  const words = text.split(" ");
  const chunks = [];
  for (let i = 0; i < words.length; i += size) {
    chunks.push(words.slice(i, i + size).join(" "));
  }
  return chunks;
}

Run indexDocument() once per document when you ingest your knowledge base. This is a one-time (or periodic) job, not something you run per query.

Step 2: Retrieve relevant chunks at query time

async function retrieve(question, topK = 4) {
  const embedding = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: question,
  });

  const results = await collection.query({
    queryEmbeddings: [embedding.data[0].embedding],
    nResults: topK,
  });

  return results.documents[0]; // array of matching text chunks
}

This returns the handful of passages most semantically similar to the question — not keyword matches, but meaning-based matches.

Step 3: Send retrieved context to Claude

async function answerQuestion(question) {
  const context = await retrieve(question);

  const response = await fetch("https://api.subtoapi.app/v1/messages", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-5",
      max_tokens: 1024,
      messages: [
        {
          role: "user",
          content: `Answer the question using only the context below. If the context doesn't contain the answer, say you don't know.

Context:
${context.join("\n\n---\n\n")}

Question: ${question}`,
        },
      ],
    }),
  });

  const data = await response.json();
  return data.content[0].text;
}

That's the full loop. Everything before this call is retrieval infrastructure; this is the one request where Claude actually reasons over your data.

Why ground the prompt instead of fine-tuning

Fine-tuning bakes knowledge into model weights and goes stale the moment your documents change. Retrieval keeps the model and the knowledge separate: update a document, re-embed it, and the next query automatically uses the new version. There's no retraining step and no deployment delay.

It also lets you cite sources. Since you control which chunks get retrieved, you can return the source document IDs alongside Claude's answer so users can verify where the information came from.

Choosing a vector database

For a small knowledge base (a few thousand documents), an embedded store like Chroma or a Postgres extension like pgvector is enough — no separate infrastructure to run. For larger scale or multi-tenant systems, managed options like Pinecone or Qdrant Cloud handle sharding and filtering for you. The Claude API side of the code above doesn't change based on which vector database you pick — only the retrieve() function does.

Keeping retrieval requests production-ready

A few things matter once this moves past a prototype:

• Chunk size tuning. Too small and you lose context within a passage; too large and you waste tokens on irrelevant text. 300–600 tokens per chunk is a reasonable starting point.
• Re-ranking. For high-stakes use cases, run a second, more precise re-ranking pass on the top 20 candidates before picking the final top-4 to send to Claude.
• Token budgeting. Retrieved context plus the question plus the system prompt all count toward your input tokens. Track usage so retrieval doesn't silently balloon your bill.
• Key management and streaming. If you're running this across a team or multiple apps, routing calls through a single API key with usage visibility (see /docs/messages and /docs/streaming) makes it easier to monitor cost per query and stream long answers back to users as they're generated.

If you're prototyping this pattern, SubToAPI gives you an sub_live_... key that works as a drop-in Claude endpoint with streaming and tool use support, so you can focus on the retrieval logic instead of managing separate Anthropic credentials. Check /pricing for plan details or start with the free trial at /signup.

questions

Do I need a specific vector database to use the Claude API for RAG? No. Claude has no dependency on any vector store — you can use Chroma, Pinecone, Qdrant, pgvector, or Weaviate. The retrieval step is entirely separate from the Claude API call.

How many retrieved chunks should I send to Claude per request? Start with 3–5 chunks of 300–600 tokens each. Test with your real questions and increase only if answers are consistently missing information that's in your knowledge base.

Can I stream the final answer after retrieval? Yes. Once you have your retrieved context, send it in a streaming request exactly as you would any other Claude call — retrieval happens before the API request, not during it. See /docs/streaming for implementation details.