- Embeddings — convert documents and queries into vectors for semantic search
- Rerank — re-score retrieved candidates for higher precision
- Responses — generate a final answer using the retrieved context
How RAG Works
A typical RAG pipeline follows these steps:- Index — chunk your documents and generate embeddings for each chunk
- Retrieve — embed the user’s query and find the most similar document chunks
- Rerank (optional) — re-score the top candidates with a cross-encoder rerank model for better precision
- Generate — pass the top documents as context to an LLM to produce a grounded answer
User Query
│
▼
┌──────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ Embed │────▶│ Vector Search│────▶│ Rerank │────▶│ LLM Generate │
│ Query │ │ (Top N docs) │ │ (Top K) │ │ with Context │
└──────────┘ └──────────────┘ └──────────┘ └──────────────┘
Step 1: Index Your Documents
Split your documents into chunks and generate embeddings for each chunk. Store the embeddings in a vector database (or in-memory for prototyping).import requests
import json
OPENROUTER_API_KEY = "<OPENROUTER_API_KEY>"
# Your documents, split into chunks
chunks = [
"OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers.",
"RAG stands for Retrieval-Augmented Generation. It grounds LLM answers in external data.",
"Embeddings convert text into numerical vectors that capture semantic meaning.",
"Reranking uses a cross-encoder to re-score documents for a given query, improving precision.",
"Vector databases like Pinecone, Weaviate, and Qdrant store embeddings for fast similarity search.",
"Prompt caching can reduce costs by reusing previous computations for repeated prefixes.",
"OpenRouter supports provider routing to control which providers serve your requests.",
]
# Generate embeddings for all chunks in one batch request
response = requests.post(
"https://orr.api.report.finance/api/v1/embeddings",
headers={
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json",
},
json={
"model": "openai/text-embedding-3-small",
"input": chunks,
},
)
data = response.json()
# Each item in data["data"] contains an "embedding" vector
# Store these alongside your chunks in a vector database
document_embeddings = [
{"text": chunks[item["index"]], "embedding": item["embedding"]}
for item in data["data"]
]
print(f"Indexed {len(document_embeddings)} chunks with {len(document_embeddings[0]['embedding'])}-dim embeddings")
const OPENROUTER_API_KEY = '<OPENROUTER_API_KEY>';
// Your documents, split into chunks
const chunks = [
'OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers.',
'RAG stands for Retrieval-Augmented Generation. It grounds LLM answers in external data.',
'Embeddings convert text into numerical vectors that capture semantic meaning.',
'Reranking uses a cross-encoder to re-score documents for a given query, improving precision.',
'Vector databases like Pinecone, Weaviate, and Qdrant store embeddings for fast similarity search.',
'Prompt caching can reduce costs by reusing previous computations for repeated prefixes.',
'OpenRouter supports provider routing to control which providers serve your requests.',
];
// Generate embeddings for all chunks in one batch request
const response = await fetch('https://orr.api.report.finance/api/v1/embeddings', {
method: 'POST',
headers: {
'Authorization': `Bearer ${OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'openai/text-embedding-3-small',
input: chunks,
}),
});
const data = await response.json();
// Each item in data.data contains an "embedding" vector
// Store these alongside your chunks in a vector database
const documentEmbeddings = data.data.map((item: { index: number; embedding: number[] }) => ({
text: chunks[item.index],
embedding: item.embedding,
}));
console.log(`Indexed ${documentEmbeddings.length} chunks with ${documentEmbeddings[0].embedding.length}-dim embeddings`);
curl https://orr.api.report.finance/api/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d '{
"model": "openai/text-embedding-3-small",
"input": [
"OpenRouter is a unified API gateway for LLMs.",
"RAG stands for Retrieval-Augmented Generation.",
"Embeddings convert text into numerical vectors."
]
}'
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
)
type DocumentEmbedding struct {
Text string
Embedding []float64
}
func main() {
// Your documents, split into chunks
chunks := []string{
"OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers.",
"RAG stands for Retrieval-Augmented Generation. It grounds LLM answers in external data.",
"Embeddings convert text into numerical vectors that capture semantic meaning.",
"Reranking uses a cross-encoder to re-score documents for a given query, improving precision.",
"Vector databases like Pinecone, Weaviate, and Qdrant store embeddings for fast similarity search.",
"Prompt caching can reduce costs by reusing previous computations for repeated prefixes.",
"OpenRouter supports provider routing to control which providers serve your requests.",
}
// Generate embeddings for all chunks in one batch request
body, _ := json.Marshal(map[string]any{
"model": "openai/text-embedding-3-small",
"input": chunks,
})
req, _ := http.NewRequest("POST", "https://orr.api.report.finance/api/v1/embeddings", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("OPENROUTER_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
log.Fatalf("HTTP %d: %s", resp.StatusCode, b)
}
var data struct {
Data []struct {
Index int `json:"index"`
Embedding []float64 `json:"embedding"`
} `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
log.Fatal(err)
}
// Each item in data.Data contains an "embedding" vector
// Store these alongside your chunks in a vector database
documentEmbeddings := make([]DocumentEmbedding, 0, len(data.Data))
for _, item := range data.Data {
documentEmbeddings = append(documentEmbeddings, DocumentEmbedding{
Text: chunks[item.Index],
Embedding: item.Embedding,
})
}
fmt.Printf("Indexed %d chunks with %d-dim embeddings\n", len(documentEmbeddings), len(documentEmbeddings[0].Embedding))
}
Output
Indexed 7 chunks with 1536-dim embeddings
In production, use a vector database (Pinecone, Weaviate, Qdrant, pgvector, etc.) to store and query embeddings efficiently. The in-memory approach shown here is for illustration only.
Step 2: Retrieve Relevant Documents
When a user asks a question, embed the query and find the most similar document chunks using cosine similarity.import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def retrieve(query, document_embeddings, top_n=5):
# Embed the query
response = requests.post(
"https://orr.api.report.finance/api/v1/embeddings",
headers={
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json",
},
json={
"model": "openai/text-embedding-3-small",
"input": query,
},
)
query_embedding = np.array(response.json()["data"][0]["embedding"])
# Score each document by cosine similarity
scored = []
for doc in document_embeddings:
score = cosine_similarity(query_embedding, np.array(doc["embedding"]))
scored.append({"text": doc["text"], "score": float(score)})
# Return the top N most similar chunks
scored.sort(key=lambda x: x["score"], reverse=True)
return scored[:top_n]
query = "What is RAG and how does it work?"
results = retrieve(query, document_embeddings, top_n=5)
print("Retrieved documents:")
for i, r in enumerate(results):
print(f" {i+1}. (score: {r['score']:.4f}) {r['text']}")
function cosineSimilarity(a: number[], b: number[]): number {
const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
return dot / (magA * magB);
}
async function retrieve(
query: string,
documentEmbeddings: { text: string; embedding: number[] }[],
topN = 5,
) {
// Embed the query
const response = await fetch('https://orr.api.report.finance/api/v1/embeddings', {
method: 'POST',
headers: {
'Authorization': `Bearer ${OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'openai/text-embedding-3-small',
input: query,
}),
});
const data = await response.json();
const queryEmbedding: number[] = data.data[0].embedding;
// Score each document by cosine similarity
const scored = documentEmbeddings.map((doc) => ({
text: doc.text,
score: cosineSimilarity(queryEmbedding, doc.embedding),
}));
// Return the top N most similar chunks
scored.sort((a, b) => b.score - a.score);
return scored.slice(0, topN);
}
const query = 'What is RAG and how does it work?';
const results = await retrieve(query, documentEmbeddings, 5);
console.log('Retrieved documents:');
results.forEach((r, i) => {
console.log(` ${i + 1}. (score: ${r.score.toFixed(4)}) ${r.text}`);
});
type ScoredDocument struct {
Text string
Score float64
}
func cosineSimilarity(a, b []float64) float64 {
var dot, magA, magB float64
for i := range a {
dot += a[i] * b[i]
magA += a[i] * a[i]
magB += b[i] * b[i]
}
return dot / (math.Sqrt(magA) * math.Sqrt(magB))
}
func retrieve(query string, documentEmbeddings []DocumentEmbedding, topN int) []ScoredDocument {
// Embed the query
body, _ := json.Marshal(map[string]any{
"model": "openai/text-embedding-3-small",
"input": query,
})
req, _ := http.NewRequest("POST", "https://orr.api.report.finance/api/v1/embeddings", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("OPENROUTER_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
log.Fatalf("HTTP %d: %s", resp.StatusCode, b)
}
var data struct {
Data []struct {
Embedding []float64 `json:"embedding"`
} `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
log.Fatal(err)
}
queryEmbedding := data.Data[0].Embedding
// Score each document by cosine similarity
scored := make([]ScoredDocument, 0, len(documentEmbeddings))
for _, doc := range documentEmbeddings {
scored = append(scored, ScoredDocument{
Text: doc.Text,
Score: cosineSimilarity(queryEmbedding, doc.Embedding),
})
}
// Return the top N most similar chunks
sort.Slice(scored, func(i, j int) bool { return scored[i].Score > scored[j].Score })
if len(scored) > topN {
scored = scored[:topN]
}
return scored
}
// Call retrieveAndPrint(documentEmbeddings) as the last line of main in step 1, and add "math", "sort", and "strings" to that file's imports. Each step passes its results to the next.
func retrieveAndPrint(documentEmbeddings []DocumentEmbedding) {
query := "What is RAG and how does it work?"
results := retrieve(query, documentEmbeddings, 5)
fmt.Println("Retrieved documents:")
for i, r := range results {
fmt.Printf(" %d. (score: %.4f) %s\n", i+1, r.Score, r.Text)
}
rerankAndPrint(query, results) // defined in step 3
}
QUERY="What is RAG and how does it work?"
DOCS='["OpenRouter is a unified API gateway for LLMs.", "RAG stands for Retrieval-Augmented Generation.", "Embeddings convert text into numerical vectors."]'
# Embed the documents and save each text next to its vector
curl -s https://orr.api.report.finance/api/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n --argjson docs "$DOCS" '{model: "openai/text-embedding-3-small", input: $docs}')" \
| jq --argjson docs "$DOCS" '[.data[] | {text: $docs[.index], embedding}]' \
> document_embeddings.json
# Embed the query
curl -s https://orr.api.report.finance/api/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n --arg q "$QUERY" '{model: "openai/text-embedding-3-small", input: $q}')" \
> query_embedding.json
# Score each document by cosine similarity and print the top 5
jq -r --slurpfile q query_embedding.json '
def dot(a; b): [a, b] | transpose | map(.[0] * .[1]) | add;
def norm(v): v | map(. * .) | add | sqrt;
$q[0].data[0].embedding as $qe
| map({text, score: (dot($qe; .embedding) / (norm($qe) * norm(.embedding)))})
| sort_by(-.score) | .[:5]
| "Retrieved documents:",
(to_entries[] | " \(.key + 1). (score: \(.value.score * 10000 | round / 10000)) \(.value.text)")
' document_embeddings.json
Step 3: Rerank for Better Precision
Embedding-based retrieval is fast but approximate. A rerank model uses a cross-encoder to compare each document directly against the query, producing more accurate relevance scores. This is especially valuable when you retrieve many candidates (e.g., 20) and want to narrow down to the best few (e.g., 3).def rerank(query, documents, top_n=3):
response = requests.post(
"https://orr.api.report.finance/api/v1/rerank",
headers={
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json",
},
json={
"model": "cohere/rerank-v3.5",
"query": query,
"documents": documents,
"top_n": top_n,
},
)
data = response.json()
return data["results"]
# Use the texts from the retrieval step
retrieved_texts = [r["text"] for r in results]
reranked = rerank(query, retrieved_texts, top_n=3)
print("Reranked documents:")
for r in reranked:
print(f" Score: {r['relevance_score']:.4f} | {r['document']['text']}")
async function rerank(query: string, documents: string[], topN = 3) {
const response = await fetch('https://orr.api.report.finance/api/v1/rerank', {
method: 'POST',
headers: {
'Authorization': `Bearer ${OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'cohere/rerank-v3.5',
query,
documents,
top_n: topN,
}),
});
const data = await response.json();
return data.results;
}
// Use the texts from the retrieval step
const retrievedTexts = results.map((r) => r.text);
const reranked = await rerank(query, retrievedTexts, 3);
console.log('Reranked documents:');
for (const r of reranked) {
console.log(` Score: ${r.relevance_score.toFixed(4)} | ${r.document.text}`);
}
# Independent of the retrieval step: a fixed query and fixed documents. To rerank your own matches, send your query and the retrieved texts as "query" and "documents".
RERANKED=$(curl -sS --fail-with-body https://orr.api.report.finance/api/v1/rerank \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d '{
"model": "cohere/rerank-v3.5",
"query": "What is RAG and how does it work?",
"documents": [
"RAG stands for Retrieval-Augmented Generation. It grounds LLM answers in external data.",
"Embeddings convert text into numerical vectors that capture semantic meaning.",
"Reranking uses a cross-encoder to re-score documents for a given query, improving precision.",
"OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers.",
"Vector databases like Pinecone, Weaviate, and Qdrant store embeddings for fast similarity search."
],
"top_n": 3
}')
echo "$RERANKED" | jq .
type RerankResult struct {
Index int `json:"index"`
RelevanceScore float64 `json:"relevance_score"`
Document struct {
Text string `json:"text"`
} `json:"document"`
}
func rerank(query string, documents []string, topN int) []RerankResult {
body, _ := json.Marshal(map[string]any{
"model": "cohere/rerank-v3.5",
"query": query,
"documents": documents,
"top_n": topN,
})
req, _ := http.NewRequest("POST", "https://orr.api.report.finance/api/v1/rerank", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("OPENROUTER_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
log.Fatalf("HTTP %d: %s", resp.StatusCode, b)
}
var data struct {
Results []RerankResult `json:"results"`
}
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
log.Fatal(err)
}
return data.Results
}
func rerankAndPrint(query string, results []ScoredDocument) {
// Use the texts from the retrieval step
retrievedTexts := make([]string, len(results))
for i, r := range results {
retrievedTexts[i] = r.Text
}
reranked := rerank(query, retrievedTexts, 3)
fmt.Println("Reranked documents:")
for _, r := range reranked {
fmt.Printf(" Score: %.4f | %s\n", r.RelevanceScore, r.Document.Text)
}
answerAndPrint(query, reranked) // defined in step 4
}
Step 4: Generate an Answer with Context
Pass the top-ranked documents as context to a chat model. The LLM generates a grounded answer based on the retrieved information.def generate_answer(query, context_docs):
# Build a context string from the reranked documents
context = "\n\n".join(
f"[{i+1}] {doc['document']['text']}"
for i, doc in enumerate(context_docs)
)
response = requests.post(
"https://orr.api.report.finance/api/v1/responses",
headers={
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json",
},
json={
"model": "~openai/gpt-mini-latest",
"instructions": "Answer the user's question based on the provided context. "
"Cite the relevant source numbers in brackets. "
"If the context doesn't contain enough information, say so.",
"input": f"Context:\n{context}\n\nQuestion: {query}",
},
)
data = response.json()
message = next(item for item in reversed(data["output"]) if item["type"] == "message")
return next(part["text"] for part in message["content"] if part["type"] == "output_text")
answer = generate_answer(query, reranked)
print(f"Question: {query}")
print(f"Answer: {answer}")
async function generateAnswer(
query: string,
contextDocs: { document: { text: string } }[],
) {
// Build a context string from the reranked documents
const context = contextDocs
.map((doc, i) => `[${i + 1}] ${doc.document.text}`)
.join('\n\n');
const response = await fetch('https://orr.api.report.finance/api/v1/responses', {
method: 'POST',
headers: {
'Authorization': `Bearer ${OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: '~openai/gpt-mini-latest',
instructions:
"Answer the user's question based on the provided context. " +
'Cite the relevant source numbers in brackets. ' +
"If the context doesn't contain enough information, say so.",
input: `Context:\n${context}\n\nQuestion: ${query}`,
}),
});
const data = await response.json();
return [...data.output]
.reverse()
.find((item: { type: string }) => item.type === 'message')
.content.find((part: { type: string }) => part.type === 'output_text').text;
}
const answer = await generateAnswer(query, reranked);
console.log(`Question: ${query}`);
console.log(`Answer: ${answer}`);
func generateAnswer(query string, contextDocs []RerankResult) string {
// Build a context string from the reranked documents
parts := make([]string, len(contextDocs))
for i, doc := range contextDocs {
parts[i] = fmt.Sprintf("[%d] %s", i+1, doc.Document.Text)
}
context := strings.Join(parts, "\n\n")
body, _ := json.Marshal(map[string]any{
"model": "~openai/gpt-mini-latest",
"instructions": "Answer the user's question based on the provided context. " +
"Cite the relevant source numbers in brackets. " +
"If the context doesn't contain enough information, say so.",
"input": fmt.Sprintf("Context:\n%s\n\nQuestion: %s", context, query),
})
req, _ := http.NewRequest("POST", "https://orr.api.report.finance/api/v1/responses", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("OPENROUTER_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
log.Fatalf("HTTP %d: %s", resp.StatusCode, b)
}
var data struct {
Output []struct {
Type string `json:"type"`
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
} `json:"output"`
}
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
log.Fatal(err)
}
for i := len(data.Output) - 1; i >= 0; i-- {
if data.Output[i].Type != "message" {
continue
}
for _, part := range data.Output[i].Content {
if part.Type == "output_text" {
return part.Text
}
}
}
return ""
}
func answerAndPrint(query string, reranked []RerankResult) {
answer := generateAnswer(query, reranked)
fmt.Printf("Question: %s\n", query)
fmt.Printf("Answer: %s\n", answer)
}
QUERY="What is RAG and how does it work?"
# RERANKED holds the rerank response captured in the previous step
CONTEXT=$(echo "$RERANKED" | jq -er '.results | select(length > 0) | [.[].document.text] | to_entries | map("[\(.key + 1)] \(.value)") | join("\n\n")') || {
echo "No reranked documents to build context from" >&2
exit 1
}
ANSWER=$(curl -s https://orr.api.report.finance/api/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n --arg context "$CONTEXT" --arg query "$QUERY" '{
model: "~openai/gpt-mini-latest",
instructions: "Answer the user'\''s question based on the provided context. Cite the relevant source numbers in brackets. If the context doesn'\''t contain enough information, say so.",
input: "Context:\n\($context)\n\nQuestion: \($query)"
}')" | jq -r '[.output[] | select(.type == "message")] | last | .content[] | select(.type == "output_text") | .text')
echo "Question: $QUERY"
echo "Answer: $ANSWER"
Complete Example
Here is a full end-to-end RAG pipeline combining all four steps:import requests
import numpy as np
OPENROUTER_API_KEY = "<OPENROUTER_API_KEY>"
EMBEDDING_MODEL = "openai/text-embedding-3-small"
RERANK_MODEL = "cohere/rerank-v3.5"
CHAT_MODEL = "~openai/gpt-mini-latest"
BASE_URL = "https://orr.api.report.finance/api/v1"
def get_headers():
return {
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json",
}
def embed(texts):
"""Generate embeddings for a list of texts."""
response = requests.post(
f"{BASE_URL}/embeddings",
headers=get_headers(),
json={"model": EMBEDDING_MODEL, "input": texts},
)
return [item["embedding"] for item in response.json()["data"]]
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def retrieve(query_embedding, doc_embeddings, top_n=10):
"""Find the top N most similar documents by cosine similarity."""
scored = [
(i, cosine_similarity(query_embedding, emb))
for i, emb in enumerate(doc_embeddings)
]
scored.sort(key=lambda x: x[1], reverse=True)
return scored[:top_n]
def rerank(query, documents, top_n=3):
"""Rerank documents using a cross-encoder model."""
response = requests.post(
f"{BASE_URL}/rerank",
headers=get_headers(),
json={
"model": RERANK_MODEL,
"query": query,
"documents": documents,
"top_n": top_n,
},
)
return response.json()["results"]
def generate(query, context_docs):
"""Generate an answer grounded in the provided context."""
context = "\n\n".join(
f"[{i+1}] {doc}"
for i, doc in enumerate(context_docs)
)
response = requests.post(
f"{BASE_URL}/responses",
headers=get_headers(),
json={
"model": CHAT_MODEL,
"instructions": (
"Answer the user's question using only the provided context. "
"Cite sources with [n]. If the context is insufficient, say so."
),
"input": f"Context:\n{context}\n\nQuestion: {query}",
},
)
data = response.json()
message = next(item for item in reversed(data["output"]) if item["type"] == "message")
return next(part["text"] for part in message["content"] if part["type"] == "output_text")
# --- Pipeline ---
# 1. Index: chunk your knowledge base and embed
chunks = [
"OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers into a single OpenAI-compatible interface.",
"RAG stands for Retrieval-Augmented Generation. It retrieves relevant documents and uses them as context for LLM generation.",
"Embeddings convert text into high-dimensional vectors where semantically similar texts are closer together.",
"Reranking uses a cross-encoder to compare each document directly against the query, producing more accurate relevance scores than embedding similarity alone.",
"Vector databases like Pinecone, Weaviate, and Qdrant are optimized for storing and querying embedding vectors at scale.",
"Prompt caching reduces costs by reusing computations for repeated prompt prefixes across requests.",
"OpenRouter supports provider routing to control which upstream providers serve your requests based on cost, latency, or privacy preferences.",
]
doc_embeddings = embed(chunks)
# 2. Retrieve: embed the query and find similar chunks
query = "How does RAG improve LLM responses?"
query_embedding = embed([query])[0]
top_matches = retrieve(query_embedding, doc_embeddings, top_n=5)
retrieved_texts = [chunks[i] for i, _ in top_matches]
# 3. Rerank: re-score with a cross-encoder for better precision
reranked = rerank(query, retrieved_texts, top_n=3)
context_texts = [r["document"]["text"] for r in reranked]
# 4. Generate: produce a grounded answer
answer = generate(query, context_texts)
print(f"Q: {query}\nA: {answer}")
const OPENROUTER_API_KEY = '<OPENROUTER_API_KEY>';
const EMBEDDING_MODEL = 'openai/text-embedding-3-small';
const RERANK_MODEL = 'cohere/rerank-v3.5';
const CHAT_MODEL = '~openai/gpt-mini-latest';
const BASE_URL = 'https://orr.api.report.finance/api/v1';
function getHeaders() {
return {
'Authorization': `Bearer ${OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
};
}
async function embed(texts: string[]): Promise<number[][]> {
const response = await fetch(`${BASE_URL}/embeddings`, {
method: 'POST',
headers: getHeaders(),
body: JSON.stringify({ model: EMBEDDING_MODEL, input: texts }),
});
const data = await response.json();
return data.data.map((item: { embedding: number[] }) => item.embedding);
}
function cosineSimilarity(a: number[], b: number[]): number {
const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
return dot / (magA * magB);
}
function retrieve(
queryEmbedding: number[],
docEmbeddings: number[][],
topN = 10,
): { index: number; score: number }[] {
const scored = docEmbeddings.map((emb, i) => ({
index: i,
score: cosineSimilarity(queryEmbedding, emb),
}));
scored.sort((a, b) => b.score - a.score);
return scored.slice(0, topN);
}
async function rerank(query: string, documents: string[], topN = 3) {
const response = await fetch(`${BASE_URL}/rerank`, {
method: 'POST',
headers: getHeaders(),
body: JSON.stringify({
model: RERANK_MODEL,
query,
documents,
top_n: topN,
}),
});
const data = await response.json();
return data.results as { index: number; relevance_score: number; document: { text: string } }[];
}
async function generate(query: string, contextDocs: string[]): Promise<string> {
const context = contextDocs
.map((doc, i) => `[${i + 1}] ${doc}`)
.join('\n\n');
const response = await fetch(`${BASE_URL}/responses`, {
method: 'POST',
headers: getHeaders(),
body: JSON.stringify({
model: CHAT_MODEL,
instructions:
"Answer the user's question using only the provided context. " +
'Cite sources with [n]. If the context is insufficient, say so.',
input: `Context:\n${context}\n\nQuestion: ${query}`,
}),
});
const data = await response.json();
return [...data.output]
.reverse()
.find((item: { type: string }) => item.type === 'message')
.content.find((part: { type: string }) => part.type === 'output_text').text;
}
// --- Pipeline ---
// 1. Index: chunk your knowledge base and embed
const chunks = [
'OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers into a single OpenAI-compatible interface.',
'RAG stands for Retrieval-Augmented Generation. It retrieves relevant documents and uses them as context for LLM generation.',
'Embeddings convert text into high-dimensional vectors where semantically similar texts are closer together.',
'Reranking uses a cross-encoder to compare each document directly against the query, producing more accurate relevance scores than embedding similarity alone.',
'Vector databases like Pinecone, Weaviate, and Qdrant are optimized for storing and querying embedding vectors at scale.',
'Prompt caching reduces costs by reusing computations for repeated prompt prefixes across requests.',
'OpenRouter supports provider routing to control which upstream providers serve your requests based on cost, latency, or privacy preferences.',
];
const docEmbeddings = await embed(chunks);
// 2. Retrieve: embed the query and find similar chunks
const query = 'How does RAG improve LLM responses?';
const queryEmbedding = (await embed([query]))[0];
const topMatches = retrieve(queryEmbedding, docEmbeddings, 5);
const retrievedTexts = topMatches.map((m) => chunks[m.index]);
// 3. Rerank: re-score with a cross-encoder for better precision
const reranked = await rerank(query, retrievedTexts, 3);
const contextTexts = reranked.map((r) => r.document.text);
// 4. Generate: produce a grounded answer
const answer = await generate(query, contextTexts);
console.log(`Q: ${query}\nA: ${answer}`);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
"math"
"net/http"
"os"
"sort"
"strings"
)
const (
embeddingModel = "openai/text-embedding-3-small"
rerankModel = "cohere/rerank-v3.5"
chatModel = "~openai/gpt-mini-latest"
baseURL = "https://orr.api.report.finance/api/v1"
)
func post(path string, payload any, out any) {
body, _ := json.Marshal(payload)
req, _ := http.NewRequest("POST", baseURL+path, bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("OPENROUTER_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
log.Fatalf("HTTP %d: %s", resp.StatusCode, b)
}
if err := json.NewDecoder(resp.Body).Decode(out); err != nil {
log.Fatal(err)
}
}
func embed(texts []string) [][]float64 {
var data struct {
Data []struct {
Index int `json:"index"`
Embedding []float64 `json:"embedding"`
} `json:"data"`
}
post("/embeddings", map[string]any{"model": embeddingModel, "input": texts}, &data)
out := make([][]float64, len(data.Data))
for _, item := range data.Data {
out[item.Index] = item.Embedding
}
return out
}
func cosineSimilarity(a, b []float64) float64 {
var dot, magA, magB float64
for i := range a {
dot += a[i] * b[i]
magA += a[i] * a[i]
magB += b[i] * b[i]
}
return dot / (math.Sqrt(magA) * math.Sqrt(magB))
}
type match struct {
Index int
Score float64
}
func retrieve(queryEmbedding []float64, docEmbeddings [][]float64, topN int) []match {
scored := make([]match, len(docEmbeddings))
for i, emb := range docEmbeddings {
scored[i] = match{i, cosineSimilarity(queryEmbedding, emb)}
}
sort.Slice(scored, func(i, j int) bool { return scored[i].Score > scored[j].Score })
if topN < len(scored) {
scored = scored[:topN]
}
return scored
}
type RerankResult struct {
Index int `json:"index"`
RelevanceScore float64 `json:"relevance_score"`
Document struct {
Text string `json:"text"`
} `json:"document"`
}
func rerank(query string, documents []string, topN int) []RerankResult {
var data struct {
Results []RerankResult `json:"results"`
}
post("/rerank", map[string]any{
"model": rerankModel, "query": query, "documents": documents, "top_n": topN,
}, &data)
return data.Results
}
func generate(query string, contextDocs []string) string {
parts := make([]string, len(contextDocs))
for i, doc := range contextDocs {
parts[i] = fmt.Sprintf("[%d] %s", i+1, doc)
}
var data struct {
Output []struct {
Type string `json:"type"`
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
} `json:"output"`
}
post("/responses", map[string]any{
"model": chatModel,
"instructions": "Answer the user's question using only the provided context. " +
"Cite sources with [n]. If the context is insufficient, say so.",
"input": fmt.Sprintf("Context:\n%s\n\nQuestion: %s", strings.Join(parts, "\n\n"), query),
}, &data)
for i := len(data.Output) - 1; i >= 0; i-- {
if data.Output[i].Type != "message" {
continue
}
for _, part := range data.Output[i].Content {
if part.Type == "output_text" {
return part.Text
}
}
}
return ""
}
func main() {
// 1. Index: chunk your knowledge base and embed
chunks := []string{
"OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers into a single OpenAI-compatible interface.",
"RAG stands for Retrieval-Augmented Generation. It retrieves relevant documents and uses them as context for LLM generation.",
"Embeddings convert text into high-dimensional vectors where semantically similar texts are closer together.",
"Reranking uses a cross-encoder to compare each document directly against the query, producing more accurate relevance scores than embedding similarity alone.",
"Vector databases like Pinecone, Weaviate, and Qdrant are optimized for storing and querying embedding vectors at scale.",
"Prompt caching reduces costs by reusing computations for repeated prompt prefixes across requests.",
"OpenRouter supports provider routing to control which upstream providers serve your requests based on cost, latency, or privacy preferences.",
}
docEmbeddings := embed(chunks)
// 2. Retrieve: embed the query and find similar chunks
query := "How does RAG improve LLM responses?"
queryEmbedding := embed([]string{query})[0]
var retrievedTexts []string
for _, m := range retrieve(queryEmbedding, docEmbeddings, 5) {
retrievedTexts = append(retrievedTexts, chunks[m.Index])
}
// 3. Rerank: re-score with a cross-encoder for better precision
var contextTexts []string
for _, r := range rerank(query, retrievedTexts, 3) {
contextTexts = append(contextTexts, r.Document.Text)
}
// 4. Generate: produce a grounded answer
answer := generate(query, contextTexts)
fmt.Printf("Q: %s\nA: %s\n", query, answer)
}
#!/bin/bash
set -eo pipefail # a failed request or an empty rerank result stops the script
BASE_URL="https://orr.api.report.finance/api/v1"
QUERY="How does RAG improve LLM responses?"
# 1. Index: chunk your knowledge base and embed
CHUNKS='[
"OpenRouter is a unified API gateway for LLMs. It aggregates models from multiple providers into a single OpenAI-compatible interface.",
"RAG stands for Retrieval-Augmented Generation. It retrieves relevant documents and uses them as context for LLM generation.",
"Embeddings convert text into high-dimensional vectors where semantically similar texts are closer together.",
"Reranking uses a cross-encoder to compare each document directly against the query, producing more accurate relevance scores than embedding similarity alone.",
"Vector databases like Pinecone, Weaviate, and Qdrant are optimized for storing and querying embedding vectors at scale.",
"Prompt caching reduces costs by reusing computations for repeated prompt prefixes across requests.",
"OpenRouter supports provider routing to control which upstream providers serve your requests based on cost, latency, or privacy preferences."
]'
embed() {
curl -sS --fail-with-body "$BASE_URL/embeddings" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n --argjson input "$1" '{model: "openai/text-embedding-3-small", input: $input}')"
}
DOC_EMB=$(embed "$CHUNKS")
# 2. Retrieve: embed the query and find the top 5 chunks by cosine similarity
QUERY_EMB=$(embed "$(jq -n --arg q "$QUERY" '[$q]')" | jq -c '.data[0].embedding')
RETRIEVED=$(echo "$DOC_EMB" | jq -c --argjson qe "$QUERY_EMB" --argjson chunks "$CHUNKS" '
def dot(a; b): [a, b] | transpose | map(.[0] * .[1]) | add;
# Pair each vector with its chunk by the index field, not by array position
[.data[] | {index, score: (dot(.embedding; $qe) / ((dot(.embedding; .embedding) | sqrt) * (dot($qe; $qe) | sqrt)))}]
| sort_by(-.score) | .[:5] | map($chunks[.index])')
# 3. Rerank: re-score with a cross-encoder for better precision
CONTEXT=$(curl -sS --fail-with-body "$BASE_URL/rerank" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n --arg q "$QUERY" --argjson docs "$RETRIEVED" '{model: "cohere/rerank-v3.5", query: $q, documents: $docs, top_n: 3}')" \
| jq -er '.results | select(length > 0) | [.[].document.text] | to_entries | map("[\(.key + 1)] \(.value)") | join("\n\n")')
# 4. Generate: produce a grounded answer
ANSWER=$(curl -sS --fail-with-body "$BASE_URL/responses" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n --arg context "$CONTEXT" --arg query "$QUERY" '{
model: "~openai/gpt-mini-latest",
instructions: "Answer the user'\''s question using only the provided context. Cite sources with [n]. If the context is insufficient, say so.",
input: "Context:\n\($context)\n\nQuestion: \($query)"
}')" | jq -r '[.output[] | select(.type == "message")] | last | .content[] | select(.type == "output_text") | .text')
echo "Q: $QUERY"
echo "A: $ANSWER"
Output
Q: How does RAG improve LLM responses?
A: RAG improves LLM responses by first retrieving relevant documents and then using those documents as context for generation, which helps the model answer with more grounded and relevant information [1].
When to Use Rerank
Reranking adds an extra API call, so it’s worth understanding when it helps most: Use rerank when:- Your knowledge base is large (hundreds or thousands of chunks) and embedding retrieval alone returns noisy results
- Precision matters more than latency (e.g., customer-facing Q&A, legal or medical documents)
- You retrieve many candidates (e.g., top 20) and need to narrow to the best 3-5
- Your knowledge base is small and embedding retrieval already returns highly relevant results
- You need the lowest possible latency (rerank adds one additional API call)
- You’re building a prototype and want to keep the pipeline simple
Multimodal Rerank (Text + Images)
Some rerank models accept images alongside text. Instead of passing plain strings, you pass document objects with an optionaltext field, an optional image field, or both. The image value can be a remote URL (http/https) or a base64-encoded data URI (data:image/...). At least one of text or image is required per document.
This is useful when your knowledge base contains diagrams, screenshots, charts, or scanned pages where the visual content carries meaning the caption alone doesn’t.
import os
import requests
OPENROUTER_API_KEY = os.environ["OPENROUTER_API_KEY"]
headers = {"Authorization": f"Bearer {OPENROUTER_API_KEY}"}
# Find rerank models that also accept image input
models = {}
for modality in ("rerank", "all"):
res = requests.get(
"https://orr.api.report.finance/api/v1/models",
params={"output_modalities": modality},
headers=headers,
)
res.raise_for_status() # a failed lookup is an HTTP error, not "no model"
for m in res.json().get("data", []):
models[m["id"]] = m
def is_image_reranker(m):
arch = m.get("architecture") or {}
outputs = arch.get("output_modalities") or []
inputs = arch.get("input_modalities") or []
return ("rerank" in outputs or "rerank" in m["id"]) and "image" in inputs
model = next((mid for mid, m in models.items() if is_image_reranker(m)), None)
if model is None:
print("No image-capable rerank model is currently available.")
else:
print(f"Using model: {model}")
response = requests.post(
"https://orr.api.report.finance/api/v1/rerank",
headers={**headers, "Content-Type": "application/json"},
json={
"model": model,
"query": "How is AI improving the capabilities of robots?",
"documents": [
{"text": "AI enables robots to perceive, plan, and act autonomously."},
{"image": "https://upload.wikimedia.org/wikipedia/commons/thumb/0/05/HONDA_ASIMO.jpg/320px-HONDA_ASIMO.jpg"},
{
"text": "A robotic arm assembling components.",
"image": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/KUKA_Industrial_Robot_KR6.jpg/320px-KUKA_Industrial_Robot_KR6.jpg",
},
],
"top_n": 2,
},
)
data = response.json()
if not response.ok:
print(data)
else:
for r in data["results"]:
print(f" Score: {r['relevance_score']:.4f} | index: {r['index']}")
// cohere/rerank-v3.5 is text-only, so look up a rerank model that also accepts images
const modelsRes = await fetch('https://orr.api.report.finance/api/v1/models?output_modalities=all', {
headers: { 'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}` },
});
if (!modelsRes.ok) {
throw new Error(`Model lookup failed: HTTP ${modelsRes.status} ${await modelsRes.text()}`);
}
const { data: models } = await modelsRes.json();
const model = models.find(
(m: any) =>
m.architecture?.output_modalities?.includes('rerank') &&
m.architecture?.input_modalities?.includes('image'),
)?.id;
if (!model) {
console.log('No image-capable rerank model is currently available.');
} else {
console.log(`Using model: ${model}`);
const response = await fetch('https://orr.api.report.finance/api/v1/rerank', {
method: 'POST',
headers: {
'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model,
query: 'How is AI improving the capabilities of robots?',
documents: [
{ text: 'AI enables robots to perceive, plan, and act autonomously.' },
{ image: 'https://upload.wikimedia.org/wikipedia/commons/thumb/0/05/HONDA_ASIMO.jpg/320px-HONDA_ASIMO.jpg' },
{
text: 'A robotic arm assembling components.',
image: 'https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/KUKA_Industrial_Robot_KR6.jpg/320px-KUKA_Industrial_Robot_KR6.jpg',
},
],
top_n: 2,
}),
});
const data = await response.json();
if (!response.ok) {
console.log(JSON.stringify(data, null, 2));
} else {
for (const r of data.results) {
console.log(` Score: ${r.relevance_score.toFixed(4)} | index: ${r.index}`);
}
}
}
curl https://orr.api.report.finance/api/v1/rerank \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d '{
"model": "nvidia/llama-nemotron-rerank-vl-1b-v2",
"query": "How is AI improving the capabilities of robots?",
"documents": [
{ "text": "AI enables robots to perceive, plan, and act autonomously." },
{ "image": "https://example.com/robot-diagram.png" },
{
"text": "A robotic arm assembling components.",
"image": "https://example.com/robot-arm.png"
}
],
"top_n": 2
}'
Output
No image-capable rerank model is currently available.
Text-only documents can still be passed as plain strings, so you can mix
"a plain string" and { "image": "..." } entries in the same documents array. Browse multimodal rerank models at orr.api.report.finance/models?output_modalities=rerank.Chunking Strategies
How you split documents significantly affects retrieval quality:- By paragraph or section — preserves semantic coherence and works well for structured documents
- Fixed-size with overlap — split into chunks of ~200-500 tokens with ~50-token overlap to avoid losing context at boundaries
- By semantic boundary — use headings, section breaks, or sentence boundaries to create natural chunks
Smaller chunks (200-300 tokens) tend to produce more precise retrieval but may lose surrounding context. Larger chunks (500-1000 tokens) preserve more context but may dilute relevance signals. Experiment with your specific data to find the right balance.
Best Practices
Use the same embedding model for indexing and querying. Mixing models produces incompatible vector spaces and will give poor retrieval results. Batch your embedding requests. Send multiple texts in a single API call to reduce latency and costs. The embeddings API accepts arrays of inputs. Cache embeddings. Embeddings for the same text are deterministic. Store them in a database to avoid recomputing. Retrieve more than you need, then rerank. A common pattern is to retrieve 10-20 candidates via embeddings, then rerank to the top 3-5. This combines the speed of embedding search with the precision of cross-encoder reranking. Include metadata in your prompt. When generating, include source metadata (document title, section, URL) alongside the text so the LLM can produce proper citations. Set a relevance threshold. After reranking, filter out documents below a minimum relevance score to avoid injecting irrelevant context that could confuse the LLM.Available Models
Browse available models for each step:- Embedding models: orr.api.report.finance/models?output_modalities=embeddings
- Rerank models: orr.api.report.finance/models?output_modalities=rerank
- Chat models: orr.api.report.finance/models
Related Resources
- Embeddings API — full API reference for generating embeddings
- Provider Routing — control which providers serve your requests
- Prompt Caching — reduce costs for repeated prompt prefixes
- Structured Outputs — enforce JSON schema on LLM responses for structured RAG output