In this tutorial, we'll learn how to build an image search engine using Pinecone and the CLIP model. We'll use a small dataset of images (included in this repository) and show how to embed and index them in Pinecone. We'll then show how to query the index to find similar images. Finally, we'll demonstrate upserting and deleting images and their vector embeddings.
To complete this tutorial, you'll need a Pinecone account. If you don't have one yet, you can sign up for free at pinecone.io.
You'll also need Node.js and npm installed. You can download Node.js from nodejs.org.
In order to run this example, you have to supply the Pinecone credentials needed to interact with the Pinecone API. You can find these credentials in the Pinecone web console under API Keys. This project uses dotenv to easily load values from the .env file into the environment when executing.
Copy the template file:
cp .env.example .envAnd fill in your API key and index name:
PINECONE_API_KEY=<your-api-key>
PINECONE_INDEX="image-search"
PINECONE_CLOUD="aws"
PINECONE_REGION="us-east-1"PINECONE_INDEX is the name of the index where this demo will store and query embeddings. You can change PINECONE_INDEX to any name you like, but make sure the name not going to collide with any indexes you are already using.
PINECONE_CLOUD and PINECONE_REGION define where the serverless index is deployed. Both are optional — if you leave them out of your .env, the app defaults to aws and us-east-1.
If you're on Pinecone's free (Starter) plan, aws/us-east-1 is the only cloud and region you can create an index in, so the defaults work out of the box with no changes. Setting them to any other region will cause index creation to fail with an error asking you to upgrade your plan. Deploying elsewhere requires a paid plan.
You can see the full list of dependencies in the package.json file, but the main ones are:
- Pinecone - the Pinecone SDK for Node.js
- Transformers.js - a library for embedding images using a pre-trained model
- Express.js - a web framework for Node.js used to build the API
- React.js - a JavaScript library for building the user interface
To install the dependencies, run:
npm installThe CLIP (Contrastive Language-Image Pretraining) model was developed by OpenAI that connects images and text in a unique way. Unlike previous models that were trained on one or the other, CLIP is trained on a variety of internet text paired with images. The model learns to understand and generate meaningful responses about images based on the context provided by the associated text.
This allows for sophisticated image search capabilities; you can find images based on a textual description or even use a detailed phrase to search for a specific type of image. The model's ability to create a shared embedding space for images and text means that CLIP can convert an image into a vector that can be indexed and searched within a vector database, like Pinecone.
Let's start by taking a look at the top level indexImages function. The function will:
- Create a serverless index in Pinecone if it doesn't already exist, and wait until it's ready to be used. The CLIP model we'll be using for embedding the images has an embedding size of 512, so we'll use that as the dimension of the index. The
cloudandregioncome from the environment (defaulting toaws/us-east-1, as described above). - Initialize the image embedding model
- Retrieve the list of all the images in the
datafolder - embed the images and upsert them into the index
const indexImages = async () => {
const indexName = getEnv("PINECONE_INDEX");
const indexCloud = getEnv("PINECONE_CLOUD", DEFAULT_PINECONE_CLOUD);
const indexRegion = getEnv("PINECONE_REGION", DEFAULT_PINECONE_REGION);
const pinecone = new Pinecone();
try {
// Create the index if it doesn't already exist
const indexList = await pinecone.listIndexes();
if (!indexList.indexes?.some((index) => index.name === indexName)) {
await pinecone.createIndex({
name: indexName,
dimension: 512,
spec: { serverless: { region: indexRegion, cloud: indexCloud } },
waitUntilReady: true,
});
}
// Get the index
const index = pinecone.index(indexName);
await embedder.ready();
const imagePaths = await listFiles("./data");
await embedAndUpsert({ imagePaths, chunkSize: 100, index });
} catch (error) {
console.error(error);
throw error;
}
};Now let's break the embedding process up a bit. The Embedder class is initialized using the AutoModel, AutoTokenizer and AutoProcessor which automatically defines the model, tokenizer and processor based on the model name. Rather than calling init directly, callers use ready(), which loads the model at most once and hands every subsequent caller the same in-flight promise — so the ~600MB CLIP weights are never loaded twice even though several entry points (indexImages, queryImages, deleteImage) all need the embedder. The embed function takes an image path and returns the embedding. The embedBatch function takes a batch of image paths and calls the onDoneBatch callback with the embeddings. For the ID of the embedding, we use the MD5 hash of the image path.
In this example, we're not going to use any text inputs, so we just pass an empty string as the input to the tokenizer.
class Embedder {
private processor: Processor;
private model: PreTrainedModel;
private tokenizer: PreTrainedTokenizer;
// Caches the in-flight/completed init so the model is only loaded once,
// no matter how many entry points call ready().
private initPromise: Promise<void> | null = null;
async init(modelName: string) {
// Load the model, tokenizer and processor
this.model = await AutoModel.from_pretrained(modelName);
this.tokenizer = await AutoTokenizer.from_pretrained(modelName);
this.processor = await AutoProcessor.from_pretrained(modelName);
}
// Idempotent, lazy initialization. Safe to call from every entry point:
// the model is loaded at most once and subsequent calls await the same promise.
async ready(modelName: string = DEFAULT_MODEL): Promise<void> {
if (!this.initPromise) {
this.initPromise = this.init(modelName);
}
return this.initPromise;
}
// Embeds an image and returns the embedding
async embed(
imagePath: string,
metadata?: RecordMetadata
): Promise<DenseRecord> {
try {
// Ensure the model is loaded before embedding.
await this.ready();
// Load the image
const image = await RawImage.read(imagePath);
// Prepare the image and text inputs
const image_inputs = await this.processor(image);
const text_inputs = this.tokenizer([""], {
padding: true,
truncation: true,
});
// Embed the image
const output = await this.model({ ...text_inputs, ...image_inputs });
const { image_embeds } = output;
const { data: embeddings } = image_embeds;
// Create an id for the image
const id = createHash("md5").update(imagePath).digest("hex");
// Return the embedding in a format ready for Pinecone
return {
id,
metadata: metadata || {
imagePath,
},
values: Array.from(embeddings) as number[],
};
} catch (e) {
console.log(`Error embedding image, ${e}`);
throw e;
}
}
// Embeds a batch of documents and calls onDoneBatch with the embeddings
async embedBatch(
imagePaths: string[],
batchSize: number,
onDoneBatch: (embeddings: PineconeRecord[]) => void
) {
await this.ready();
const batches = sliceIntoChunks<string>(imagePaths, batchSize);
for (const batch of batches) {
const embeddings = await Promise.all(
batch.map((imagePath) => this.embed(imagePath))
);
await onDoneBatch(embeddings);
}
}
}The embedAndUpsert function takes a list of image paths, a chunk size, and the target index, then proceeds to embed and upsert these images in chunks. It's shared by both indexImages (the initial bulk index) and upsertImages (uploads), so the caller passes in the index it already holds:
async function embedAndUpsert({
imagePaths,
chunkSize,
index,
}: {
imagePaths: string[];
chunkSize: number;
index: Index;
}) {
// Chunk the image paths into batches of size chunkSize
const chunkGenerator = chunkArray(imagePaths, chunkSize);
// Embed each batch and upsert the embeddings into the index
for await (const imagePaths of chunkGenerator) {
await embedder.embedBatch(
imagePaths,
chunkSize,
async (embeddings: PineconeRecord[]) => {
await chunkedUpsert(index, embeddings, "default");
}
);
}
}We expose the indexImages function through the /indexImages route (registered in server/routes.ts). It isn't run at server start — it fires when you click the "Index" button in the UI, which we'll wire up later on.
In order to easily present the results of our image query, we created a simple UI using React. The UI will present a set of images and allow us to select one of them. The query will then return the most similar images to the selected image.
Once an image is selected, it's path will be sent to the /search endpoint, which will query the index and return the results. The queryImages function will:
- Confine the client-supplied
imagePathto thedatadirectory before touching the filesystem — the path arrives from an HTTP request, so we guard against../traversal before reading it - Embed the selected image (the embedder lazily loads the CLIP model on first use)
- Use the embedding to query the index
- Return the matching images and their respective scores
Note that we pass a metadata type parameter to pinecone.index<Metadata>(...), which gives us type-checked access to match.metadata.imagePath in the results.
export type Metadata = {
imagePath: string;
};
let index: Index<Metadata>;
const getIndex = (): Index<Metadata> => {
if (!index) {
const pinecone = new Pinecone();
index = pinecone.index<Metadata>(getEnv("PINECONE_INDEX"));
}
return index;
};
const queryImages = async (imagePath: string) => {
// Confine the client-supplied path to the data directory before reading it:
// path.resolve collapses any "..", and a path.relative that starts with ".."
// means the resolved path escaped DATA_DIR.
const safePath = path.resolve(imagePath);
if (path.relative(DATA_DIR, safePath).startsWith("..")) {
throw new Error(`Invalid image path: ${imagePath}`);
}
const queryEmbedding = await embedder.embed(safePath);
const queryResult = await getIndex().namespace("default").query({
vector: queryEmbedding.values,
includeMetadata: true,
includeValues: true,
topK: 6,
});
return queryResult.matches?.map((match) => {
const { metadata } = match;
return {
src: metadata ? metadata.imagePath : "",
score: match.score,
};
});
};The application is a simple React app that allows us to select an image and query the index for similar images. The app is served by the Express server, which also exposes the /getImages, /search, /indexImages, /uploadImages, and /deleteImage endpoints.
The front end paginates the query results and displays them in a grid. The main App component renders the UI and calls the /search endpoint when an image is selected.
When an image is clicked, we call the following function:
const handleImageClick = async (imagePath: string) => {
setSelectedImage(imagePath);
const response = await fetch(
`/search?imagePath=${encodeURIComponent(imagePath)}`
);
const matchingImages: SearchResult[] = await response.json();
setSearchResults(matchingImages);
};When uploading a new image, we reuse the embedAndUpsert function to embed and upsert the image into the index. The new image is immediately queryable without needing to rebuild the index. This data freshness is a major advantage of using a vector database like Pinecone.
To delete an image, we call the /deleteImage endpoint with the image path. We need to find the imageʼs Pinecone ID before deleting the vector embedding. We can then remove the vector embedding from the index.
In order to run the application:
- If you haven't already, clone this repository.
- Then run:
npm run devYou should receive a message: "Server started at http://localhost:3000". Copy this url into your web browser.
- Click the green "Index" button in the top left of the screen. This embeds the sample images and upserts them into Pinecone, so it can take a moment — watch the terminal running
npm run dev, where you'll see the index being created and vectors being upserted. Wait for the app to finish loading before continuing. - Select an image.
- You will now be shown all similar images found within the dataset.
And here's the final result:
npm run dev is all you need for local development — it runs the Vite dev server for the frontend and a tsx-powered, auto-reloading Express server for the backend.
To produce a production build of the frontend (type-checks with tsc, then bundles into app/dist):
npm run build:appThere is no separate build step for the server; it is run directly from TypeScript via tsx.
The images under data/ are already flattened and ready to index. flatten.sh is a one-off helper for flattening a nested train/ directory into a single level (renaming files to <subdir>-<filename>) if you supply your own hierarchical image dataset. It is not part of the normal run/build flow.
The project uses Vitest. Tests are organized in layers so that dependency bumps (e.g. Dependabot PRs) are caught before they silently break the example.
npm test # run the whole suite once (server + frontend)
npm run test:watch # watch mode
npm run test:server # server/model/Pinecone tests only
npm run test:app # React component tests only
npm run typecheck # tsc --noEmitWhat's covered:
- Unit tests (
tests/server/util.test.ts,pagination.test.ts) — pure logic: chunking,getEnv, file listing/filtering, and pagination. Fast, no network or model. - Model integration test (
tests/server/embeddings.integration.test.ts) — loads the real CLIP model via@huggingface/transformersand embeds committed fixture images (tests/fixtures/images/). It asserts the embeddings are 512-dimensional and finite, deterministic across runs, and semantically meaningful (a nearest-neighbor search returns same-subject images — the same operation the app performs). This is the test most likely to catch a breaking change in@huggingface/transformersoronnxruntime-node. The first run downloads the model weights (~600MB); subsequent runs use the local cache. - Pinecone wrapper tests (
tests/server/pinecone.mocked.test.ts) — the SDK and model are mocked to verify our query/delete logic (result mapping, namespace/params, id lookup + file rename) quickly and offline. - API tests (
tests/server/api.test.ts) — drive the real Express app withsupertest, exercising routing,multerfile uploads, query params, status codes, and error handling (the Pinecone/model layer mocked). - React component tests (
app/src/App.test.tsx) — render the realAppwithfetchstubbed and verify the frontend calls the right endpoints and reflects responses (mount fetch, search, indexing, pagination).
tests/server/pinecone.e2e.test.ts runs the real SDK operations (createIndex, upsert, query, deleteOne) against a live serverless index — the truest signal that a @pinecone-database/pinecone bump hasn't broken the app. It is skipped unless PINECONE_API_KEY is set, and creates then deletes a uniquely-named throwaway index:
PINECONE_API_KEY=... npm run test:serverTests run automatically in CI (.github/workflows/test.yml) on every push and pull request, including Dependabot bumps; the model download is cached. The live e2e test runs as a separate job on pushes to main and manual dispatch, and only does real work when the PINECONE_API_KEY repository secret is configured.