diff --git a/authors.yaml b/authors.yaml index 92b89168e9..f26877a415 100644 --- a/authors.yaml +++ b/authors.yaml @@ -547,3 +547,9 @@ derrickchoi-openai: name: "Derrick Choi" website: "https://www.linkedin.com/in/derrickchoi/" avatar: "https://avatars.githubusercontent.com/u/211427900" + + +mhartington: + name: "Mike Hartington" + website: "https://mhartington.io" + avatar: "https://avatars.githubusercontent.com/u/2835826" diff --git a/examples/vector_databases/prisma/README.md b/examples/vector_databases/prisma/README.md new file mode 100644 index 0000000000..ed501f7e45 --- /dev/null +++ b/examples/vector_databases/prisma/README.md @@ -0,0 +1,35 @@ +# What is Prisma Postgres? + +[Prisma Postgres](https://pris.ly/Wd7WWHD) is a managed Postgres solution built for the cloud. Your app stays lightning-fast and responsive, whether you’re going from zero, or handling a spike. Free to start, pay when you grow. + +## Vector search + +Prisma Postgres supports vector search using the [pgvector](https://www.prisma.io/docs/postgres/database/postgres-extensions) open-source PostgreSQL extension, which enables Postgres as a vector database for storing and querying embeddings. + +## OpenAI cookbook notebook + +Check out the notebook in this repo for working with Prisma Postgres as your vector database. + +### Semantic search using Prisma Postgres with pgvector and OpenAI + +In this notebook you will learn how to: + +1. Use embeddings created by OpenAI API +2. Store embeddings in a Prisma Postgres database +3. Convert a raw text query to an embedding with OpenAI API +4. Use Prisma Postgres with the `pgvector` extension to perform vector similarity search + +## Scaling Support + +Prisma Postgres enables you to scale your AI applications with the following features: + +- Autoscaling: If your AI application experiences heavy load during certain hours of the day or at different times, Prisma can automatically scale compute resources without manual intervention. During periods of inactivity, Prisma is able to scale to zero. + +- [The Prisma Postgres serverless driver](https://www.prisma.io/docs/postgres/database/serverless-driver): Prisma Postgres supports a low-latency serverless PostgreSQL driver for JavaScript and TypeScript applications that allows you to query data from serverless and edge environments. + +## Additional Resources + +- [Prisma MCP Server](https://www.prisma.io/mcp) +- [Prisma Postgres](https://www.prisma.io/postgres) +- [Vibe Coding with Limits — How to Build Apps in the Age of AI](https://www.prisma.io/blog/vibe-coding-with-limits-how-to-build-apps-in-the-age-of-ai) +- [pgvector GitHub repository](https://github.com/pgvector/pgvector) diff --git a/examples/vector_databases/prisma/prisma-postgres-vector-search-pgvector.ipynb b/examples/vector_databases/prisma/prisma-postgres-vector-search-pgvector.ipynb new file mode 100644 index 0000000000..5f3f37acee --- /dev/null +++ b/examples/vector_databases/prisma/prisma-postgres-vector-search-pgvector.ipynb @@ -0,0 +1,468 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Vector similarity search using Prisma Postgres\n", + "\n", + "This notebook guides you through using [Prisma Postgres](https://pris.ly/Wd7WWHD) as a vector database for OpenAI embeddings. It demonstrates how to:\n", + "\n", + "1. Use embeddings created by OpenAI API.\n", + "2. Store embeddings in a Prisma Postgres database.\n", + "3. Convert a raw text query to an embedding with OpenAI API.\n", + "4. Use Prisma with the `pgvector` extension to perform vector similarity search." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "Before you begin, ensure that you have the following:\n", + "\n", + "1. A Prisma Postgres database. You can create an account and set up a project with a ready-to-use database in a few simple steps. For instructions, see [Sign up](https://pris.ly/Wd7WWHD) or create a database with the online [create-db](https://create-db.prisma.io) interface.\n", + "2. A connection string for your Prisma database. You can copy it from the **Connection Details** widget on the Prisma **Dashboard**. See [Direct Connections](https://www.prisma.io/docs/postgres/database/direct-connections).\n", + "3. The `pgvector` extension. For instructions, see [Postgres Extensions](https://www.prisma.io/docs/postgres/database/postgres-extensions#using-extensions-with-prisma-orm). \n", + "4. Your [OpenAI API key](https://platform.openai.com/account/api-keys).\n", + "5. Python and `pip`." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install required modules\n", + "\n", + "This notebook requires the `openai`, `psycopg2`, `pandas`, `wget`, and `python-dotenv` packages. You can install them with `pip`:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "! pip install openai psycopg2 pandas wget python-dotenv" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Prepare your OpenAI API key\n", + "\n", + "An OpenAI API key is required to generate vectors for documents and queries.\n", + "\n", + "If you do not have an OpenAI API key, obtain one from https://platform.openai.com/account/api-keys.\n", + "\n", + "Add the OpenAI API key as an operating system environment variable or provide it for the session when prompted. If you define an environment variable, name the variable `OPENAI_API_KEY`.\n", + "\n", + "For information about configuring your OpenAI API key as an environment variable, refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test your OpenAPI key" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Test to ensure that your OpenAI API key is defined as an environment variable or provide it when prompted\n", + "# If you run this notebook locally, you may have to reload the terminal and the notebook to make the environment available\n", + "\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "# Check if OPENAI_API_KEY is set as an environment variable\n", + "if os.getenv(\"OPENAI_API_KEY\") is not None:\n", + " print(\"Your OPENAI_API_KEY is ready\")\n", + "else:\n", + " # If not, prompt for it\n", + " api_key = getpass(\"Enter your OPENAI_API_KEY: \")\n", + " if api_key:\n", + " print(\"Your OPENAI_API_KEY is now available for this session\")\n", + " # Optionally, you can set it as an environment variable for the current session\n", + " os.environ[\"OPENAI_API_KEY\"] = api_key\n", + " else:\n", + " print(\"You did not enter your OPENAI_API_KEY\")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Connect to your Prisma Postgres database\n", + "\n", + "Provide your Prisma Postgres database connection string below or define it in an `.env` file using a `DATABASE_URL` variable. For information about obtaining a connection string, see [Direct Connections](https://www.prisma.io/docs/postgres/database/direct-connections)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import psycopg2\n", + "from dotenv import load_dotenv\n", + "\n", + "# Load environment variables from .env file\n", + "load_dotenv()\n", + "\n", + "# The connection string can be provided directly here.\n", + "# Replace the next line with Your Prisma connection string.\n", + "connection_string = \"postgres://:@/\"\n", + "\n", + "# If connection_string is not directly provided above, \n", + "# then check if DATABASE_URL is set in the environment or .env.\n", + "if not connection_string:\n", + " connection_string = os.environ.get(\"DATABASE_URL\")\n", + "\n", + " # If neither method provides a connection string, raise an error.\n", + " if not connection_string:\n", + " raise ValueError(\"Please provide a valid connection string either in the code or in the .env file as DATABASE_URL.\")\n", + "\n", + "# Connect using the connection string\n", + "connection = psycopg2.connect(connection_string)\n", + "\n", + "# Create a new cursor object\n", + "cursor = connection.cursor()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Test the connection to your database:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Your database connection was successful!\n" + ] + } + ], + "source": [ + "# Execute this query to test the database connection\n", + "cursor.execute(\"SELECT 1;\")\n", + "result = cursor.fetchone()\n", + "\n", + "# Check the query result\n", + "if result == (1,):\n", + " print(\"Your database connection was successful!\")\n", + "else:\n", + " print(\"Your connection failed.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n", + "\n", + "Import the pre-computed embeddings zip file:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import wget\n", + "\n", + "embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n", + "\n", + "# The file is ~700 MB. Importing it will take several minutes.\n", + "wget.download(embeddings_url)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Extract the downloaded zip file:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import zipfile\n", + "import os\n", + "import re\n", + "import tempfile\n", + "\n", + "current_directory = os.getcwd()\n", + "zip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\n", + "output_directory = os.path.join(current_directory, \"../../data\")\n", + "\n", + "with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n", + " zip_ref.extractall(output_directory)\n", + "\n", + "\n", + "# Check to see if the csv file was extracted\n", + "file_name = \"vector_database_wikipedia_articles_embedded.csv\"\n", + "data_directory = os.path.join(current_directory, \"../../data\")\n", + "file_path = os.path.join(data_directory, file_name)\n", + "\n", + "\n", + "if os.path.exists(file_path):\n", + " print(f\"The csv file {file_name} exists in the data directory.\")\n", + "else:\n", + " print(f\"The csv file {file_name} does not exist in the data directory.\")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create a table and add indexes for your vector embeddings\n", + "\n", + "The vector table created in your database is called **articles**. Each object has **title** and **content** vectors. \n", + "\n", + "An index is defined on both the **title** and **content** vector columns." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "create_table_sql = '''\n", + "CREATE TABLE IF NOT EXISTS public.articles (\n", + " id INTEGER NOT NULL,\n", + " url TEXT,\n", + " title TEXT,\n", + " content TEXT,\n", + " title_vector vector(1536),\n", + " content_vector vector(1536),\n", + " vector_id INTEGER\n", + ");\n", + "\n", + "ALTER TABLE public.articles ADD PRIMARY KEY (id);\n", + "'''\n", + "\n", + "# SQL statement for creating indexes\n", + "create_indexes_sql = '''\n", + "CREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n", + "\n", + "CREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n", + "'''\n", + "\n", + "# Execute the SQL statements\n", + "cursor.execute(create_table_sql)\n", + "cursor.execute(create_indexes_sql)\n", + "\n", + "# Commit the changes\n", + "connection.commit()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load the data\n", + "\n", + "Load the pre-computed vector data into your `articles` table from the `.csv` file. There are 25000 records, so expect the operation to take several minutes." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "import io\n", + "\n", + "# Path to your local CSV file\n", + "csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n", + "\n", + "# Define a generator function to process the csv file\n", + "def process_file(file_path):\n", + " with open(file_path, 'r', encoding='utf-8') as file:\n", + " for line in file:\n", + " yield line\n", + "\n", + "# Create a StringIO object to store the modified lines\n", + "modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n", + "\n", + "# Create the COPY command for copy_expert\n", + "copy_command = '''\n", + "COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\n", + "FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n", + "'''\n", + "\n", + "# Execute the COPY command using copy_expert\n", + "cursor.copy_expert(copy_command, modified_lines)\n", + "\n", + "# Commit the changes\n", + "connection.commit()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Check the number of records to ensure the data has been been loaded. There should be 25000 records." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Count:25000\n" + ] + } + ], + "source": [ + "# Check the size of the data\n", + "count_sql = \"\"\"select count(*) from public.articles;\"\"\"\n", + "cursor.execute(count_sql)\n", + "result = cursor.fetchone()\n", + "print(f\"Count:{result[0]}\")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Search your data\n", + "\n", + "After the data is stored in your Prisma Postgres database, you can query the data for nearest neighbors. \n", + "\n", + "Start by defining the `query_prisma` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n", + "\n", + "A `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\"." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def query_prisma(query, collection_name, vector_name=\"title_vector\", top_k=20):\n", + "\n", + " # Create an embedding vector from the user query\n", + " embedded_query = client.embeddings.create(input=query,\n", + " model=\"text-embedding-3-small\")[\"data\"][0][\"embedding\"]\n", + "\n", + " # Convert the embedded_query to PostgreSQL compatible format\n", + " embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n", + "\n", + " # Create the SQL query\n", + " query_sql = f\"\"\"\n", + " SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n", + " FROM {collection_name}\n", + " ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n", + " LIMIT {top_k};\n", + " \"\"\"\n", + " # Execute the query\n", + " cursor.execute(query_sql)\n", + " results = cursor.fetchall()\n", + "\n", + " return results\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Run a similarity search based on `title_vector` embeddings:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Query based on `title_vector` embeddings\n", + "from openai import OpenAI\n", + "\n", + "client = OpenAI()\n", + "\n", + "\n", + "query_results = query_prisma(\"Greek mythology\", \"Articles\")\n", + "for i, result in enumerate(query_results):\n", + " print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Run a similarity search based on `content_vector` embeddings:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Query based on `content_vector` embeddings\n", + "query_results = query_prisma(\"Famous battles in Greek history\", \"Articles\", \"content_vector\")\n", + "for i, result in enumerate(query_results):\n", + " print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.17" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/registry.yaml b/registry.yaml index bfebed7d39..04723470bf 100644 --- a/registry.yaml +++ b/registry.yaml @@ -1650,6 +1650,24 @@ tags: - embeddings + +- title: Prisma as a vector database + path: examples/vector_databases/prisma/README.md + date: 2025-11-27 + authors: + - mhartington + tags: + - embeddings + +- title: Vector similarity search using Prisma Postgres + path: examples/vector_databases/prisma/prisma-postgres-vector-search-pgvector.ipynb + date: 2025-11-27 + authors: + - mhartington + tags: + - embeddings + + - title: Question answering with LangChain, Deep Lake, & OpenAI path: examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb date: 2023-09-30