diff --git a/crewai/fts/frontmatter.md b/crewai/fts/frontmatter.md deleted file mode 100644 index 0f64bf06..00000000 --- a/crewai/fts/frontmatter.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -# frontmatter -path: "/tutorial-crewai-couchbase-rag-with-fts" -title: Retrieval-Augmented Generation (RAG) with Couchbase and CrewAI using FTS Service -short_title: RAG with Couchbase and CrewAI using FTS -description: - - Learn how to build a semantic search engine using Couchbase and CrewAI. - - This tutorial demonstrates how to integrate Couchbase's vector search capabilities with CrewAI's agent-based approach. - - You'll understand how to perform Retrieval-Augmented Generation (RAG) using LangChain, CrewAI, and Couchbase. -content_type: tutorial -filter: sdk -technology: - - vector search -tags: - - FTS - - Artificial Intelligence - - LangChain - - CrewAI -sdk_language: - - python -length: 60 Mins ---- diff --git a/crewai/gsi/frontmatter.md b/crewai/gsi/frontmatter.md deleted file mode 100644 index 450d7f41..00000000 --- a/crewai/gsi/frontmatter.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -# frontmatter -path: "/tutorial-crewai-couchbase-rag-with-global-secondary-index" -title: Retrieval-Augmented Generation (RAG) with Couchbase and CrewAI with GSI -short_title: RAG with Couchbase and CrewAI with GSI -description: - - Learn how to build a semantic search engine using Couchbase and CrewAI. - - This tutorial demonstrates how to integrate Couchbase's vector search capabilities with CrewAI's agent-based approach. - - You'll understand how to perform Retrieval-Augmented Generation (RAG) using LangChain, CrewAI, and Couchbase with GSI. -content_type: tutorial -filter: sdk -technology: - - vector search -tags: - - GSI - - Artificial Intelligence - - LangChain - - CrewAI -sdk_language: - - python -length: 60 Mins ---- diff --git a/crewai/gsi/.env.sample b/crewai/query_based/.env.sample similarity index 100% rename from crewai/gsi/.env.sample rename to crewai/query_based/.env.sample diff --git a/crewai/gsi/RAG_with_Couchbase_and_CrewAI.ipynb b/crewai/query_based/RAG_with_Couchbase_and_CrewAI.ipynb similarity index 84% rename from crewai/gsi/RAG_with_Couchbase_and_CrewAI.ipynb rename to crewai/query_based/RAG_with_Couchbase_and_CrewAI.ipynb index ddd66fe4..5247102b 100644 --- a/crewai/gsi/RAG_with_Couchbase_and_CrewAI.ipynb +++ b/crewai/query_based/RAG_with_Couchbase_and_CrewAI.ipynb @@ -2,26 +2,11 @@ "cells": [ { "cell_type": "markdown", - "id": "82d610e0", + "id": "e16d35e8", "metadata": {}, "source": [ - "# Agent-Based RAG with Couchbase GSI Vector Search and CrewAI" - ] - }, - { - "cell_type": "markdown", - "id": "a3073978", - "metadata": {}, - "source": [ - "## Overview" - ] - }, - { - "cell_type": "markdown", - "id": "7e91202c", - "metadata": {}, - "source": [ - "In this guide, we will walk you through building a powerful semantic search engine using [Couchbase](https://www.couchbase.com) as the backend database and [CrewAI](https://github.com/crewAIInc/crewAI) for agent-based RAG operations. CrewAI allows us to create specialized agents that can work together to handle different aspects of the RAG workflow, from document retrieval to response generation. This tutorial uses Couchbase's **Global Secondary Index (GSI)** vector search capabilities, which offer high-performance vector search optimized for large-scale applications. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system from scratch. Alternatively if you want to perform semantic search using the FTS index, please take a look at [this.](https://developer.couchbase.com/tutorial-crewai-couchbase-rag-with-fts/)" + "## Introduction\n", + "In this guide, we will walk you through building a powerful semantic search engine using [Couchbase](https://www.couchbase.com) as the backend database and [CrewAI](https://github.com/crewAIInc/crewAI) for agent-based RAG operations. CrewAI allows us to create specialized agents that can work together to handle different aspects of the RAG workflow, from document retrieval to response generation. This tutorial uses Couchbase's **Hyperscale or Composite Index** vector search capabilities, which offer high-performance vector search optimized for large-scale applications. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system from scratch. Alternatively if you want to perform semantic search using the **Search Vector Index**, please take a look at [this.](https://developer.couchbase.com/tutorial-crewai-couchbase-rag-with-search-vector-index/)" ] }, { @@ -37,7 +22,7 @@ "id": "4e84bba4", "metadata": {}, "source": [ - "This tutorial is available as a Jupyter Notebook (.ipynb file) that you can run interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai/gsi/RAG_with_Couchbase_and_CrewAI.ipynb).\n", + "This tutorial is available as a Jupyter Notebook (.ipynb file) that you can run interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai/query_based/RAG_with_Couchbase_and_CrewAI.ipynb).\n", "\n", "You can either:\n", "- Download the notebook file and run it on [Google Colab](https://colab.research.google.com)\n", @@ -69,7 +54,7 @@ " - To get started with [Couchbase Capella](https://cloud.couchbase.com), create an account and use it to deploy a free tier operational cluster\n", " - This account provides you with an environment where you can explore and learn about Capella\n", " - To learn more, please follow the [Getting Started Guide](https://docs.couchbase.com/cloud/get-started/create-account.html)\n", - " - **Important**: This tutorial requires Couchbase Server **8.0+** for GSI vector search capabilities" + " - **Important**: This tutorial requires Couchbase Server **8.0+** for Hyperscale and Composite vector index capabilities" ] }, { @@ -113,12 +98,12 @@ "source": [ "We'll install the following key libraries:\n", "- `datasets`: For loading and managing our training data\n", - "- `langchain-couchbase`: To integrate Couchbase with LangChain for GSI vector storage and caching\n", + "- `langchain-couchbase`: To integrate Couchbase with LangChain for Hyperscale and Composite vector storage and caching\n", "- `langchain-openai`: For accessing OpenAI's embedding and chat models\n", "- `crewai`: To create and orchestrate our AI agents for RAG operations\n", "- `python-dotenv`: For securely managing environment variables and API keys\n", "\n", - "These libraries provide the foundation for building a semantic search engine with GSI vector embeddings, database integration, and agent-based RAG capabilities." + "These libraries provide the foundation for building a semantic search engine with Hyperscale and Composite vector embeddings, database integration, and agent-based RAG capabilities." ] }, { @@ -458,7 +443,7 @@ "id": "fa4faf3f", "metadata": {}, "source": [ - "## Understanding GSI Vector Search" + "## Understanding Hyperscale and Composite Vector Indexes" ] }, { @@ -466,7 +451,7 @@ "id": "69c7d28f", "metadata": {}, "source": [ - "### GSI Vector Index Configuration" + "### Hyperscale and Composite Vector Index Configuration" ] }, { @@ -474,7 +459,7 @@ "id": "90080454", "metadata": {}, "source": [ - "Semantic search with GSI requires creating a Global Secondary Index optimized for vector operations. Unlike FTS-based vector search, GSI vector indexes offer two distinct types optimized for different use cases:" + "Semantic search with Hyperscale and Composite Vector Indexes requires creating indexes optimized for vector operations. Unlike Search Vector Index-based vector search, Hyperscale and Composite vector indexes offer two distinct types optimized for different use cases. Learn more about these index types in the [Couchbase Vector Index Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html)." ] }, { @@ -482,7 +467,7 @@ "id": "72154198", "metadata": {}, "source": [ - "#### GSI Vector Index Types" + "#### Vector Index Types" ] }, { @@ -490,7 +475,7 @@ "id": "b55cb1f4", "metadata": {}, "source": [ - "##### Hyperscale Vector Indexes (BHIVE)" + "##### Hyperscale Vector Indexes" ] }, { @@ -558,12 +543,12 @@ "\n", "For detailed configuration options, see the [Quantization & Centroid Settings](https://docs.couchbase.com/cloud/vector-index/hyperscale-vector-index.html#algo_settings).\n", "\n", - "For more information on GSI vector indexes, see [Couchbase GSI Vector Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html)." + "For more information on Hyperscale and Composite vector indexes, see [Couchbase Vector Index Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html)." ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "id": "88e4c207", "metadata": {}, "outputs": [ @@ -571,24 +556,24 @@ "name": "stdout", "output_type": "stream", "text": [ - "GSI vector index configuration prepared\n" + "Hyperscale and Composite vector index configuration prepared\n" ] } ], "source": [ - "# GSI Vector Index Configuration\n", - "# Unlike FTS indexes, GSI vector indexes are created programmatically through the vector store\n", + "# Hyperscale and Composite Vector Index Configuration\n", + "# Unlike Search Vector Index, Hyperscale and Composite vector indexes are created programmatically through the vector store\n", "# We'll configure the parameters that will be used for index creation\n", "\n", "# Vector configuration\n", "DISTANCE_STRATEGY = DistanceStrategy.COSINE # Cosine similarity\n", - "INDEX_TYPE = IndexType.BHIVE # Using BHIVE for high-performance vector \n", + "INDEX_TYPE = IndexType.HYPERSCALE # Using HYPERSCALE for high-performance vector search\n", "INDEX_DESCRIPTION = \"IVF,SQ8\" # Auto-selected centroids with 8-bit scalar quantization\n", "\n", "# To create a Composite Index instead, use the following:\n", "# INDEX_TYPE = IndexType.COMPOSITE # Combines vector search with scalar filtering\n", "\n", - "print(\"GSI vector index configuration prepared\")" + "print(\"Hyperscale and Composite vector index configuration prepared\")" ] }, { @@ -608,7 +593,7 @@ "\n", "```python\n", "# Alternative configuration for Composite index\n", - "INDEX_TYPE = IndexType.COMPOSITE # Instead of IndexType.BHIVE\n", + "INDEX_TYPE = IndexType.COMPOSITE # Instead of IndexType.HYPERSCALE\n", "INDEX_DESCRIPTION = \"IVF,SQ8\" # Same quantization settings\n", "DISTANCE_STRATEGY = DistanceStrategy.COSINE # Same distance metric\n", "\n", @@ -703,7 +688,7 @@ "id": "7340c7ce", "metadata": {}, "source": [ - "### Create Couchbase GSI Vector Store" + "### Create Couchbase Hyperscale Vector Store" ] }, { @@ -711,12 +696,12 @@ "id": "2d202628", "metadata": {}, "source": [ - "Set up the GSI vector store where we'll store document embeddings for high-performance semantic search." + "Set up the Hyperscale vector store where we'll store document embeddings for high-performance semantic search." ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": null, "id": "a877a51d", "metadata": {}, "outputs": [ @@ -724,19 +709,19 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-10-06 10:18:05 [INFO] GSI Vector store setup completed\n" + "2025-10-06 10:18:05 [INFO] Hyperscale Vector store setup completed\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "GSI Vector store initialized successfully\n" + "Hyperscale Vector store initialized successfully\n" ] } ], "source": [ - "# Setup GSI vector store with OpenAI embeddings\n", + "# Setup Hyperscale vector store with OpenAI embeddings\n", "try:\n", " vector_store = CouchbaseQueryVectorStore(\n", " cluster=cluster,\n", @@ -746,11 +731,11 @@ " embedding=embeddings,\n", " distance_metric=DISTANCE_STRATEGY\n", " )\n", - " print(\"GSI Vector store initialized successfully\")\n", - " logging.info(\"GSI Vector store setup completed\")\n", + " print(\"Hyperscale Vector store initialized successfully\")\n", + " logging.info(\"Hyperscale Vector store setup completed\")\n", "except Exception as e:\n", - " logging.error(f\"Failed to initialize GSI vector store: {str(e)}\")\n", - " raise RuntimeError(f\"GSI Vector store initialization failed: {str(e)}\")" + " logging.error(f\"Failed to initialize Hyperscale vector store: {str(e)}\")\n", + " raise RuntimeError(f\"Hyperscale Vector store initialization failed: {str(e)}\")" ] }, { @@ -912,13 +897,13 @@ "id": "51df07c1", "metadata": {}, "source": [ - "Now let's demonstrate the performance benefits of GSI optimization by testing pure vector search performance. We'll compare three optimization levels:\n", + "Now let's demonstrate the performance benefits of Hyperscale vector index optimization by testing pure vector search performance. We'll compare three optimization levels:\n", "\n", - "1. **Baseline Performance**: Vector search without GSI optimization\n", - "2. **GSI-Optimized Performance**: Same search with BHIVE GSI index\n", - "3. **Cache Benefits**: Show how caching can be applied on top of GSI for repeated queries\n", + "1. **Baseline Performance**: Vector search without Hyperscale vector index optimization\n", + "2. **Hyperscale-Optimized Performance**: Same search with Hyperscale vector index\n", + "3. **Cache Benefits**: Show how caching can be applied on top of Hyperscale vector index for repeated queries\n", "\n", - "**Important**: This testing focuses on pure vector search performance, isolating the GSI improvements from other workflow overhead." + "**Important**: This testing focuses on pure vector search performance, isolating the Hyperscale vector index improvements from other workflow overhead." ] }, { @@ -931,14 +916,14 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": null, "id": "c43f62fa", "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", - "# Create GSI vector retriever optimized for high-performance searches\n", + "# Create Hyperscale vector retriever optimized for high-performance searches\n", "retriever = vector_store.as_retriever(\n", " search_type=\"similarity\",\n", " search_kwargs={\"k\": 4} # Return top 4 most similar documents\n", @@ -976,7 +961,7 @@ "id": "f939b9e1", "metadata": {}, "source": [ - "### Test 1: Baseline Performance (No GSI Index)" + "### Test 1: Baseline Performance (No Hyperscale Vector Index)" ] }, { @@ -984,12 +969,12 @@ "id": "e20d10ad", "metadata": {}, "source": [ - "Test pure vector search performance without GSI optimization." + "Test pure vector search performance without Hyperscale vector index optimization." ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "id": "71ceaa56", "metadata": {}, "outputs": [ @@ -997,7 +982,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Testing baseline vector search performance without GSI optimization...\n", + "Testing baseline vector search performance without Hyperscale vector index optimization...\n", "\n", "[Baseline Search] Testing vector search performance\n", "[Baseline Search] Query: 'What are the latest developments in football transfers?'\n", @@ -1005,17 +990,17 @@ "[Baseline Search] Found 4 relevant documents\n", "[Baseline Search] Top result preview: The latest updates and analysis from the BBC.\n", "\n", - "Baseline vector search time (without GSI): 1.3999 seconds\n", + "Baseline vector search time (without Hyperscale vector index): 1.3999 seconds\n", "\n" ] } ], "source": [ - "# Test baseline vector search performance without GSI index\n", + "# Test baseline vector search performance without Hyperscale vector index\n", "test_query = \"What are the latest developments in football transfers?\"\n", - "print(\"Testing baseline vector search performance without GSI optimization...\")\n", + "print(\"Testing baseline vector search performance without Hyperscale vector index optimization...\")\n", "baseline_time = test_vector_search_performance(test_query, \"Baseline Search\")\n", - "print(f\"\\nBaseline vector search time (without GSI): {baseline_time:.4f} seconds\\n\")" + "print(f\"\\nBaseline vector search time (without Hyperscale vector index): {baseline_time:.4f} seconds\\n\")" ] }, { @@ -1023,7 +1008,7 @@ "id": "90d304e9", "metadata": {}, "source": [ - "### Create BHIVE GSI Index" + "### Create Hyperscale Vector Index" ] }, { @@ -1031,12 +1016,12 @@ "id": "6ae9cef0", "metadata": {}, "source": [ - "Now let's create a BHIVE GSI vector index to enable high-performance vector searches. The index creation is done programmatically through the vector store, which will optimize the index settings based on our data and requirements." + "Now let's create a Hyperscale vector index to enable high-performance vector searches. The index creation is done programmatically through the vector store, which will optimize the index settings based on our data and requirements." ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": null, "id": "389d1358", "metadata": {}, "outputs": [ @@ -1044,36 +1029,36 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating BHIVE GSI vector index...\n" + "Creating Hyperscale vector index...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "2025-10-06 10:20:15 [INFO] BHIVE index created with description 'IVF,SQ8'\n" + "2025-10-06 10:20:15 [INFO] Hyperscale index created with description 'IVF,SQ8'\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "GSI Vector index created successfully\n", + "Hyperscale Vector index created successfully\n", "Waiting for index to become available...\n" ] } ], "source": [ - "# Create GSI Vector Index for high-performance searches\n", - "print(\"Creating BHIVE GSI vector index...\")\n", + "# Create Hyperscale Vector Index for high-performance searches\n", + "print(\"Creating Hyperscale vector index...\")\n", "try:\n", - " # Create a BHIVE index optimized for pure vector searches\n", + " # Create a Hyperscale index optimized for pure vector searches\n", " vector_store.create_index(\n", - " index_type=INDEX_TYPE, # BHIVE index type\n", + " index_type=INDEX_TYPE, # Hyperscale index type\n", " index_description=INDEX_DESCRIPTION # IVF,SQ8 for optimized performance\n", " )\n", - " print(f\"GSI Vector index created successfully\")\n", - " logging.info(f\"BHIVE index created with description '{INDEX_DESCRIPTION}'\")\n", + " print(f\"Hyperscale Vector index created successfully\")\n", + " logging.info(f\"Hyperscale index created with description '{INDEX_DESCRIPTION}'\")\n", " \n", " # Wait a moment for index to be available\n", " print(\"Waiting for index to become available...\")\n", @@ -1082,11 +1067,11 @@ "except Exception as e:\n", " # Index might already exist, which is fine\n", " if \"already exists\" in str(e).lower():\n", - " print(f\"GSI Vector index already exists, proceeding...\")\n", + " print(f\"Hyperscale Vector index already exists, proceeding...\")\n", " logging.info(f\"Index already exists\")\n", " else:\n", - " logging.error(f\"Failed to create GSI index: {str(e)}\")\n", - " raise RuntimeError(f\"GSI index creation failed: {str(e)}\")" + " logging.error(f\"Failed to create Hyperscale vector index: {str(e)}\")\n", + " raise RuntimeError(f\"Hyperscale vector index creation failed: {str(e)}\")" ] }, { @@ -1094,7 +1079,7 @@ "id": "6b9e5763", "metadata": {}, "source": [ - "### Test 2: GSI-Optimized Performance" + "### Test 2: Hyperscale-Optimized Performance" ] }, { @@ -1102,12 +1087,12 @@ "id": "8388f41b", "metadata": {}, "source": [ - "Test the same vector search with BHIVE GSI optimization." + "Test the same vector search with Hyperscale vector index optimization." ] }, { "cell_type": "code", - "execution_count": 16, + "execution_count": null, "id": "b1b89f5b", "metadata": {}, "outputs": [ @@ -1115,22 +1100,22 @@ "name": "stdout", "output_type": "stream", "text": [ - "Testing vector search performance with BHIVE GSI optimization...\n", + "Testing vector search performance with Hyperscale vector index optimization...\n", "\n", - "[GSI-Optimized Search] Testing vector search performance\n", - "[GSI-Optimized Search] Query: 'What are the latest developments in football transfers?'\n", - "[GSI-Optimized Search] Vector search completed in 0.5885 seconds\n", - "[GSI-Optimized Search] Found 4 relevant documents\n", - "[GSI-Optimized Search] Top result preview: Four key areas for Everton's new owners to address\n", + "[Hyperscale-Optimized Search] Testing vector search performance\n", + "[Hyperscale-Optimized Search] Query: 'What are the latest developments in football transfers?'\n", + "[Hyperscale-Optimized Search] Vector search completed in 0.5885 seconds\n", + "[Hyperscale-Optimized Search] Found 4 relevant documents\n", + "[Hyperscale-Optimized Search] Top result preview: Four key areas for Everton's new owners to address\n", "\n", "Everton fans last saw silverware in 1995 when th...\n" ] } ], "source": [ - "# Test vector search performance with GSI index\n", - "print(\"Testing vector search performance with BHIVE GSI optimization...\")\n", - "gsi_search_time = test_vector_search_performance(test_query, \"GSI-Optimized Search\")" + "# Test vector search performance with Hyperscale vector index\n", + "print(\"Testing vector search performance with Hyperscale vector index optimization...\")\n", + "hyperscale_search_time = test_vector_search_performance(test_query, \"Hyperscale-Optimized Search\")" ] }, { @@ -1146,7 +1131,7 @@ "id": "1cc73249", "metadata": {}, "source": [ - "Now let's demonstrate how caching can improve performance for repeated queries. **Note**: Caching benefits apply to both baseline and GSI-optimized searches." + "Now let's demonstrate how caching can improve performance for repeated queries. **Note**: Caching benefits apply to both baseline and Hyperscale-optimized searches." ] }, { @@ -1212,7 +1197,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": null, "id": "388ca617", "metadata": {}, "outputs": [ @@ -1224,8 +1209,8 @@ "================================================================================\n", "VECTOR SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n", "================================================================================\n", - "Phase 1 - Baseline Search (No GSI): 1.3999 seconds\n", - "Phase 2 - GSI-Optimized Search: 0.5885 seconds\n", + "Phase 1 - Baseline Search (No Hyperscale): 1.3999 seconds\n", + "Phase 2 - Hyperscale-Optimized Search: 0.5885 seconds\n", "Phase 3 - Cache Benefits:\n", " First execution (cache miss): 0.6450 seconds\n", " Second execution (cache hit): 0.4306 seconds\n", @@ -1233,14 +1218,14 @@ "--------------------------------------------------------------------------------\n", "VECTOR SEARCH OPTIMIZATION IMPACT:\n", "--------------------------------------------------------------------------------\n", - "GSI Index Benefit: 2.38x faster (58.0% improvement)\n", + "Hyperscale Index Benefit: 2.38x faster (58.0% improvement)\n", "Cache Benefit: 1.50x faster (33.2% improvement)\n", "\n", "Key Insights for Vector Search Performance:\n", - "• GSI BHIVE indexes provide significant performance improvements for vector similarity search\n", + "• Hyperscale indexes provide significant performance improvements for vector similarity search\n", "• Performance gains are most dramatic for complex semantic queries\n", - "• BHIVE optimization is particularly effective for high-dimensional embeddings\n", - "• Combined with proper quantization (SQ8), GSI delivers production-ready performance\n", + "• Hyperscale optimization is particularly effective for high-dimensional embeddings\n", + "• Combined with proper quantization (SQ8), Hyperscale delivers production-ready performance\n", "• These performance improvements directly benefit any application using the vector store\n" ] } @@ -1250,8 +1235,8 @@ "print(\"VECTOR SEARCH PERFORMANCE OPTIMIZATION SUMMARY\")\n", "print(\"=\"*80)\n", "\n", - "print(f\"Phase 1 - Baseline Search (No GSI): {baseline_time:.4f} seconds\")\n", - "print(f\"Phase 2 - GSI-Optimized Search: {gsi_search_time:.4f} seconds\")\n", + "print(f\"Phase 1 - Baseline Search (No Hyperscale): {baseline_time:.4f} seconds\")\n", + "print(f\"Phase 2 - Hyperscale-Optimized Search: {hyperscale_search_time:.4f} seconds\")\n", "if cache_time_1 and cache_time_2:\n", " print(f\"Phase 3 - Cache Benefits:\")\n", " print(f\" First execution (cache miss): {cache_time_1:.4f} seconds\")\n", @@ -1261,12 +1246,12 @@ "print(\"VECTOR SEARCH OPTIMIZATION IMPACT:\")\n", "print(\"-\"*80)\n", "\n", - "# GSI improvement analysis\n", - "if baseline_time and gsi_search_time:\n", - " speedup = baseline_time / gsi_search_time if gsi_search_time > 0 else float('inf')\n", - " time_saved = baseline_time - gsi_search_time\n", + "# Hyperscale improvement analysis\n", + "if baseline_time and hyperscale_search_time:\n", + " speedup = baseline_time / hyperscale_search_time if hyperscale_search_time > 0 else float('inf')\n", + " time_saved = baseline_time - hyperscale_search_time\n", " percent_improvement = (time_saved / baseline_time) * 100\n", - " print(f\"GSI Index Benefit: {speedup:.2f}x faster ({percent_improvement:.1f}% improvement)\")\n", + " print(f\"Hyperscale Index Benefit: {speedup:.2f}x faster ({percent_improvement:.1f}% improvement)\")\n", "\n", "# Cache improvement analysis\n", "if cache_time_1 and cache_time_2 and cache_time_2 < cache_time_1:\n", @@ -1277,10 +1262,10 @@ " print(f\"Cache Benefit: Variable (depends on query complexity and caching mechanism)\")\n", "\n", "print(f\"\\nKey Insights for Vector Search Performance:\")\n", - "print(f\"• GSI BHIVE indexes provide significant performance improvements for vector similarity search\")\n", + "print(f\"• Hyperscale indexes provide significant performance improvements for vector similarity search\")\n", "print(f\"• Performance gains are most dramatic for complex semantic queries\")\n", - "print(f\"• BHIVE optimization is particularly effective for high-dimensional embeddings\")\n", - "print(f\"• Combined with proper quantization (SQ8), GSI delivers production-ready performance\")\n", + "print(f\"• Hyperscale optimization is particularly effective for high-dimensional embeddings\")\n", + "print(f\"• Combined with proper quantization (SQ8), Hyperscale vector indexes deliver production-ready performance\")\n", "print(f\"• These performance improvements directly benefit any application using the vector store\")" ] }, @@ -1311,7 +1296,7 @@ "- **Writer Agent**: Takes research findings and creates polished, structured responses\n", "- **Collaborative Workflow**: Agents work together, with the writer building on the researcher's findings\n", "\n", - "This multi-agent approach produces higher-quality responses than single-agent systems by separating research and writing expertise, while benefiting from the GSI performance improvements we just demonstrated." + "This multi-agent approach produces higher-quality responses than single-agent systems by separating research and writing expertise, while benefiting from the Hyperscale vector index performance improvements we just demonstrated." ] }, { @@ -1324,20 +1309,20 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": null, "id": "c7b379d0", "metadata": {}, "outputs": [], "source": [ - "# Define the GSI vector search tool using the @tool decorator\n", - "@tool(\"gsi_vector_search\")\n", + "# Define the Hyperscale vector search tool using the @tool decorator\n", + "@tool(\"hyperscale_vector_search\")\n", "def search_tool(query: str) -> str:\n", - " \"\"\"Search for relevant documents using GSI vector similarity.\n", + " \"\"\"Search for relevant documents using Hyperscale vector similarity.\n", " Input should be a simple text query string.\n", - " Returns a list of relevant document contents from GSI vector search.\n", - " Use this tool to find detailed information about topics using high-performance GSI indexes.\"\"\"\n", + " Returns a list of relevant document contents from Hyperscale vector search.\n", + " Use this tool to find detailed information about topics using high-performance Hyperscale indexes.\"\"\"\n", " \n", - " # Invoke the GSI vector retriever (now optimized with BHIVE index)\n", + " # Invoke the Hyperscale vector retriever (now optimized with HYPERSCALE index)\n", " docs = retriever.invoke(query)\n", "\n", " # Format the results with distance information\n", @@ -1358,7 +1343,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": null, "id": "73c44437", "metadata": {}, "outputs": [ @@ -1366,7 +1351,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "CrewAI agents created successfully with optimized GSI vector search\n" + "CrewAI agents created successfully with optimized Hyperscale vector search\n" ] } ], @@ -1403,7 +1388,7 @@ " allow_delegation=False\n", ")\n", "\n", - "print(\"CrewAI agents created successfully with optimized GSI vector search\")" + "print(\"CrewAI agents created successfully with optimized Hyperscale vector search\")" ] }, { @@ -1421,7 +1406,7 @@ "source": [ "The complete optimized RAG process:\n", "1. **User Query** → Research Agent\n", - "2. **Vector Search** → GSI BHIVE index finds similar documents (now with proven performance improvements)\n", + "2. **Vector Search** → Hyperscale index finds similar documents (now with proven performance improvements)\n", "3. **Document Analysis** → Research Agent analyzes and synthesizes findings\n", "4. **Response Writing** → Writer Agent creates polished, structured response\n", "5. **Final Output** → User receives comprehensive, well-formatted answer\n", @@ -1442,7 +1427,7 @@ "id": "3e7d956a", "metadata": {}, "source": [ - "Now let's demonstrate the complete optimized agent-based RAG system in action, benefiting from the GSI performance improvements we validated earlier." + "Now let's demonstrate the complete optimized agent-based RAG system in action, benefiting from the Hyperscale vector index performance improvements we validated earlier." ] }, { @@ -1455,13 +1440,13 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": null, "id": "2176b29d", "metadata": {}, "outputs": [], "source": [ "def process_interactive_query(query, researcher, writer):\n", - " \"\"\"Run complete RAG workflow with CrewAI agents using optimized GSI vector search\"\"\"\n", + " \"\"\"Run complete RAG workflow with CrewAI agents using optimized Hyperscale vector search\"\"\"\n", " print(f\"\\nProcessing Query: {query}\")\n", " print(\"=\" * 80)\n", " \n", @@ -1545,7 +1530,7 @@ "id": "82ad950f", "metadata": {}, "source": [ - "You have successfully built a powerful agent-based RAG system that combines Couchbase's high-performance GSI vector storage capabilities with CrewAI's multi-agent architecture. This tutorial demonstrated the complete pipeline from data ingestion to intelligent response generation, with real performance benchmarks showing the dramatic improvements GSI indexing provides." + "You have successfully built a powerful agent-based RAG system that combines Couchbase's high-performance Hyperscale and Composite vector storage capabilities with CrewAI's multi-agent architecture. This tutorial demonstrated the complete pipeline from data ingestion to intelligent response generation, with real performance benchmarks showing the dramatic improvements Hyperscale vector indexing provides." ] } ], diff --git a/crewai/query_based/frontmatter.md b/crewai/query_based/frontmatter.md new file mode 100644 index 00000000..139299ef --- /dev/null +++ b/crewai/query_based/frontmatter.md @@ -0,0 +1,25 @@ +--- +# frontmatter +path: "/tutorial-crewai-couchbase-rag-with-hyperscale-or-composite-vector-index" +title: RAG with CrewAI using Couchbase Hyperscale and Composite Vector Index +short_title: RAG with CrewAI using Couchbase Hyperscale and Composite Vector Index +description: + - Learn to build a semantic search engine using [Couchbase](https://www.couchbase.com) and agent-based RAG workflows powered by [CrewAI](https://github.com/crewAIInc/crewAI). + - Explore Couchbase Hyperscale and Composite Vector Indexes for high-performance vector search, including pure vector and filtered similarity queries. + - Follow beginner-friendly, step-by-step instructions to build a fully functional semantic search system from scratch. + +content_type: tutorial +filter: sdk +technology: + - vector search +tags: + - Hyperscale Vector Index + - Composite Vector Index + - Artificial Intelligence + - LangChain + - CrewAI +sdk_language: + - python +length: 60 Mins +alt_paths: ["/tutorial-crewai-couchbase-rag-with-hyperscale-vector-index", "/tutorial-crewai-couchbase-rag-with-composite-vector-index"] +--- diff --git a/crewai/fts/.env.sample b/crewai/search_based/.env.sample similarity index 100% rename from crewai/fts/.env.sample rename to crewai/search_based/.env.sample diff --git a/crewai/fts/RAG_with_Couchbase_and_CrewAI.ipynb b/crewai/search_based/RAG_with_Couchbase_and_CrewAI.ipynb similarity index 93% rename from crewai/fts/RAG_with_Couchbase_and_CrewAI.ipynb rename to crewai/search_based/RAG_with_Couchbase_and_CrewAI.ipynb index ad9c4e61..824d7f0e 100644 --- a/crewai/fts/RAG_with_Couchbase_and_CrewAI.ipynb +++ b/crewai/search_based/RAG_with_Couchbase_and_CrewAI.ipynb @@ -4,21 +4,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Introduction\n", - "\n", - "In this guide, we will walk you through building a powerful semantic search engine using [Couchbase](https://www.couchbase.com) as the backend database and [CrewAI](https://github.com/crewAIInc/crewAI) for agent-based RAG operations. CrewAI allows us to create specialized agents that can work together to handle different aspects of the RAG workflow, from document retrieval to response generation. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system from scratch. Alternatively if you want to perform semantic search using the GSI index, please take a look at [this.](https://developer.couchbase.com/tutorial-crewai-couchbase-rag-with-global-secondary-index)\n", + "## Introduction\n", + "In this guide, we will walk you through building a powerful semantic search engine using [Couchbase](https://www.couchbase.com) as the backend database and [CrewAI](https://github.com/crewAIInc/crewAI) for agent-based RAG operations. CrewAI allows us to create specialized agents that can work together to handle different aspects of the RAG workflow, from document retrieval to response generation. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system from scratch. Alternatively if you want to perform semantic search using the **Hyperscale or Composite Vector Index**, please take a look at [this.](https://developer.couchbase.com/tutorial-crewai-couchbase-rag-with-hyperscale-or-composite-vector-index)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## How to Run This Tutorial\n", "\n", - "How to run this tutorial\n", - "----------------------\n", "This tutorial is available as a Jupyter Notebook (.ipynb file) that you can run \n", - "interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai/fts/RAG_with_Couchbase_and_CrewAI.ipynb).\n", + "interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/crewai/search_based/RAG_with_Couchbase_and_CrewAI.ipynb).\n", "\n", "You can either:\n", "- Download the notebook file and run it on [Google Colab](https://colab.research.google.com)\n", "- Run it on your system by setting up the Python environment\n", "\n", - "Before you start\n", - "---------------\n", + "## Prerequisites\n", + "\n", + "### Couchbase Requirements\n", "\n", "1. Create and Deploy Your Free Tier Operational cluster on [Capella](https://cloud.couchbase.com/sign-up)\n", " - To get started with [Couchbase Capella](https://cloud.couchbase.com), create an account and use it to deploy \n", @@ -27,8 +32,9 @@ " about Capella with no time constraint\n", " - To learn more, please follow the [Getting Started Guide](https://docs.couchbase.com/cloud/get-started/create-account.html)\n", "\n", - "2. Couchbase Capella Configuration\n", - " When running Couchbase using Capella, the following prerequisites need to be met:\n", + "### Couchbase Capella Configuration\n", + "\n", + "When running Couchbase using Capella, the following prerequisites need to be met:\n", " - Create the database credentials to access the required bucket (Read and Write) used in the application\n", " - Allow access to the Cluster from the IP on which the application is running by following the [Network Security documentation](https://docs.couchbase.com/cloud/security/security.html#public-access)" ] @@ -37,16 +43,18 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Setting the Stage: Installing Necessary Libraries\n", + "## Setup and Installation\n", + "\n", + "### Installing Necessary Libraries\n", "\n", "We'll install the following key libraries:\n", "- `datasets`: For loading and managing our training data\n", - "- `langchain-couchbase`: To integrate Couchbase with LangChain for vector storage and caching\n", + "- `langchain-couchbase`: To integrate Couchbase with LangChain for Search Vector Index storage and caching\n", "- `langchain-openai`: For accessing OpenAI's embedding and chat models\n", "- `crewai`: To create and orchestrate our AI agents for RAG operations\n", "- `python-dotenv`: For securely managing environment variables and API keys\n", "\n", - "These libraries provide the foundation for building a semantic search engine with vector embeddings, \n", + "These libraries provide the foundation for building a semantic search engine with Search Vector Index embeddings, \n", "database integration, and agent-based RAG capabilities." ] }, @@ -71,7 +79,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Importing Necessary Libraries\n", + "### Import Required Modules\n", + "\n", "The script starts by importing a series of libraries required for various tasks, including handling JSON, logging, time tracking, Couchbase connections, embedding generation, and dataset loading." ] }, @@ -110,7 +119,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Setup Logging\n", + "### Configure Logging\n", + "\n", "Logging is configured to track the progress of the script and capture any errors or warnings." ] }, @@ -134,7 +144,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Loading Sensitive Information\n", + "### Load Environment Configuration\n", + "\n", "In this section, we prompt the user to input essential configuration settings needed. These settings include sensitive information like database credentials, and specific configuration names. Instead of hardcoding these details into the script, we request the user to provide them at runtime, ensuring flexibility and security.\n", "\n", "The script uses environment variables to store sensitive information, enhancing the overall security and maintainability of your code by avoiding hardcoded values." @@ -177,7 +188,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Connecting to the Couchbase Cluster\n", + "## Couchbase Connection Setup\n", + "\n", + "### Connect to Cluster\n", + "\n", "Connecting to a Couchbase cluster is the foundation of our project. Couchbase will serve as our primary data store, handling all the storage and retrieval operations required for our semantic search engine. By establishing this connection, we enable our application to interact with the database, allowing us to perform operations such as storing embeddings, querying data, and managing collections. This connection is the gateway through which all data will flow, so ensuring it's set up correctly is paramount." ] }, @@ -211,13 +225,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Verifying Search Service Availability\n", - " In this section, we verify that the Couchbase Search (FTS) service is available and responding correctly. This is a crucial check because our vector search functionality depends on it. If any issues are detected with the Search service, the function will raise an exception, allowing us to catch and handle problems early before attempting vector operations.\n" + "### Verify Search Service Availability\n", + "\n", + "In this section, we verify that the Couchbase Search service is available and responding correctly. This is a crucial check because our Search Vector Index functionality depends on it. If any issues are detected with the Search service, the function will raise an exception, allowing us to catch and handle problems early before attempting vector operations." ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -248,7 +263,7 @@ " break\n", "\n", " if not search_available:\n", - " raise RuntimeError(\"Search/FTS service not found or not responding\")\n", + " raise RuntimeError(\"Search service not found or not responding\")\n", " \n", " print(\"Search service check passed successfully\")\n", " except Exception as e:\n", @@ -265,32 +280,30 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Setting Up Collections in Couchbase\n", + "### Setup Collections\n", "\n", - "The setup_collection() function handles creating and configuring the hierarchical data organization in Couchbase:\n", + "Create and configure Couchbase bucket, scope, and collection for storing our vector data.\n", "\n", - "1. Bucket Creation:\n", + "1. **Bucket Creation:**\n", " - Checks if specified bucket exists, creates it if not\n", " - Sets bucket properties like RAM quota (1024MB) and replication (disabled)\n", - " - Note: If you are using Capella, create a bucket manually called vector-search-testing(or any name you prefer) with the same properties.\n", + " - Note: If you are using Capella, create a bucket manually called vector-search-testing (or any name you prefer) with the same properties.\n", "\n", - "2. Scope Management: \n", + "2. **Scope Management:** \n", " - Verifies if requested scope exists within bucket\n", " - Creates new scope if needed (unless it's the default \"_default\" scope)\n", "\n", - "3. Collection Setup:\n", + "3. **Collection Setup:**\n", " - Checks for collection existence within scope\n", " - Creates collection if it doesn't exist\n", " - Waits 2 seconds for collection to be ready\n", "\n", - "Additional Tasks:\n", + "**Additional Tasks:**\n", "- Creates primary index on collection for query performance\n", "- Clears any existing documents for clean state\n", "- Implements comprehensive error handling and logging\n", "\n", - "The function is called twice to set up:\n", - "1. Main collection for vector embeddings\n", - "2. Cache collection for storing results\n" + "The function is called to set up the main collection for vector embeddings." ] }, { @@ -398,13 +411,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Configuring and Initializing Couchbase Vector Search Index for Semantic Document Retrieval\n", + "## Understanding Search Vector Index\n", "\n", - "Semantic search requires an efficient way to retrieve relevant documents based on a user's query. This is where the Couchbase Vector Search Index comes into play. In this step, we load the Vector Search Index definition from a JSON file, which specifies how the index should be structured. This includes the fields to be indexed, the dimensions of the vectors, and other parameters that determine how the search engine processes queries based on vector similarity.\n", + "### Configuring and Initializing Couchbase Search Vector Index for Semantic Document Retrieval\n", "\n", - "This CrewAI vector search index configuration requires specific default settings to function properly. This tutorial uses the bucket named `vector-search-testing` with the scope `shared` and collection `crew`. The configuration is set up for vectors with exactly `1536 dimensions`, using `dot product` similarity and optimized for `recall`. If you want to use a different bucket, scope, or collection, you will need to modify the index configuration accordingly.\n", + "Semantic search requires an efficient way to retrieve relevant documents based on a user's query. This is where the Couchbase Search Vector Index comes into play. In this step, we load the Search Vector Index definition from a JSON file, which specifies how the index should be structured. This includes the fields to be indexed, the dimensions of the vectors, and other parameters that determine how the search engine processes queries based on vector similarity.\n", "\n", - "For more information on creating a vector search index, please follow the instructions at [Couchbase Vector Search Documentation](https://docs.couchbase.com/cloud/vector-search/create-vector-search-index-ui.html)." + "This CrewAI Search Vector Index configuration requires specific default settings to function properly. This tutorial uses the bucket named `vector-search-testing` with the scope `shared` and collection `crew`. The configuration is set up for vectors with exactly `1536 dimensions`, using `dot product` similarity and optimized for `recall`. If you want to use a different bucket, scope, or collection, you will need to modify the index configuration accordingly.\n", + "\n", + "For more information on creating a Search Vector Index, please follow the instructions at [Couchbase Vector Search Documentation](https://docs.couchbase.com/cloud/vector-search/create-vector-search-index-ui.html)." ] }, { @@ -432,9 +447,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Creating or Updating Search Indexes\n", + "### Creating or Updating Search Vector Index\n", "\n", - "With the index definition loaded, the next step is to create or update the **Vector Search Index** in Couchbase. This step is crucial because it optimizes our database for vector similarity search operations, allowing us to perform searches based on the semantic content of documents rather than just keywords. By creating or updating a Vector Search Index, we enable our search engine to handle complex queries that involve finding semantically similar documents using vector embeddings, which is essential for a robust semantic search engine." + "With the index definition loaded, the next step is to create or update the **Search Vector Index** in Couchbase. This step is crucial because it optimizes our database for vector similarity search operations, allowing us to perform searches based on the semantic content of documents rather than just keywords. By creating or updating a Search Vector Index, we enable our search engine to handle complex queries that involve finding semantically similar documents using vector embeddings, which is essential for a robust semantic search engine." ] }, { @@ -484,17 +499,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Setting Up OpenAI Components\n", + "## OpenAI Configuration\n", "\n", "This section initializes two key OpenAI components needed for our RAG system:\n", "\n", - "1. OpenAI Embeddings:\n", + "1. **OpenAI Embeddings:**\n", " - Uses the 'text-embedding-3-small' model\n", " - Converts text into high-dimensional vector representations (embeddings)\n", " - These embeddings enable semantic search by capturing the meaning of text\n", " - Required for vector similarity search in Couchbase\n", "\n", - "2. ChatOpenAI Language Model:\n", + "2. **ChatOpenAI Language Model:**\n", " - Uses the 'gpt-4o' model\n", " - Temperature set to 0.2 for balanced creativity and focus\n", " - Serves as the cognitive engine for CrewAI agents\n", @@ -504,13 +519,10 @@ " - Generate thoughtful responses based on that context\n", " - Follow instructions defined in agent roles and goals\n", " - Collaborate with other agents in the crew\n", - " - The relatively low temperature (0.2) ensures agents produce reliable,\n", - " consistent outputs while maintaining some creative problem-solving ability\n", + " - The relatively low temperature (0.2) ensures agents produce reliable, consistent outputs while maintaining some creative problem-solving ability\n", "\n", "Both components require a valid OpenAI API key (OPENAI_API_KEY) for authentication.\n", - "In the CrewAI framework, the LLM acts as the \"brain\" for each agent, allowing them\n", - "to interpret tasks, retrieve relevant information via the RAG system, and generate\n", - "appropriate outputs based on their specialized roles and expertise." + "In the CrewAI framework, the LLM acts as the \"brain\" for each agent, allowing them to interpret tasks, retrieve relevant information via the RAG system, and generate appropriate outputs based on their specialized roles and expertise." ] }, { @@ -546,8 +558,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Setting Up the Couchbase Vector Store\n", - "A vector store is where we'll keep our embeddings. Unlike the FTS index, which is used for text-based search, the vector store is specifically designed to handle embeddings and perform similarity searches. When a user inputs a query, the search engine converts the query into an embedding and compares it against the embeddings stored in the vector store. This allows the engine to find documents that are semantically similar to the query, even if they don't contain the exact same words. By setting up the vector store in Couchbase, we create a powerful tool that enables our search engine to understand and retrieve information based on the meaning and context of the query, rather than just the specific words used." + "## Document Processing and Vector Store Setup\n", + "\n", + "### Create Couchbase Search Vector Store\n", + "\n", + "A vector store is where we'll keep our embeddings. Unlike traditional text-based search, the Search Vector Store is specifically designed to handle embeddings and perform similarity searches. When a user inputs a query, the search engine converts the query into an embedding and compares it against the embeddings stored in the vector store. This allows the engine to find documents that are semantically similar to the query, even if they don't contain the exact same words. By setting up the Search Vector Store in Couchbase, we create a powerful tool that enables our search engine to understand and retrieve information based on the meaning and context of the query, rather than just the specific words used." ] }, { @@ -580,7 +595,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Load the BBC News Dataset\n", + "### Load BBC News Dataset\n", + "\n", "To build a search engine, we need data to search through. We use the BBC News dataset from RealTimeData, which provides real-world news articles. This dataset contains news articles from BBC covering various topics and time periods. Loading the dataset is a crucial step because it provides the raw material that our search engine will work with. The quality and diversity of the news articles make it an excellent choice for testing and refining our search engine, ensuring it can handle real-world news content effectively.\n", "\n", "The BBC News dataset allows us to work with authentic news articles, enabling us to build and test a search engine that can effectively process and retrieve relevant news content. The dataset is loaded using the Hugging Face datasets library, specifically accessing the \"RealTimeData/bbc_news_alltime\" dataset with the \"2024-12\" version." @@ -621,7 +637,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Cleaning up the Data\n", + "#### Data Cleaning\n", + "\n", "We will use the content of the news articles for our RAG system.\n", "\n", "The dataset contains a few duplicate records. We are removing them to avoid duplicate results in the retrieval stage of our RAG system." @@ -654,25 +671,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Saving Data to the Vector Store\n", + "#### Save Data to Vector Store\n", + "\n", "To efficiently handle the large number of articles, we process them in batches of articles at a time. This batch processing approach helps manage memory usage and provides better control over the ingestion process.\n", "\n", "We first filter out any articles that exceed 50,000 characters to avoid potential issues with token limits. Then, using the vector store's add_texts method, we add the filtered articles to our vector database. The batch_size parameter controls how many articles are processed in each iteration.\n", "\n", "This approach offers several benefits:\n", - "1. Memory Efficiency: Processing in smaller batches prevents memory overload\n", - "2. Error Handling: If an error occurs, only the current batch is affected\n", - "3. Progress Tracking: Easier to monitor and track the ingestion progress\n", - "4. Resource Management: Better control over CPU and network resource utilization\n", - "\n", - "We use a conservative batch size of 50 to ensure reliable operation.\n", - "The optimal batch size depends on many factors including:\n", - "- Document sizes being inserted\n", - "- Available system resources\n", - "- Network conditions\n", - "- Concurrent workload\n", - "\n", - "Consider measuring performance with your specific workload before adjusting.\n" + "1. **Memory Efficiency**: Processing in smaller batches prevents memory overload\n", + "2. **Error Handling**: If an error occurs, only the current batch is affected\n", + "3. **Progress Tracking**: Easier to monitor and track the ingestion progress\n", + "4. **Resource Management**: Better control over CPU and network resource utilization\n", + "\n", + "We use a conservative batch size of 50 to ensure reliable operation. The optimal batch size depends on many factors including document sizes, available system resources, network conditions, and concurrent workload." ] }, { @@ -708,20 +719,23 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Creating a Vector Search Tool\n", + "## CrewAI Agent Setup\n", + "\n", + "### Create Vector Search Tool\n", + "\n", "After loading our data into the vector store, we need to create a tool that can efficiently search through these vector embeddings. This involves two key components:\n", "\n", - "### Vector Retriever\n", + "**Vector Retriever:**\n", "The vector retriever is configured to perform similarity searches. This creates a retriever that performs semantic similarity searches against our vector database. The similarity search finds documents whose vector embeddings are closest to the query's embedding in the vector space.\n", "\n", - "### Search Tool\n", + "**Search Tool:**\n", "The search tool wraps the retriever in a user-friendly interface that:\n", "- Takes a query string as input\n", "- Passes the query to the retriever to find relevant documents\n", "- Formats the results with clear document separation using document numbers and dividers\n", "- Returns the formatted results as a single string with each document clearly delineated\n", "\n", - "The tool is designed to integrate seamlessly with our AI agents, providing them with reliable access to our knowledge base through vector similarity search. The lambda function in the tool handles both direct string queries and structured query objects, ensuring flexibility in how the tool can be invoked.\n" + "The tool is designed to integrate seamlessly with our AI agents, providing them with reliable access to our knowledge base through vector similarity search." ] }, { @@ -762,19 +776,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Creating CrewAI Agents\n", + "### Create CrewAI Agents\n", "\n", "We'll create two specialized AI agents using the CrewAI framework to handle different aspects of our information retrieval and analysis system:\n", "\n", - "## Research Expert Agent\n", + "**Research Expert Agent:**\n", "This agent is designed to:\n", - "- Execute semantic searches using our vector store\n", + "- Execute semantic searches using our Search Vector Store\n", "- Analyze and evaluate search results \n", "- Identify key information and insights\n", "- Verify facts across multiple sources\n", "- Synthesize findings into comprehensive research summaries\n", "\n", - "## Technical Writer Agent \n", + "**Technical Writer Agent:**\n", "This agent is responsible for:\n", "- Taking research findings and structuring them logically\n", "- Converting technical concepts into clear explanations\n", @@ -782,6 +796,7 @@ "- Maintaining engaging yet informative tone\n", "- Producing well-formatted final outputs\n", "\n", + "**Agent Workflow:**\n", "The agents work together in a coordinated way:\n", "1. Research agent finds and analyzes relevant documents\n", "2. Writer agent takes those findings and crafts polished responses\n", @@ -869,15 +884,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## How CrewAI Agents Work in this RAG System\n", - "\n", - "### Agent-Based RAG Architecture\n", + "### How the RAG Workflow Works\n", "\n", "This system uses a two-agent approach to implement Retrieval-Augmented Generation (RAG):\n", "\n", "1. **Research Expert Agent**:\n", " - Receives the user query\n", - " - Uses the vector search tool to retrieve relevant documents from Couchbase\n", + " - Uses the vector search tool to retrieve relevant documents from Couchbase Search Vector Store\n", " - Analyzes and synthesizes information from retrieved documents\n", " - Produces a comprehensive research summary with key findings\n", "\n", @@ -887,41 +900,42 @@ " - Creates a polished, user-friendly response\n", " - Ensures proper attribution and citation\n", "\n", - "#### How the Process Works:\n", + "**The Complete RAG Process:**\n", "\n", "1. **Query Processing**: User query is passed to the Research Agent\n", "2. **Vector Search**: Query is converted to embeddings and matched against document vectors\n", - "3. **Document Retrieval**: Most similar documents are retrieved from Couchbase\n", + "3. **Document Retrieval**: Most similar documents are retrieved from Couchbase Search Vector Store\n", "4. **Analysis**: Research Agent analyzes documents for relevance and extracts key information\n", "5. **Synthesis**: Research Agent combines findings into a coherent summary\n", "6. **Refinement**: Writer Agent restructures and enhances the content\n", "7. **Response Generation**: Final polished response is returned to the user\n", "\n", - "This multi-agent approach separates concerns (research vs. writing) and leverages\n", - "specialized expertise for each task, resulting in higher quality responses.\n" + "This multi-agent approach separates concerns (research vs. writing) and leverages specialized expertise for each task, resulting in higher quality responses." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# Testing the Search System\n", + "## CrewAI Agent Demo\n", "\n", - "Test the system with some example queries." + "### Demo Function\n", + "\n", + "Test the system with example queries." ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def process_query(query, researcher, writer):\n", " \"\"\"\n", - " Test the complete RAG system with a user query.\n", + " Run complete RAG workflow with CrewAI agents using Search Vector Store.\n", " \n", " This function tests both the vector search capability and the agent-based processing:\n", - " 1. Vector search: Retrieves relevant documents from Couchbase\n", + " 1. Vector search: Retrieves relevant documents from Couchbase Search Vector Store\n", " 2. Agent processing: Uses CrewAI agents to analyze and format the response\n", " \n", " The function measures performance and displays detailed outputs from each step.\n", @@ -1428,10 +1442,11 @@ "metadata": {}, "source": [ "## Conclusion\n", - "By following these steps, you've built a powerful RAG system that combines Couchbase's vector storage capabilities with CrewAI's agent-based architecture. This multi-agent approach separates research and writing concerns, resulting in higher quality responses to user queries.\n", + "\n", + "By following these steps, you've built a powerful RAG system that combines Couchbase's Search Vector Index storage capabilities with CrewAI's agent-based architecture. This multi-agent approach separates research and writing concerns, resulting in higher quality responses to user queries.\n", "\n", "The system demonstrates several key advantages:\n", - "1. Efficient vector search using Couchbase's vector store\n", + "1. Efficient vector search using Couchbase's Search Vector Index\n", "2. Specialized AI agents that focus on different aspects of the RAG pipeline\n", "3. Collaborative workflow between agents to produce comprehensive, well-structured responses\n", "4. Scalable architecture that can be extended with additional agents for more complex tasks\n", diff --git a/crewai/fts/crew_index.json b/crewai/search_based/crew_index.json similarity index 100% rename from crewai/fts/crew_index.json rename to crewai/search_based/crew_index.json diff --git a/crewai/search_based/frontmatter.md b/crewai/search_based/frontmatter.md new file mode 100644 index 00000000..9ba219cf --- /dev/null +++ b/crewai/search_based/frontmatter.md @@ -0,0 +1,22 @@ +--- +# frontmatter +path: "/tutorial-crewai-couchbase-rag-with-search-vector-index" +title: RAG with CrewAI and Couchbase Search Vector Index +short_title: RAG with CrewAI and Couchbase Search Vector Index +description: + - Learn to build a semantic search engine using [Couchbase](https://www.couchbase.com) and agent-based RAG workflows powered by [CrewAI](https://github.com/crewAIInc/crewAI). + - Use CrewAI to create specialized agents that collaborate across the RAG pipeline, from retrieving documents to generating responses. + - Follow beginner-friendly, step-by-step instructions to build a complete semantic search system from scratch. +content_type: tutorial +filter: sdk +technology: + - vector search +tags: + - Search Vector Index + - Artificial Intelligence + - LangChain + - CrewAI +sdk_language: + - python +length: 60 Mins +---