![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Chat with your webpage with `scrapegraph`, `burr` and `lancedb`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-burr-lancedb) [](https://colab.research.google.com/drive/1pbDpLKW9pbSIPrI5DbQQE2q_7eaDUAH9?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Chat with your webpage with `scrapegraph`, `burr` and `lancedb`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-burr-lancedb) [](https://colab.research.google.com/drive/1pbDpLKW9pbSIPrI5DbQQE2q_7eaDUAH9?usp=sharing)",
+ "text/markdown": "## 🕷️ Chat with your webpage with `scrapegraph`, `burr` and `lancedb`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-burr-lancedb) [](https://colab.research.google.com/drive/1pbDpLKW9pbSIPrI5DbQQE2q_7eaDUAH9?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Chat with your webpage with `scrapegraph`, `burr` and `lancedb`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:00:53.415Z",
+ "executionStartTime": "2026-03-26T00:00:53.413Z"
},
{
- "cell_type": "markdown",
- "source": [
- ""
- ],
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "dsJlQnfA60D0"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 1,
+ "source": "%%capture\n!pip install scrapegraph-py burr[start,opentelemetry] lancedb openai opentelemetry-instrumentation-openai opentelemetry-instrumentation-lancedb",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install scrapegraph-py burr[start,opentelemetry] lancedb openai opentelemetry-instrumentation-openai opentelemetry-instrumentation-lancedb"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 1,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph` and `OpenAI` API keys",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph` and `OpenAI` API keys"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "sffqFG2EJ8bI",
- "outputId": "bd2fbb27-980c-420f-ef50-fc1f23bdae0a"
- },
+ "source": "import getpass\nimport os\n\nif not os.environ.get(\"SGAI_API_KEY\"):\n os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n\nif not os.environ.get(\"OPENAI_API_KEY\"):\n os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"Scrapegraph API key:\n",
"··········\n",
"OpenAI API key:\n",
"··········\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "import getpass\n",
- "import os\n",
- "\n",
- "if not os.environ.get(\"SGAI_API_KEY\"):\n",
- " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n",
- "\n",
- "if not os.environ.get(\"OPENAI_API_KEY\"):\n",
- " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"
- ]
+ "metadata": {
+ "id": "sffqFG2EJ8bI",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "bd2fbb27-980c-420f-ef50-fc1f23bdae0a"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 2,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🚀 Define the extraction and query flow using `burr` and run it",
+ "outputs": [],
"metadata": {
"id": "cDGH0b2DkY63"
},
- "source": [
- "### 🚀 Define the extraction and query flow using `burr` and run it"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "`burr` is an open-source `orchestrator framework` that makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...). It also features a cool self-hosted UI to trace what's happening in the application.\n",
- "\n",
- "> Check the [Github Repo](https://github.com/DAGWorks-Inc/burr)"
- ],
+ "source": "`burr` is an open-source `orchestrator framework` that makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...). It also features a cool self-hosted UI to trace what's happening in the application.\n\n> Check the [Github Repo](https://github.com/DAGWorks-Inc/burr)",
+ "outputs": [],
"metadata": {
"id": "M3gmZCI0eqS1"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "Our goal is to define a flow (DAG) that:\n",
- "\n",
- "1. Fetches markdown from webpages ([scrapegraph](https://scrapegraphai.com/))\n",
- "2. Chunks the content and stores it in a vector store ([lancedb](https://lancedb.com/))\n",
- "3. Allows to query the db and generate an answer using a LLM\n",
- "\n",
- "We can see all of this as *Nodes* connected together in a *Graph*, where the *Nodes* are the `actions` we want to perform.\n",
- "\n",
- "In burr we define actions by simply adding the `action decorator` to each function and specifying what that function needs to `read` and `write` from the graph's state.\n"
- ],
+ "source": "Our goal is to define a flow (DAG) that:\n\n1. Fetches markdown from webpages ([scrapegraph](https://scrapegraphai.com/))\n2. Chunks the content and stores it in a vector store ([lancedb](https://lancedb.com/))\n3. Allows to query the db and generate an answer using a LLM\n\nWe can see all of this as *Nodes* connected together in a *Graph*, where the *Nodes* are the `actions` we want to perform.\n\nIn burr we define actions by simply adding the `action decorator` to each function and specifying what that function needs to `read` and `write` from the graph's state.\n",
+ "outputs": [],
"metadata": {
"id": "359QH2__gAZo"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "All imports"
- ],
+ "source": "All imports",
+ "outputs": [],
"metadata": {
"id": "2aWKZrJgijJR"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "from burr.core import action, State, ApplicationBuilder\n",
- "\n",
- "from scrapegraph_py import Client\n",
- "from scrapegraph_py.logger import sgai_logger\n",
- "\n",
- "import re\n",
- "import lancedb\n",
- "from lancedb.pydantic import LanceModel, Vector\n",
- "from lancedb.embeddings import get_registry\n",
- "\n",
- "import openai\n",
- "import tiktoken\n",
- "from typing import List, Optional\n",
- "\n",
- "from opentelemetry.instrumentation.lancedb import LanceInstrumentor\n",
- "from opentelemetry.instrumentation.openai import OpenAIInstrumentor"
- ],
+ "source": "from burr.core import action, State, ApplicationBuilder\n\nfrom scrapegraph_py import Client\nfrom scrapegraph_py.logger import sgai_logger\n\nimport re\nimport lancedb\nfrom lancedb.pydantic import LanceModel, Vector\nfrom lancedb.embeddings import get_registry\n\nimport openai\nimport tiktoken\nfrom typing import List, Optional\n\nfrom opentelemetry.instrumentation.lancedb import LanceInstrumentor\nfrom opentelemetry.instrumentation.openai import OpenAIInstrumentor",
+ "outputs": [],
"metadata": {
"id": "HeYtTm0DigBM"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 66,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "#### 🔍 Define `fetch_webpage` action to fetch and convert a webpage into `markdown`"
- ],
+ "source": "#### 🔍 Define `fetch_webpage` action to fetch and convert a webpage into `markdown`",
+ "outputs": [],
"metadata": {
"id": "i1Ei6gDTdzYW"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "Here we use `markdownify` to fetch a webpage and convert it into markdown format, which is suitable to LLM ingestion.\n\nYou can find more info in the [official scrapegraph documentation](https://docs.scrapegraphai.com/services/markdownify)\n\n",
+ "outputs": [],
"metadata": {
"id": "M1KSXffZopUD"
},
- "source": [
- "Here we use `markdownify` to fetch a webpage and convert it into markdown format, which is suitable to LLM ingestion.\n",
- "\n",
- "You can find more info in the [official scrapegraph documentation](https://docs.scrapegraphai.com/services/markdownify)\n",
- "\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "sgai_logger.set_logging(level=\"INFO\") #Optional\n",
- "\n",
- "# Define scrapegraph sync client (API KEY is a env variable)\n",
- "sgai_client = Client()\n"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "JS3DZIAhbjD1",
- "outputId": "de22fff8-96b1-489f-bbd7-e2eef24c99c5"
- },
- "execution_count": 67,
+ "source": "sgai_logger.set_logging(level=\"INFO\") #Optional\n\n# Define scrapegraph sync client (API KEY is a env variable)\nsgai_client = Client()\n",
"outputs": [
{
- "output_type": "stream",
"name": "stderr",
"text": [
"💬 2024-12-29 13:56:08,576 🔑 Initializing Client\n",
"INFO:scrapegraph:🔑 Initializing Client\n",
"💬 2024-12-29 13:56:08,582 ✅ Client initialized successfully\n",
"INFO:scrapegraph:✅ Client initialized successfully\n"
- ]
+ ],
+ "output_type": "stream"
}
- ]
+ ],
+ "metadata": {
+ "id": "JS3DZIAhbjD1",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "de22fff8-96b1-489f-bbd7-e2eef24c99c5"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 67,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 68,
+ "source": "@action(reads=[], writes=[\"markdown_content\"])\ndef fetch_webpage(state: State, webpage_url: str) -> State:\n \"\"\"Fetch a webpage and convert it to Markdown.\"\"\"\n try:\n response = sgai_client.markdownify(website_url=webpage_url)\n markdown_content = response[\"result\"]\n print(f\"Request ID: {response['request_id']}\")\n print(f\"Markdown Content: {markdown_content[:200]}... (truncated)\")\n return state.update(markdown_content=markdown_content)\n except Exception as e:\n sgai_logger.error(f\"Failed to fetch webpage: {e}\")\n raise",
+ "outputs": [],
"metadata": {
"id": "ySoE0Rowjgp1"
},
- "outputs": [],
- "source": [
- "@action(reads=[], writes=[\"markdown_content\"])\n",
- "def fetch_webpage(state: State, webpage_url: str) -> State:\n",
- " \"\"\"Fetch a webpage and convert it to Markdown.\"\"\"\n",
- " try:\n",
- " response = sgai_client.markdownify(website_url=webpage_url)\n",
- " markdown_content = response[\"result\"]\n",
- " print(f\"Request ID: {response['request_id']}\")\n",
- " print(f\"Markdown Content: {markdown_content[:200]}... (truncated)\")\n",
- " return state.update(markdown_content=markdown_content)\n",
- " except Exception as e:\n",
- " sgai_logger.error(f\"Failed to fetch webpage: {e}\")\n",
- " raise"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 68,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "#### 📁 Define `embed_and_store` action to chunk the markdown and store it in a local vector store"
- ],
+ "source": "#### 📁 Define `embed_and_store` action to chunk the markdown and store it in a local vector store",
+ "outputs": [],
"metadata": {
"id": "BNXvZkkLsOed"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "Define the data structure to hold the chunks in the `lancedb` vector store"
- ],
+ "source": "Define the data structure to hold the chunks in the `lancedb` vector store",
+ "outputs": [],
"metadata": {
"id": "zj-fQ7rbt4uc"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "embedding_model = get_registry().get(\"openai\").create()\n",
- "\n",
- "class TextDocument(LanceModel):\n",
- " \"\"\"Simple data structure to hold a piece of text associated with a url.\"\"\"\n",
- " url: str\n",
- " position: int\n",
- " text: str = embedding_model.SourceField()\n",
- " vector: Vector(dim=embedding_model.ndims()) = embedding_model.VectorField()"
- ],
+ "source": "embedding_model = get_registry().get(\"openai\").create()\n\nclass TextDocument(LanceModel):\n \"\"\"Simple data structure to hold a piece of text associated with a url.\"\"\"\n url: str\n position: int\n text: str = embedding_model.SourceField()\n vector: Vector(dim=embedding_model.ndims()) = embedding_model.VectorField()",
+ "outputs": [],
"metadata": {
"id": "dqVyq0ONl6DY"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 69,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "Utils to create chunks based on the `number of tokens`"
- ],
+ "source": "Utils to create chunks based on the `number of tokens`",
+ "outputs": [],
"metadata": {
"id": "R97nRo-EtsGP"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "# Constants\n",
- "CHUNK_SIZE = 2000 # Target size of each chunk in tokens\n",
- "MIN_CHUNK_LENGTH_TO_EMBED = 5 # Discard chunks shorter than this\n",
- "\n",
- "# Initialize the tokenizer\n",
- "tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
- "\n",
- "def get_text_chunks(text: str, chunk_token_size: Optional[int] = CHUNK_SIZE) -> List[str]:\n",
- " \"\"\"\n",
- " Split text into chunks of approximately `chunk_token_size` tokens.\n",
- "\n",
- " Args:\n",
- " text: The input text to chunk.\n",
- " chunk_token_size: Target size of each chunk in tokens.\n",
- "\n",
- " Returns:\n",
- " A list of text chunks.\n",
- " \"\"\"\n",
- " if not text.strip():\n",
- " return []\n",
- "\n",
- " tokens = tokenizer.encode(text)\n",
- " chunks = []\n",
- " chunk_size = chunk_token_size or CHUNK_SIZE\n",
- "\n",
- " while tokens:\n",
- " # Extract chunk tokens and decode to text\n",
- " chunk_tokens = tokens[:chunk_size]\n",
- " chunk_text = tokenizer.decode(chunk_tokens).strip()\n",
- "\n",
- " # Skip chunks that are too short\n",
- " if len(chunk_text) > MIN_CHUNK_LENGTH_TO_EMBED:\n",
- " chunks.append(chunk_text)\n",
- "\n",
- " # Remove processed tokens\n",
- " tokens = tokens[len(chunk_tokens):]\n",
- "\n",
- " return chunks"
- ],
+ "source": "# Constants\nCHUNK_SIZE = 2000 # Target size of each chunk in tokens\nMIN_CHUNK_LENGTH_TO_EMBED = 5 # Discard chunks shorter than this\n\n# Initialize the tokenizer\ntokenizer = tiktoken.get_encoding(\"cl100k_base\")\n\ndef get_text_chunks(text: str, chunk_token_size: Optional[int] = CHUNK_SIZE) -> List[str]:\n \"\"\"\n Split text into chunks of approximately `chunk_token_size` tokens.\n\n Args:\n text: The input text to chunk.\n chunk_token_size: Target size of each chunk in tokens.\n\n Returns:\n A list of text chunks.\n \"\"\"\n if not text.strip():\n return []\n\n tokens = tokenizer.encode(text)\n chunks = []\n chunk_size = chunk_token_size or CHUNK_SIZE\n\n while tokens:\n # Extract chunk tokens and decode to text\n chunk_tokens = tokens[:chunk_size]\n chunk_text = tokenizer.decode(chunk_tokens).strip()\n\n # Skip chunks that are too short\n if len(chunk_text) > MIN_CHUNK_LENGTH_TO_EMBED:\n chunks.append(chunk_text)\n\n # Remove processed tokens\n tokens = tokens[len(chunk_tokens):]\n\n return chunks",
+ "outputs": [],
"metadata": {
"id": "OfFv8Zb9qaTD"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 52,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "Let's define the action. It creates a webpages `local vector store` if not present and add the chunks to the `chunks table`"
- ],
+ "source": "Let's define the action. It creates a webpages `local vector store` if not present and add the chunks to the `chunks table`",
+ "outputs": [],
"metadata": {
"id": "Ep4THhetuTey"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "@action(reads=[\"markdown_content\"], writes=[])\n",
- "def embed_and_store(state: State, webpage_url: str) -> State:\n",
- " \"\"\"Embed and store the Markdown content.\"\"\"\n",
- " markdown_content = state[\"markdown_content\"]\n",
- "\n",
- " chunks = get_text_chunks(markdown_content)\n",
- "\n",
- " # Embed and store the chunks using LanceDB\n",
- " con = lancedb.connect(\"./webpages\")\n",
- " table = con.create_table(\"chunks\", exist_ok=True, schema=TextDocument)\n",
- " table.add([\n",
- " {\"text\": c, \"url\": webpage_url, \"position\": i}\n",
- " for i, c in enumerate(chunks)\n",
- " ])\n",
- "\n",
- " return state"
- ],
+ "source": "@action(reads=[\"markdown_content\"], writes=[])\ndef embed_and_store(state: State, webpage_url: str) -> State:\n \"\"\"Embed and store the Markdown content.\"\"\"\n markdown_content = state[\"markdown_content\"]\n\n chunks = get_text_chunks(markdown_content)\n\n # Embed and store the chunks using LanceDB\n con = lancedb.connect(\"./webpages\")\n table = con.create_table(\"chunks\", exist_ok=True, schema=TextDocument)\n table.add([\n {\"text\": c, \"url\": webpage_url, \"position\": i}\n for i, c in enumerate(chunks)\n ])\n\n return state",
+ "outputs": [],
"metadata": {
"id": "RMk8gjUtk0gL"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 53,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "#### 💬 Define `ask_question` action to retrieve the most relevant chunks from the vector store and query them with a llm"
- ],
+ "source": "#### 💬 Define `ask_question` action to retrieve the most relevant chunks from the vector store and query them with a llm",
+ "outputs": [],
"metadata": {
"id": "zbP0km8BtN73"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "Fetches the first 3 relevant chunks based on the user query and generate and answer"
- ],
+ "source": "Fetches the first 3 relevant chunks based on the user query and generate and answer",
+ "outputs": [],
"metadata": {
"id": "HTShnZowukj8"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "@action(reads=[], writes=[\"llm_answer\"])\n",
- "def ask_question(state: State, user_query: str) -> State:\n",
- " \"\"\"Reply to the user's query using the webpage's content.\"\"\"\n",
- " # Retrieve the most relevant chunks\n",
- " chunks_table = lancedb.connect(\"./webpages\").open_table(\"chunks\")\n",
- " search_results = (\n",
- " chunks_table\n",
- " .search(user_query)\n",
- " .select([\"text\", \"url\", \"position\"])\n",
- " .limit(3)\n",
- " .to_list()\n",
- " )\n",
- " relevant_content = \"\\n\".join([r[\"text\"] for r in search_results])\n",
- "\n",
- " # Prompt the LLM with the relevant content\n",
- " system_prompt = (\n",
- " \"Answer the user's questions based on the provided webpage content. \"\n",
- " f\"WEBPAGE CONTENT:\\n{relevant_content}\"\n",
- " )\n",
- "\n",
- " client = openai.OpenAI()\n",
- " response = client.chat.completions.create(\n",
- " model=\"gpt-4o-mini\",\n",
- " messages=[\n",
- " {\"role\": \"system\", \"content\": system_prompt},\n",
- " {\"role\": \"user\", \"content\": user_query}\n",
- " ],\n",
- " )\n",
- " llm_answer = response.choices[0].message.content\n",
- " return state.update(llm_answer=llm_answer)"
- ],
+ "source": "@action(reads=[], writes=[\"llm_answer\"])\ndef ask_question(state: State, user_query: str) -> State:\n \"\"\"Reply to the user's query using the webpage's content.\"\"\"\n # Retrieve the most relevant chunks\n chunks_table = lancedb.connect(\"./webpages\").open_table(\"chunks\")\n search_results = (\n chunks_table\n .search(user_query)\n .select([\"text\", \"url\", \"position\"])\n .limit(3)\n .to_list()\n )\n relevant_content = \"\\n\".join([r[\"text\"] for r in search_results])\n\n # Prompt the LLM with the relevant content\n system_prompt = (\n \"Answer the user's questions based on the provided webpage content. \"\n f\"WEBPAGE CONTENT:\\n{relevant_content}\"\n )\n\n client = openai.OpenAI()\n response = client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": user_query}\n ],\n )\n llm_answer = response.choices[0].message.content\n return state.update(llm_answer=llm_answer)",
+ "outputs": [],
"metadata": {
"id": "VXHbMtHxldKQ"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 54,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
- "source": [
- "#### 🤖 Define the burr `application graph` and run it"
- ],
+ "source": "#### 🤖 Define the burr `application graph` and run it",
+ "outputs": [],
"metadata": {
"id": "OOnKVmJTvORy"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "# Instrumenting OpenTelemetry for LanceDB and OpenAI\n",
- "OpenAIInstrumentor().instrument()\n",
- "LanceInstrumentor().instrument()"
- ],
+ "source": "# Instrumenting OpenTelemetry for LanceDB and OpenAI\nOpenAIInstrumentor().instrument()\nLanceInstrumentor().instrument()",
+ "outputs": [],
"metadata": {
"id": "PL9oRtxpnYHy"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": null,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "# Define the application graph\n",
- "app = (\n",
- " ApplicationBuilder()\n",
- " .with_actions(fetch_webpage=fetch_webpage, embed_and_store=embed_and_store, ask_question=ask_question)\n",
- " .with_transitions((\"fetch_webpage\", \"embed_and_store\"), (\"embed_and_store\", \"ask_question\"))\n",
- " .with_entrypoint(\"fetch_webpage\")\n",
- " .with_tracker(project=\"chat-webpage-simple-rag\", use_otel_tracing=True)\n",
- " .build()\n",
- ")"
- ],
+ "source": "# Define the application graph\napp = (\n ApplicationBuilder()\n .with_actions(fetch_webpage=fetch_webpage, embed_and_store=embed_and_store, ask_question=ask_question)\n .with_transitions((\"fetch_webpage\", \"embed_and_store\"), (\"embed_and_store\", \"ask_question\"))\n .with_entrypoint(\"fetch_webpage\")\n .with_tracker(project=\"chat-webpage-simple-rag\", use_otel_tracing=True)\n .build()\n)",
+ "outputs": [],
"metadata": {
"id": "W3f0m3CCm50A"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 63,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "# Run the application graph\n",
- "action_name, results, state = app.run(\n",
- " halt_after=[\"ask_question\"],\n",
- " inputs={\n",
- " \"webpage_url\": \"https://scrapegraphai.com/\",\n",
- " \"user_query\": \"Who are the founders?\"\n",
- " }\n",
- ")"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "TRPUyVtbnGBZ",
- "outputId": "9ddfbfbf-60fe-4592-aee7-caba292ce2eb"
- },
- "execution_count": 64,
+ "source": "# Run the application graph\naction_name, results, state = app.run(\n halt_after=[\"ask_question\"],\n inputs={\n \"webpage_url\": \"https://scrapegraphai.com/\",\n \"user_query\": \"Who are the founders?\"\n }\n)",
"outputs": [
{
- "output_type": "stream",
"name": "stderr",
"text": [
"💬 2024-12-29 13:51:54,568 🔍 Starting markdownify request for https://scrapegraphai.com/\n",
@@ -505,10 +459,10 @@
"INFO:scrapegraph:✅ Request completed successfully: POST https://api.scrapegraphai.com/v1/markdownify\n",
"💬 2024-12-29 13:51:57,649 ✨ Markdownify request completed successfully\n",
"INFO:scrapegraph:✨ Markdownify request completed successfully\n"
- ]
+ ],
+ "output_type": "stream"
},
{
- "output_type": "stream",
"name": "stdout",
"text": [
"Request ID: d646737f-2dbd-4c6d-aecb-fc5f2c3e132d\n",
@@ -520,26 +474,29 @@
"### Just One Prompt Away\n",
"\n",
"Transform any website into clean, organized data for AI a... (truncated)\n"
- ]
+ ],
+ "output_type": "stream"
}
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "print(state[\"llm_answer\"])"
],
"metadata": {
+ "id": "TRPUyVtbnGBZ",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "j1Ht3OCdozIY",
- "outputId": "fc4ff9c3-1675-447e-e8f8-b228731de9fc"
+ "outputId": "9ddfbfbf-60fe-4592-aee7-caba292ce2eb"
},
- "execution_count": 65,
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 64,
+ "executionEndTime": null,
+ "executionStartTime": null
+ },
+ {
+ "source": "print(state[\"llm_answer\"])",
"outputs": [
{
- "output_type": "stream",
"name": "stdout",
"text": [
"The founders of ScrapeGraphAI are:\n",
@@ -552,56 +509,58 @@
"\n",
"3. **Lorenzo Padoan** - Founder & Product Engineer\n",
" - [LinkedIn profile of Lorenzo Padoan](https://www.linkedin.com/in/lorenzo-padoan-4521a2154)\n"
- ]
+ ],
+ "output_type": "stream"
}
- ]
+ ],
+ "metadata": {
+ "id": "j1Ht3OCdozIY",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "fc4ff9c3-1675-447e-e8f8-b228731de9fc"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 65,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🖼️ Visualize the traces with `Burr UI`",
+ "outputs": [],
"metadata": {
"id": "2as65QLypwdb"
},
- "source": [
- "### 🖼️ Visualize the traces with `Burr UI`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "# execute to load the Burr extension\n",
- "%load_ext burr.integrations.notebook"
- ],
+ "source": "# execute to load the Burr extension\n%load_ext burr.integrations.notebook",
+ "outputs": [],
"metadata": {
"id": "C_amp8mOwcx3"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": 70,
- "outputs": []
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "source": [
- "%burr_ui\n",
- "\n",
- "# If you want to open the ui in another window\n",
- "# from google.colab import output\n",
- "# output.serve_kernel_port_as_window(7241)"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 421
- },
- "id": "6mr_3Mp5wgrL",
- "outputId": "567a1499-346d-416f-c4ca-9045bb7bb26b"
- },
- "execution_count": 74,
+ "source": "%burr_ui\n\n# If you want to open the ui in another window\n# from google.colab import output\n# output.serve_kernel_port_as_window(7241)",
"outputs": [
{
- "output_type": "display_data",
"data": {
- "text/plain": [
- "\n ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n)
\n
\n",
- " ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",)
\n",
- "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Company Info with langchain-scrapegraph"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"8JeaikoxiiLZ"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734622068116,"user_tz":-60,"elapsed":2038,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"14655445-fe17-4d6d-e09e-f198f3d0aad3"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | company_name | \n","description | \n","logo | \n","contact_emails | \n","privacy_policy | \n","terms_of_service | \n","api_status | \n","github | \n","||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","ScrapeGraphAI | \n","ScrapeGraphAI is a powerful AI scraping API de... | \n","https://scrapegraphai.com/images/scrapegraphai... | \n","contact@scrapegraphai.com | \n","https://scrapegraphai.com/privacy | \n","https://scrapegraphai.com/terms | \n","https://scrapegraphapi.openstatus.dev | \n","https://www.linkedin.com/company/101881123 | \n","https://x.com/scrapegraphai | \n","https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n","
| \n"," | name | \n","role | \n","|
|---|---|---|---|
| 0 | \n","\n"," | Founder & Technical Lead | \n","https://www.linkedin.com/in/perinim/ | \n","
| 1 | \n","Marco Vinciguerra | \n","Founder & Software Engineer | \n","https://www.linkedin.com/in/marco-vinciguerra-... | \n","
| 2 | \n","Lorenzo Padoan | \n","Founder & Product Engineer | \n","https://www.linkedin.com/in/lorenzo-padoan-452... | \n","
| \n"," | tier | \n","price | \n","credits | \n","
|---|---|---|---|
| 0 | \n","Free | \n","$0 | \n","100 | \n","
| 1 | \n","Starter | \n","$20/month | \n","5000 | \n","
| 2 | \n","Growth | \n","$100/month | \n","40000 | \n","
| 3 | \n","Pro | \n","$500/month | \n","250000 | \n","
| \n"," | partner | \n","
|---|---|
| 0 | \n","PostHog | \n","
| 1 | \n","AWS | \n","
| 2 | \n","NVIDIA | \n","
| 3 | \n","JinaAI | \n","
| 4 | \n","DagWorks | \n","
| 5 | \n","Browserbase | \n","
| 6 | \n","ScrapeDo | \n","
| 7 | \n","HackerNews | \n","
| 8 | \n","Medium | \n","
| 9 | \n","HackADay | \n","
\n"," ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",")
\n","
| \n", + " | company_name | \n", + "description | \n", + "logo | \n", + "contact_emails | \n", + "privacy_policy | \n", + "terms_of_service | \n", + "api_status | \n", + "github | \n", + "||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "ScrapeGraphAI | \n", + "ScrapeGraphAI is a powerful AI scraping API de... | \n", + "https://scrapegraphai.com/images/scrapegraphai... | \n", + "contact@scrapegraphai.com | \n", + "https://scrapegraphai.com/privacy | \n", + "https://scrapegraphai.com/terms | \n", + "https://scrapegraphapi.openstatus.dev | \n", + "https://www.linkedin.com/company/101881123 | \n", + "https://x.com/scrapegraphai | \n", + "https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n", + "
| \n", + " | name | \n", + "role | \n", + "|
|---|---|---|---|
| 0 | \n", + "\n", + " | Founder & Technical Lead | \n", + "https://www.linkedin.com/in/perinim/ | \n", + "
| 1 | \n", + "Marco Vinciguerra | \n", + "Founder & Software Engineer | \n", + "https://www.linkedin.com/in/marco-vinciguerra-... | \n", + "
| 2 | \n", + "Lorenzo Padoan | \n", + "Founder & Product Engineer | \n", + "https://www.linkedin.com/in/lorenzo-padoan-452... | \n", + "
| \n", + " | tier | \n", + "price | \n", + "credits | \n", + "
|---|---|---|---|
| 0 | \n", + "Free | \n", + "$0 | \n", + "100 | \n", + "
| 1 | \n", + "Starter | \n", + "$20/month | \n", + "5000 | \n", + "
| 2 | \n", + "Growth | \n", + "$100/month | \n", + "40000 | \n", + "
| 3 | \n", + "Pro | \n", + "$500/month | \n", + "250000 | \n", + "
| \n", + " | partner | \n", + "
|---|---|
| 0 | \n", + "PostHog | \n", + "
| 1 | \n", + "AWS | \n", + "
| 2 | \n", + "NVIDIA | \n", + "
| 3 | \n", + "JinaAI | \n", + "
| 4 | \n", + "DagWorks | \n", + "
| 5 | \n", + "Browserbase | \n", + "
| 6 | \n", + "ScrapeDo | \n", + "
| 7 | \n", + "HackerNews | \n", + "
| 8 | \n", + "Medium | \n", + "
| 9 | \n", + "HackADay | \n", + "
\n \n
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Extract Company Info with `llama-index-tools-scrapegraphai`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index) [](https://colab.research.google.com/drive/1sd8WwWrc8Y7QmUh9j_CjlmLc7n7lq8YX?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Extract Company Info with `llama-index-tools-scrapegraphai`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index) [](https://colab.research.google.com/drive/1sd8WwWrc8Y7QmUh9j_CjlmLc7n7lq8YX?usp=sharing)",
+ "text/markdown": "## 🕷️ Extract Company Info with `llama-index-tools-scrapegraphai`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index) [](https://colab.research.google.com/drive/1sd8WwWrc8Y7QmUh9j_CjlmLc7n7lq8YX?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Extract Company Info with `llama-index-tools-scrapegraphai`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:07:54.289Z",
+ "executionStartTime": "2026-03-26T00:07:54.289Z"
},
{
- "cell_type": "markdown",
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "RaHnNF9bYX8N"
},
- "source": [
- ""
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "%%capture\n!pip install llama-index-tools-scrapegraphai",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install llama-index-tools-scrapegraphai"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph` API key",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph` API key"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "import getpass\nimport os\n\nif not os.environ.get(\"SGAI_API_KEY\"):\n os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")",
+ "outputs": [],
"metadata": {
"id": "sffqFG2EJ8bI"
},
- "outputs": [],
- "source": [
- "import getpass\n",
- "import os\n",
- "\n",
- "if not os.environ.get(\"SGAI_API_KEY\"):\n",
- " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n",
+ "outputs": [],
"metadata": {
"id": "jnqMB2-xVYQ7"
},
- "source": [
- "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n\n\n \n
\n",
- " \n",
- "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Company Info with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"3Q5VM3SsRlxO"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734532300517,"user_tz":-60,"elapsed":6877,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"f6b837cd-0f00-49cc-cb6f-f2bca57544f5"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | company_name | \n","description | \n","logo | \n","contact_emails | \n","privacy_policy | \n","terms_of_service | \n","api_status | \n","github | \n","||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","ScrapeGraphAI | \n","ScrapeGraphAI is a powerful AI scraping API de... | \n","https://scrapegraphai.com/images/scrapegraphai... | \n","contact@scrapegraphai.com | \n","https://scrapegraphai.com/privacy | \n","https://scrapegraphai.com/terms | \n","https://scrapegraphapi.openstatus.dev | \n","https://www.linkedin.com/company/101881123 | \n","https://x.com/scrapegraphai | \n","https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n","
| \n"," | name | \n","role | \n","|
|---|---|---|---|
| 0 | \n","\n"," | Founder & Technical Lead | \n","https://www.linkedin.com/in/perinim/ | \n","
| 1 | \n","Marco Vinciguerra | \n","Founder & Software Engineer | \n","https://www.linkedin.com/in/marco-vinciguerra-... | \n","
| 2 | \n","Lorenzo Padoan | \n","Founder & Product Engineer | \n","https://www.linkedin.com/in/lorenzo-padoan-452... | \n","
| \n"," | tier | \n","price | \n","credits | \n","
|---|---|---|---|
| 0 | \n","Free | \n","$0 | \n","100 | \n","
| 1 | \n","Starter | \n","$20/month | \n","5000 | \n","
| 2 | \n","Growth | \n","$100/month | \n","40000 | \n","
| 3 | \n","Pro | \n","$500/month | \n","250000 | \n","
| \n"," | partner | \n","
|---|---|
| 0 | \n","PostHog | \n","
| 1 | \n","AWS | \n","
| 2 | \n","NVIDIA | \n","
| 3 | \n","JinaAI | \n","
| 4 | \n","DagWorks | \n","
| 5 | \n","Browserbase | \n","
| 6 | \n","ScrapeDo | \n","
| 7 | \n","HackerNews | \n","
| 8 | \n","Medium | \n","
| 9 | \n","HackADay | \n","
\n"," \n","
| \n", + " | company_name | \n", + "description | \n", + "logo | \n", + "contact_emails | \n", + "privacy_policy | \n", + "terms_of_service | \n", + "api_status | \n", + "github | \n", + "||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "ScrapeGraphAI | \n", + "ScrapeGraphAI is a powerful AI scraping API de... | \n", + "https://scrapegraphai.com/images/scrapegraphai... | \n", + "contact@scrapegraphai.com | \n", + "https://scrapegraphai.com/privacy | \n", + "https://scrapegraphai.com/terms | \n", + "https://scrapegraphapi.openstatus.dev | \n", + "https://www.linkedin.com/company/101881123 | \n", + "https://x.com/scrapegraphai | \n", + "https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n", + "
| \n", + " | name | \n", + "role | \n", + "|
|---|---|---|---|
| 0 | \n", + "\n", + " | Founder & Technical Lead | \n", + "https://www.linkedin.com/in/perinim/ | \n", + "
| 1 | \n", + "Marco Vinciguerra | \n", + "Founder & Software Engineer | \n", + "https://www.linkedin.com/in/marco-vinciguerra-... | \n", + "
| 2 | \n", + "Lorenzo Padoan | \n", + "Founder & Product Engineer | \n", + "https://www.linkedin.com/in/lorenzo-padoan-452... | \n", + "
| \n", + " | tier | \n", + "price | \n", + "credits | \n", + "
|---|---|---|---|
| 0 | \n", + "Free | \n", + "$0 | \n", + "100 | \n", + "
| 1 | \n", + "Starter | \n", + "$20/month | \n", + "5000 | \n", + "
| 2 | \n", + "Growth | \n", + "$100/month | \n", + "40000 | \n", + "
| 3 | \n", + "Pro | \n", + "$500/month | \n", + "250000 | \n", + "
| \n", + " | partner | \n", + "
|---|---|
| 0 | \n", + "PostHog | \n", + "
| 1 | \n", + "AWS | \n", + "
| 2 | \n", + "NVIDIA | \n", + "
| 3 | \n", + "JinaAI | \n", + "
| 4 | \n", + "DagWorks | \n", + "
| 5 | \n", + "Browserbase | \n", + "
| 6 | \n", + "ScrapeDo | \n", + "
| 7 | \n", + "HackerNews | \n", + "
| 8 | \n", + "Medium | \n", + "
| 9 | \n", + "HackADay | \n", + "
\n \n
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️🦜 Extract Github Trending Repositories with langchain-scrapegraph\n"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"IhozYNwsgJzt"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734619154386,"user_tz":-60,"elapsed":4395,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"07af4bbe-c226-4fb3-8f68-7ccd429f59cc"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | name | \n","description | \n","stars | \n","forks | \n","today_stars | \n","language | \n","
|---|---|---|---|---|---|---|
| 0 | \n","XiaoMi/ha_xiaomi_home | \n","Xiaomi Home Integration for Home Assistant | \n","11097 | \n","472 | \n","3023 | \n","Python | \n","
| 1 | \n","comet-ml/opik | \n","Open-source end-to-end LLM Development Platform | \n","2741 | \n","169 | \n","91 | \n","Java | \n","
| 2 | \n","EbookFoundation/free-programming-books | \n","📚 Freely available programming books | \n","341919 | \n","62038 | \n","225 | \n","HTML | \n","
| 3 | \n","konfig-dev/konfig | \n","Sunset as of December 2024 | \n","689 | \n","192 | \n","224 | \n","TypeScript | \n","
| 4 | \n","anoma/anoma | \n","Reference implementation of Anoma | \n","9451 | \n","452 | \n","4129 | \n","Elixir | \n","
| 5 | \n","stripe/stripe-ios | \n","Stripe iOS SDK | \n","2292 | \n","1004 | \n","49 | \n","Swift | \n","
| 6 | \n","Guovin/iptv-api | \n","IPTV live TV source update tool | \n","9385 | \n","2010 | \n","91 | \n","Python | \n","
| 7 | \n","facebookresearch/AnimatedDrawings | \n","Code to accompany \"A Method for Animating Chil... | \n","11473 | \n","988 | \n","398 | \n","Python | \n","
| 8 | \n","apache/airflow | \n","Apache Airflow - A platform to programmaticall... | \n","37690 | \n","14411 | \n","25 | \n","Python | \n","
| 9 | \n","seleniumbase/SeleniumBase | \n","Python APIs for web automation, testing, and b... | \n","6646 | \n","1028 | \n","624 | \n","Python | \n","
\n"," \n","
| \n", + " | name | \n", + "description | \n", + "stars | \n", + "forks | \n", + "today_stars | \n", + "language | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "XiaoMi/ha_xiaomi_home | \n", + "Xiaomi Home Integration for Home Assistant | \n", + "11097 | \n", + "472 | \n", + "3023 | \n", + "Python | \n", + "
| 1 | \n", + "comet-ml/opik | \n", + "Open-source end-to-end LLM Development Platform | \n", + "2741 | \n", + "169 | \n", + "91 | \n", + "Java | \n", + "
| 2 | \n", + "EbookFoundation/free-programming-books | \n", + "📚 Freely available programming books | \n", + "341919 | \n", + "62038 | \n", + "225 | \n", + "HTML | \n", + "
| 3 | \n", + "konfig-dev/konfig | \n", + "Sunset as of December 2024 | \n", + "689 | \n", + "192 | \n", + "224 | \n", + "TypeScript | \n", + "
| 4 | \n", + "anoma/anoma | \n", + "Reference implementation of Anoma | \n", + "9451 | \n", + "452 | \n", + "4129 | \n", + "Elixir | \n", + "
| 5 | \n", + "stripe/stripe-ios | \n", + "Stripe iOS SDK | \n", + "2292 | \n", + "1004 | \n", + "49 | \n", + "Swift | \n", + "
| 6 | \n", + "Guovin/iptv-api | \n", + "IPTV live TV source update tool | \n", + "9385 | \n", + "2010 | \n", + "91 | \n", + "Python | \n", + "
| 7 | \n", + "facebookresearch/AnimatedDrawings | \n", + "Code to accompany \"A Method for Animating Chil... | \n", + "11473 | \n", + "988 | \n", + "398 | \n", + "Python | \n", + "
| 8 | \n", + "apache/airflow | \n", + "Apache Airflow - A platform to programmaticall... | \n", + "37690 | \n", + "14411 | \n", + "25 | \n", + "Python | \n", + "
| 9 | \n", + "seleniumbase/SeleniumBase | \n", + "Python APIs for web automation, testing, and b... | \n", + "6646 | \n", + "1028 | \n", + "624 | \n", + "Python | \n", + "
\n \n
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Extract Github Trending Repositories with llama index and scrapegraphai's APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-2) [](https://colab.research.google.com/drive/18Y1lwdrUm1qVG_Yh1Em_1z_3zyLcWAjV?usp=share_link)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Extract Github Trending Repositories with llama index and scrapegraphai's APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-2) [](https://colab.research.google.com/drive/18Y1lwdrUm1qVG_Yh1Em_1z_3zyLcWAjV?usp=share_link)",
+ "text/markdown": "## 🕷️ Extract Github Trending Repositories with llama index and scrapegraphai's APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-2) [](https://colab.research.google.com/drive/18Y1lwdrUm1qVG_Yh1Em_1z_3zyLcWAjV?usp=share_link)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Extract Github Trending Repositories with llama index and scrapegraphai's APIs\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:05:24.782Z",
+ "executionStartTime": "2026-03-26T00:05:24.782Z"
},
{
- "cell_type": "markdown",
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "IhozYNwsgJzt"
},
- "source": [
- ""
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "%%capture\n!pip install llama-index\n!pip install llama-index-tools-scrapegraphai\n",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install llama-index\n",
- "!pip install llama-index-tools-scrapegraphai\n"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph` API key",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph` API key"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "sffqFG2EJ8bI",
- "outputId": "07af4bbe-c226-4fb3-8f68-7ccd429f59cc"
- },
+ "source": "import os\nfrom getpass import getpass\n\n# Check if the API key is already set in the environment\nsgai_api_key = os.getenv(\"SGAI_API_KEY\")\n\nif sgai_api_key:\n print(\"SGAI_API_KEY found in environment.\")\nelse:\n print(\"SGAI_API_KEY not found in environment.\")\n # Prompt the user to input the API key securely (hidden input)\n sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n if sgai_api_key:\n # Set the API key in the environment\n os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n print(\"SGAI_API_KEY has been set in the environment.\")\n else:\n print(\"No API key entered. Please set the API key to continue.\")\n",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"SGAI_API_KEY not found in environment.\n",
"SGAI_API_KEY has been set in the environment.\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "import os\n",
- "from getpass import getpass\n",
- "\n",
- "# Check if the API key is already set in the environment\n",
- "sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n",
- "\n",
- "if sgai_api_key:\n",
- " print(\"SGAI_API_KEY found in environment.\")\n",
- "else:\n",
- " print(\"SGAI_API_KEY not found in environment.\")\n",
- " # Prompt the user to input the API key securely (hidden input)\n",
- " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
- " if sgai_api_key:\n",
- " # Set the API key in the environment\n",
- " os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n",
- " print(\"SGAI_API_KEY has been set in the environment.\")\n",
- " else:\n",
- " print(\"No API key entered. Please set the API key to continue.\")\n"
- ]
+ "metadata": {
+ "id": "sffqFG2EJ8bI",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "07af4bbe-c226-4fb3-8f68-7ccd429f59cc"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n",
+ "outputs": [],
"metadata": {
"id": "jnqMB2-xVYQ7"
},
- "source": [
- "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n\n\n \n
\n",
- " \n",
- "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Github Trending Repositories with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"d7Zro0xiuo-l"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734439787062,"user_tz":-60,"elapsed":5826,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"ab74193e-e746-4de6-d65d-33a2a26b5d86"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | name | \n","description | \n","stars | \n","forks | \n","today_stars | \n","language | \n","
|---|---|---|---|---|---|---|
| 0 | \n","Byaidu/PDFMathTranslate | \n","PDF scientific paper translation with preserve... | \n","8902 | \n","633 | \n","816 | \n","Python | \n","
| 1 | \n","bigskysoftware/htmx | \n","htmx - high power tools for HTML | \n","39143 | \n","1324 | \n","186 | \n","JavaScript | \n","
| 2 | \n","commaai/openpilot | \n","openpilot is an operating system for robotics.... | \n","50945 | \n","9206 | \n","132 | \n","Python | \n","
| 3 | \n","google-gemini/cookbook | \n","Examples and guides for using the Gemini API | \n","8108 | \n","1011 | \n","1221 | \n","Jupyter Notebook | \n","
| 4 | \n","stripe/stripe-ios | \n","Stripe iOS SDK | \n","2179 | \n","994 | \n","19 | \n","Swift | \n","
| 5 | \n","RIOT-OS/RIOT | \n","RIOT - The friendly OS for IoT | \n","5234 | \n","2017 | \n","168 | \n","C | \n","
| 6 | \n","zju3dv/EasyVolcap | \n","EasyVolcap: Accelerating Neural Volumetric Vid... | \n","802 | \n","52 | \n","30 | \n","Python | \n","
| 7 | \n","TEN-framework/TEN-Agent | \n","TEN Agent is a conversational AI powered by TE... | \n","3245 | \n","313 | \n","296 | \n","Python | \n","
| 8 | \n","DS4SD/docling | \n","Get your documents ready for gen AI | \n","15201 | \n","774 | \n","281 | \n","Python | \n","
| 9 | \n","Guovin/iptv-api | \n","📺IPTV电视直播源更新工具：✨央视频、📱卫视、☕各省份地方台、🌏港·澳·台、🎥电影、🎮游戏... | \n","9046 | \n","1938 | \n","101 | \n","Python | \n","
| 10 | \n","fatedier/frp | \n","A fast reverse proxy to help you expose a loca... | \n","87828 | \n","13502 | \n","64 | \n","Go | \n","
| 11 | \n","facebookresearch/AnimatedDrawings | \n","Code to accompany \"A Method for Animating Chil... | \n","10766 | \n","955 | \n","38 | \n","Python | \n","
| 12 | \n","gorilla/websocket | \n","Package gorilla/websocket is a fast, well-test... | \n","22633 | \n","3495 | \n","13 | \n","Go | \n","
| 13 | \n","DefiLlama/chainlist | \n","NA | \n","2368 | \n","2476 | \n","5 | \n","JavaScript | \n","
| 14 | \n","open-telemetry/opentelemetry-collector | \n","OpenTelemetry Collector | \n","4570 | \n","1497 | \n","4 | \n","Go | \n","
| 15 | \n","RocketChat/Rocket.Chat | \n","The communications platform that puts data pro... | \n","41169 | \n","10877 | \n","73 | \n","TypeScript | \n","
| 16 | \n","langgenius/dify | \n","Dify is an open-source LLM app development pla... | \n","54976 | \n","8083 | \n","127 | \n","TypeScript | \n","
\n"," \n","
| \n", + " | name | \n", + "description | \n", + "stars | \n", + "forks | \n", + "today_stars | \n", + "language | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "Byaidu/PDFMathTranslate | \n", + "PDF scientific paper translation with preserve... | \n", + "8902 | \n", + "633 | \n", + "816 | \n", + "Python | \n", + "
| 1 | \n", + "bigskysoftware/htmx | \n", + "htmx - high power tools for HTML | \n", + "39143 | \n", + "1324 | \n", + "186 | \n", + "JavaScript | \n", + "
| 2 | \n", + "commaai/openpilot | \n", + "openpilot is an operating system for robotics.... | \n", + "50945 | \n", + "9206 | \n", + "132 | \n", + "Python | \n", + "
| 3 | \n", + "google-gemini/cookbook | \n", + "Examples and guides for using the Gemini API | \n", + "8108 | \n", + "1011 | \n", + "1221 | \n", + "Jupyter Notebook | \n", + "
| 4 | \n", + "stripe/stripe-ios | \n", + "Stripe iOS SDK | \n", + "2179 | \n", + "994 | \n", + "19 | \n", + "Swift | \n", + "
| 5 | \n", + "RIOT-OS/RIOT | \n", + "RIOT - The friendly OS for IoT | \n", + "5234 | \n", + "2017 | \n", + "168 | \n", + "C | \n", + "
| 6 | \n", + "zju3dv/EasyVolcap | \n", + "EasyVolcap: Accelerating Neural Volumetric Vid... | \n", + "802 | \n", + "52 | \n", + "30 | \n", + "Python | \n", + "
| 7 | \n", + "TEN-framework/TEN-Agent | \n", + "TEN Agent is a conversational AI powered by TE... | \n", + "3245 | \n", + "313 | \n", + "296 | \n", + "Python | \n", + "
| 8 | \n", + "DS4SD/docling | \n", + "Get your documents ready for gen AI | \n", + "15201 | \n", + "774 | \n", + "281 | \n", + "Python | \n", + "
| 9 | \n", + "Guovin/iptv-api | \n", + "📺IPTV电视直播源更新工具：✨央视频、📱卫视、☕各省份地方台、🌏港·澳·台、🎥电影、🎮游戏... | \n", + "9046 | \n", + "1938 | \n", + "101 | \n", + "Python | \n", + "
| 10 | \n", + "fatedier/frp | \n", + "A fast reverse proxy to help you expose a loca... | \n", + "87828 | \n", + "13502 | \n", + "64 | \n", + "Go | \n", + "
| 11 | \n", + "facebookresearch/AnimatedDrawings | \n", + "Code to accompany \"A Method for Animating Chil... | \n", + "10766 | \n", + "955 | \n", + "38 | \n", + "Python | \n", + "
| 12 | \n", + "gorilla/websocket | \n", + "Package gorilla/websocket is a fast, well-test... | \n", + "22633 | \n", + "3495 | \n", + "13 | \n", + "Go | \n", + "
| 13 | \n", + "DefiLlama/chainlist | \n", + "NA | \n", + "2368 | \n", + "2476 | \n", + "5 | \n", + "JavaScript | \n", + "
| 14 | \n", + "open-telemetry/opentelemetry-collector | \n", + "OpenTelemetry Collector | \n", + "4570 | \n", + "1497 | \n", + "4 | \n", + "Go | \n", + "
| 15 | \n", + "RocketChat/Rocket.Chat | \n", + "The communications platform that puts data pro... | \n", + "41169 | \n", + "10877 | \n", + "73 | \n", + "TypeScript | \n", + "
| 16 | \n", + "langgenius/dify | \n", + "Dify is an open-source LLM app development pla... | \n", + "54976 | \n", + "8083 | \n", + "127 | \n", + "TypeScript | \n", + "
\n \n
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Houses Listing with langchain-scrapegraph"]},{"cell_type":"markdown","metadata":{"id":"5cVkde_LpVkF"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install langchain-scrapegraph"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` API key"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","outputId":"c588274d-64ce-4d13-f12d-0458d9c4839d"},"outputs":[{"name":"stdout","output_type":"stream","text":["SGAI_API_KEY found in environment.\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | price | \n","bedrooms | \n","bathrooms | \n","square_feet | \n","address | \n","city | \n","state | \n","zip_code | \n","tags | \n","agent_name | \n","agency | \n","
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","549000 | \n","1 | \n","1 | \n","0 | \n","380 14th St Unit 405 | \n","San Francisco | \n","CA | \n","94103 | \n","[New construction] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 1 | \n","1799000 | \n","4 | \n","2 | \n","2735 | \n","123 Grattan St | \n","San Francisco | \n","CA | \n","94117 | \n","[Edwardian-style, investment, owner-occupants] | \n","Sean Engmann | \n","eXp Realty of Northern CA Inc. | \n","
| 2 | \n","1995000 | \n","7 | \n","3 | \n","3330 | \n","1590 Washington St | \n","San Francisco | \n","CA | \n","94109 | \n","[Nob Hill, 3-unit building, investment] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 3 | \n","549000 | \n","0 | \n","1 | \n","477 | \n","240 Lombard St Unit 835 | \n","San Francisco | \n","CA | \n","94111 | \n","[Bay view, studio, modern appliances] | \n","Tim Gullicksen | \n","Corcoran Icon Properties | \n","
| 4 | \n","5495000 | \n","10 | \n","7 | \n","6505 | \n","1057 Steiner St | \n","San Francisco | \n","CA | \n","94115 | \n","[Victorian, Bed & Breakfast, Gilded Age] | \n","Bonnie Spindler | \n","Corcoran Icon Properties | \n","
| 5 | \n","925000 | \n","2 | \n","1 | \n","779 | \n","2 Fallon Place Unit 57 | \n","San Francisco | \n","CA | \n","94133 | \n","[Russian Hill, views, exclusive-use deck] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 6 | \n","898000 | \n","2 | \n","2 | \n","1175 | \n","5160 Diamond Heights Blvd Unit 208C | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Joe Polyak | \n","Rise Homes | \n","
| 7 | \n","1700000 | \n","4 | \n","2 | \n","1950 | \n","1351 26th Ave | \n","San Francisco | \n","CA | \n","94122 | \n","[] | \n","Glenda Queensbury | \n","Referral Realty-BV | \n","
| 8 | \n","1899000 | \n","3 | \n","2 | \n","1560 | \n","340 Yerba Buena Ave | \n","San Francisco | \n","CA | \n","94127 | \n","[] | \n","Jeannie Anderson | \n","Coldwell Banker Realty | \n","
| 9 | \n","850000 | \n","2 | \n","2 | \n","1055 | \n","588 Minna Unit 604 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Mohamed Lakdawala | \n","Remax Prestigious Properties | \n","
| 10 | \n","1990000 | \n","3 | \n","1 | \n","1280 | \n","1450 Diamond St | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Mary Anne Villamil | \n","Kinetic Real Estate | \n","
| 11 | \n","849000 | \n","1 | \n","1 | \n","855 | \n","81 Lansing St Unit 401 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Kristen Haenggi | \n","Compass | \n","
| 12 | \n","1080000 | \n","2 | \n","2 | \n","936 | \n","451 Kansas St Unit 466 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Maureen DeBoer | \n","LKJ Realty | \n","
| 13 | \n","1499000 | \n","4 | \n","2 | \n","2145 | \n","486 Yale St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Alicia Atienza | \n","Statewide Realty | \n","
| 14 | \n","1140000 | \n","2 | \n","2 | \n","998 | \n","588 Minna Unit 801 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Milan Jezdimirovic | \n","Compass | \n","
| 15 | \n","1988000 | \n","2 | \n","1 | \n","3800 | \n","183 19th Ave | \n","San Francisco | \n","CA | \n","94121 | \n","[Amazing Property, Marina Style, Needs TLC] | \n","Leo Cheung | \n","eXp Realty of California, Inc | \n","
| 16 | \n","1218000 | \n","2 | \n","2 | \n","1275 | \n","1998 Pacific Ave Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[Light-filled, Freshly painted, Walker's parad... | \n","Grace Sun | \n","Compass | \n","
| 17 | \n","895000 | \n","1 | \n","1 | \n","837 | \n","425 1st St Unit 2501 | \n","San Francisco | \n","CA | \n","94105 | \n","[Unobstructed bay bridge views, Open layout] | \n","Matt Fuller | \n","Jackson Fuller Real Estate | \n","
| 18 | \n","1499000 | \n","3 | \n","1 | \n","1500 | \n","Unlisted Address | \n","San Francisco | \n","CA | \n","NA | \n","[Contractor's Special, Fixer-upper] | \n","Jaymee Faith Sagisi | \n","IMPACT | \n","
| 19 | \n","900000 | \n","1 | \n","1 | \n","930 | \n","1101 Green St Unit 302 | \n","San Francisco | \n","CA | \n","94109 | \n","[Historic Art Deco, Abundant natural light] | \n","NA | \n","NA | \n","
| 20 | \n","858000 | \n","1 | \n","1 | \n","1104 | \n","260 King St Unit 557 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Miyuki Takami | \n","eXp Realty of California, Inc | \n","
| 21 | \n","945000 | \n","2 | \n","1 | \n","767 | \n","307 Page St Unit 1 | \n","San Francisco | \n","CA | \n","94102 | \n","[] | \n","NA | \n","NA | \n","
| 22 | \n","1099000 | \n","2 | \n","2 | \n","1330 | \n","1080 Sutter St Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Annette Liberty | \n","Coldwell Banker Realty | \n","
| 23 | \n","950000 | \n","4 | \n","3 | \n","2090 | \n","3328 26th St Unit 3330 | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Isaac Munene | \n","Coldwell Banker Realty | \n","
| 24 | \n","1088000 | \n","2 | \n","2 | \n","1065 | \n","1776 Sacramento St Unit 503 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Marilyn Becklehimer | \n","Dio Real Estate | \n","
| 25 | \n","1788888 | \n","4 | \n","3 | \n","1856 | \n","2317 15th St | \n","San Francisco | \n","CA | \n","94114 | \n","[] | \n","Joel Gile | \n","Sequoia Real Estate | \n","
| 26 | \n","1650000 | \n","3 | \n","2 | \n","1547 | \n","2475 47th Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Lucy Goldenshteyn | \n","Redfin | \n","
| 27 | \n","998000 | \n","2 | \n","2 | \n","1202 | \n","50 Lansing St Unit 201 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Tracey Broadman | \n","Vanguard Properties | \n","
| 28 | \n","1595000 | \n","3 | \n","5 | \n","1995 | \n","15 Joy St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Mike Stack | \n","Vanguard Properties | \n","
| 29 | \n","1028000 | \n","2 | \n","2 | \n","1065 | \n","50 Lansing St Unit 403 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Robyn Kaufman | \n","Vivre Real Estate | \n","
| 30 | \n","999000 | \n","1 | \n","1 | \n","1021 | \n","338 Spear St Unit 6J | \n","San Francisco | \n","CA | \n","94105 | \n","[Spacious, Balcony, Bright courtyard views] | \n","Paul Hwang | \n","Skybox Realty | \n","
| 31 | \n","799800 | \n","2 | \n","2 | \n","1109 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction, 1-car garage] | \n","Lennar | \n","Lennar | \n","
| 32 | \n","529880 | \n","1 | \n","1 | \n","740 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction, 1-car garage] | \n","Lennar | \n","Lennar | \n","
| 33 | \n","489000 | \n","1 | \n","1 | \n","741 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction, 1-car garage] | \n","Lennar | \n","Lennar | \n","
| 34 | \n","1359000 | \n","4 | \n","2 | \n","1845 | \n","170 Thrift St | \n","San Francisco | \n","CA | \n","94112 | \n","[Updated, Natural light] | \n","Cristal Wright | \n","Vanguard Properties | \n","
| 35 | \n","1295000 | \n","3 | \n","1 | \n","1214 | \n","1922 43rd Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Mila Romprey | \n","Premier Realty Associates | \n","
| 36 | \n","1098000 | \n","3 | \n","1 | \n","1006 | \n","150 Putnam St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Genie Mantzoros | \n","Epic Real Estate & Asso. Inc. | \n","
| 37 | \n","1189870 | \n","3 | \n","2 | \n","1436 | \n","327 Ordway St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Shawn Zahraie | \n","Affinity Enterprises, Inc | \n","
| 38 | \n","899000 | \n","2 | \n","1 | \n","1118 | \n","272 Farallones St | \n","San Francisco | \n","CA | \n","94112 | \n","[] | \n","Janice Lee | \n","Coldwell Banker Realty | \n","
| 39 | \n","30000 | \n","0 | \n","0 | \n","0 | \n","0 Evans Ave | \n","San Francisco | \n","CA | \n","94124 | \n","[Land, 0.12 Acre, $251,467 per Acre] | \n","Heidy Carrera | \n","Berkshire Hathaway HomeService | \n","
\n"," \n","
| \n", + " | price | \n", + "bedrooms | \n", + "bathrooms | \n", + "square_feet | \n", + "address | \n", + "city | \n", + "state | \n", + "zip_code | \n", + "tags | \n", + "agent_name | \n", + "agency | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "549000 | \n", + "1 | \n", + "1 | \n", + "0 | \n", + "380 14th St Unit 405 | \n", + "San Francisco | \n", + "CA | \n", + "94103 | \n", + "[New construction] | \n", + "Eddie O'Sullivan | \n", + "Elevation Real Estate | \n", + "
| 1 | \n", + "1799000 | \n", + "4 | \n", + "2 | \n", + "2735 | \n", + "123 Grattan St | \n", + "San Francisco | \n", + "CA | \n", + "94117 | \n", + "[Edwardian-style, investment, owner-occupants] | \n", + "Sean Engmann | \n", + "eXp Realty of Northern CA Inc. | \n", + "
| 2 | \n", + "1995000 | \n", + "7 | \n", + "3 | \n", + "3330 | \n", + "1590 Washington St | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[Nob Hill, 3-unit building, investment] | \n", + "Eddie O'Sullivan | \n", + "Elevation Real Estate | \n", + "
| 3 | \n", + "549000 | \n", + "0 | \n", + "1 | \n", + "477 | \n", + "240 Lombard St Unit 835 | \n", + "San Francisco | \n", + "CA | \n", + "94111 | \n", + "[Bay view, studio, modern appliances] | \n", + "Tim Gullicksen | \n", + "Corcoran Icon Properties | \n", + "
| 4 | \n", + "5495000 | \n", + "10 | \n", + "7 | \n", + "6505 | \n", + "1057 Steiner St | \n", + "San Francisco | \n", + "CA | \n", + "94115 | \n", + "[Victorian, Bed & Breakfast, Gilded Age] | \n", + "Bonnie Spindler | \n", + "Corcoran Icon Properties | \n", + "
| 5 | \n", + "925000 | \n", + "2 | \n", + "1 | \n", + "779 | \n", + "2 Fallon Place Unit 57 | \n", + "San Francisco | \n", + "CA | \n", + "94133 | \n", + "[Russian Hill, views, exclusive-use deck] | \n", + "Eddie O'Sullivan | \n", + "Elevation Real Estate | \n", + "
| 6 | \n", + "898000 | \n", + "2 | \n", + "2 | \n", + "1175 | \n", + "5160 Diamond Heights Blvd Unit 208C | \n", + "San Francisco | \n", + "CA | \n", + "94131 | \n", + "[] | \n", + "Joe Polyak | \n", + "Rise Homes | \n", + "
| 7 | \n", + "1700000 | \n", + "4 | \n", + "2 | \n", + "1950 | \n", + "1351 26th Ave | \n", + "San Francisco | \n", + "CA | \n", + "94122 | \n", + "[] | \n", + "Glenda Queensbury | \n", + "Referral Realty-BV | \n", + "
| 8 | \n", + "1899000 | \n", + "3 | \n", + "2 | \n", + "1560 | \n", + "340 Yerba Buena Ave | \n", + "San Francisco | \n", + "CA | \n", + "94127 | \n", + "[] | \n", + "Jeannie Anderson | \n", + "Coldwell Banker Realty | \n", + "
| 9 | \n", + "850000 | \n", + "2 | \n", + "2 | \n", + "1055 | \n", + "588 Minna Unit 604 | \n", + "San Francisco | \n", + "CA | \n", + "94103 | \n", + "[] | \n", + "Mohamed Lakdawala | \n", + "Remax Prestigious Properties | \n", + "
| 10 | \n", + "1990000 | \n", + "3 | \n", + "1 | \n", + "1280 | \n", + "1450 Diamond St | \n", + "San Francisco | \n", + "CA | \n", + "94131 | \n", + "[] | \n", + "Mary Anne Villamil | \n", + "Kinetic Real Estate | \n", + "
| 11 | \n", + "849000 | \n", + "1 | \n", + "1 | \n", + "855 | \n", + "81 Lansing St Unit 401 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[] | \n", + "Kristen Haenggi | \n", + "Compass | \n", + "
| 12 | \n", + "1080000 | \n", + "2 | \n", + "2 | \n", + "936 | \n", + "451 Kansas St Unit 466 | \n", + "San Francisco | \n", + "CA | \n", + "94107 | \n", + "[] | \n", + "Maureen DeBoer | \n", + "LKJ Realty | \n", + "
| 13 | \n", + "1499000 | \n", + "4 | \n", + "2 | \n", + "2145 | \n", + "486 Yale St | \n", + "San Francisco | \n", + "CA | \n", + "94134 | \n", + "[] | \n", + "Alicia Atienza | \n", + "Statewide Realty | \n", + "
| 14 | \n", + "1140000 | \n", + "2 | \n", + "2 | \n", + "998 | \n", + "588 Minna Unit 801 | \n", + "San Francisco | \n", + "CA | \n", + "94103 | \n", + "[] | \n", + "Milan Jezdimirovic | \n", + "Compass | \n", + "
| 15 | \n", + "1988000 | \n", + "2 | \n", + "1 | \n", + "3800 | \n", + "183 19th Ave | \n", + "San Francisco | \n", + "CA | \n", + "94121 | \n", + "[Amazing Property, Marina Style, Needs TLC] | \n", + "Leo Cheung | \n", + "eXp Realty of California, Inc | \n", + "
| 16 | \n", + "1218000 | \n", + "2 | \n", + "2 | \n", + "1275 | \n", + "1998 Pacific Ave Unit 202 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[Light-filled, Freshly painted, Walker's parad... | \n", + "Grace Sun | \n", + "Compass | \n", + "
| 17 | \n", + "895000 | \n", + "1 | \n", + "1 | \n", + "837 | \n", + "425 1st St Unit 2501 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[Unobstructed bay bridge views, Open layout] | \n", + "Matt Fuller | \n", + "Jackson Fuller Real Estate | \n", + "
| 18 | \n", + "1499000 | \n", + "3 | \n", + "1 | \n", + "1500 | \n", + "Unlisted Address | \n", + "San Francisco | \n", + "CA | \n", + "NA | \n", + "[Contractor's Special, Fixer-upper] | \n", + "Jaymee Faith Sagisi | \n", + "IMPACT | \n", + "
| 19 | \n", + "900000 | \n", + "1 | \n", + "1 | \n", + "930 | \n", + "1101 Green St Unit 302 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[Historic Art Deco, Abundant natural light] | \n", + "NA | \n", + "NA | \n", + "
| 20 | \n", + "858000 | \n", + "1 | \n", + "1 | \n", + "1104 | \n", + "260 King St Unit 557 | \n", + "San Francisco | \n", + "CA | \n", + "94107 | \n", + "[] | \n", + "Miyuki Takami | \n", + "eXp Realty of California, Inc | \n", + "
| 21 | \n", + "945000 | \n", + "2 | \n", + "1 | \n", + "767 | \n", + "307 Page St Unit 1 | \n", + "San Francisco | \n", + "CA | \n", + "94102 | \n", + "[] | \n", + "NA | \n", + "NA | \n", + "
| 22 | \n", + "1099000 | \n", + "2 | \n", + "2 | \n", + "1330 | \n", + "1080 Sutter St Unit 202 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Annette Liberty | \n", + "Coldwell Banker Realty | \n", + "
| 23 | \n", + "950000 | \n", + "4 | \n", + "3 | \n", + "2090 | \n", + "3328 26th St Unit 3330 | \n", + "San Francisco | \n", + "CA | \n", + "94110 | \n", + "[] | \n", + "Isaac Munene | \n", + "Coldwell Banker Realty | \n", + "
| 24 | \n", + "1088000 | \n", + "2 | \n", + "2 | \n", + "1065 | \n", + "1776 Sacramento St Unit 503 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Marilyn Becklehimer | \n", + "Dio Real Estate | \n", + "
| 25 | \n", + "1788888 | \n", + "4 | \n", + "3 | \n", + "1856 | \n", + "2317 15th St | \n", + "San Francisco | \n", + "CA | \n", + "94114 | \n", + "[] | \n", + "Joel Gile | \n", + "Sequoia Real Estate | \n", + "
| 26 | \n", + "1650000 | \n", + "3 | \n", + "2 | \n", + "1547 | \n", + "2475 47th Ave | \n", + "San Francisco | \n", + "CA | \n", + "94116 | \n", + "[] | \n", + "Lucy Goldenshteyn | \n", + "Redfin | \n", + "
| 27 | \n", + "998000 | \n", + "2 | \n", + "2 | \n", + "1202 | \n", + "50 Lansing St Unit 201 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[] | \n", + "Tracey Broadman | \n", + "Vanguard Properties | \n", + "
| 28 | \n", + "1595000 | \n", + "3 | \n", + "5 | \n", + "1995 | \n", + "15 Joy St | \n", + "San Francisco | \n", + "CA | \n", + "94110 | \n", + "[] | \n", + "Mike Stack | \n", + "Vanguard Properties | \n", + "
| 29 | \n", + "1028000 | \n", + "2 | \n", + "2 | \n", + "1065 | \n", + "50 Lansing St Unit 403 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[] | \n", + "Robyn Kaufman | \n", + "Vivre Real Estate | \n", + "
| 30 | \n", + "999000 | \n", + "1 | \n", + "1 | \n", + "1021 | \n", + "338 Spear St Unit 6J | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[Spacious, Balcony, Bright courtyard views] | \n", + "Paul Hwang | \n", + "Skybox Realty | \n", + "
| 31 | \n", + "799800 | \n", + "2 | \n", + "2 | \n", + "1109 | \n", + "10 Innes Ct | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[New Construction, 1-car garage] | \n", + "Lennar | \n", + "Lennar | \n", + "
| 32 | \n", + "529880 | \n", + "1 | \n", + "1 | \n", + "740 | \n", + "10 Innes Ct | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[New Construction, 1-car garage] | \n", + "Lennar | \n", + "Lennar | \n", + "
| 33 | \n", + "489000 | \n", + "1 | \n", + "1 | \n", + "741 | \n", + "10 Innes Ct | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[New Construction, 1-car garage] | \n", + "Lennar | \n", + "Lennar | \n", + "
| 34 | \n", + "1359000 | \n", + "4 | \n", + "2 | \n", + "1845 | \n", + "170 Thrift St | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[Updated, Natural light] | \n", + "Cristal Wright | \n", + "Vanguard Properties | \n", + "
| 35 | \n", + "1295000 | \n", + "3 | \n", + "1 | \n", + "1214 | \n", + "1922 43rd Ave | \n", + "San Francisco | \n", + "CA | \n", + "94116 | \n", + "[] | \n", + "Mila Romprey | \n", + "Premier Realty Associates | \n", + "
| 36 | \n", + "1098000 | \n", + "3 | \n", + "1 | \n", + "1006 | \n", + "150 Putnam St | \n", + "San Francisco | \n", + "CA | \n", + "94110 | \n", + "[] | \n", + "Genie Mantzoros | \n", + "Epic Real Estate & Asso. Inc. | \n", + "
| 37 | \n", + "1189870 | \n", + "3 | \n", + "2 | \n", + "1436 | \n", + "327 Ordway St | \n", + "San Francisco | \n", + "CA | \n", + "94134 | \n", + "[] | \n", + "Shawn Zahraie | \n", + "Affinity Enterprises, Inc | \n", + "
| 38 | \n", + "899000 | \n", + "2 | \n", + "1 | \n", + "1118 | \n", + "272 Farallones St | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[] | \n", + "Janice Lee | \n", + "Coldwell Banker Realty | \n", + "
| 39 | \n", + "30000 | \n", + "0 | \n", + "0 | \n", + "0 | \n", + "0 Evans Ave | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[Land, 0.12 Acre, $251,467 per Acre] | \n", + "Heidy Carrera | \n", + "Berkshire Hathaway HomeService | \n", + "
\n \n
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Extract Houses Listing on Zillow with llama-index and ScrapegraphAI APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-1) [](https://colab.research.google.com/drive/1MlWu2HBwmvvnXWyezwl93hyApHOufAQi?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Extract Houses Listing on Zillow with llama-index and ScrapegraphAI APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-1) [](https://colab.research.google.com/drive/1MlWu2HBwmvvnXWyezwl93hyApHOufAQi?usp=sharing)",
+ "text/markdown": "## 🕷️ Extract Houses Listing on Zillow with llama-index and ScrapegraphAI APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-1) [](https://colab.research.google.com/drive/1MlWu2HBwmvvnXWyezwl93hyApHOufAQi?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Extract Houses Listing on Zillow with llama-index and ScrapegraphAI APIs"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:09:56.460Z",
+ "executionStartTime": "2026-03-26T00:09:56.460Z"
},
{
- "cell_type": "markdown",
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "5cVkde_LpVkF"
},
- "source": [
- ""
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "%%capture\n!pip install llama-index\n!pip install llama-index-tools-scrapegraphai",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install llama-index\n",
- "!pip install llama-index-tools-scrapegraphai"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph` API key",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph` API key"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "sffqFG2EJ8bI",
- "outputId": "c588274d-64ce-4d13-f12d-0458d9c4839d"
- },
+ "source": "import os\nfrom getpass import getpass\n\n# Check if the API key is already set in the environment\nsgai_api_key = os.getenv(\"SGAI_API_KEY\")\n\nif sgai_api_key:\n print(\"SGAI_API_KEY found in environment.\")\nelse:\n print(\"SGAI_API_KEY not found in environment.\")\n # Prompt the user to input the API key securely (hidden input)\n sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n if sgai_api_key:\n # Set the API key in the environment\n os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n print(\"SGAI_API_KEY has been set in the environment.\")\n else:\n print(\"No API key entered. Please set the API key to continue.\")\n",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"SGAI_API_KEY not found in environment.\n",
"SGAI_API_KEY has been set in the environment.\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "import os\n",
- "from getpass import getpass\n",
- "\n",
- "# Check if the API key is already set in the environment\n",
- "sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n",
- "\n",
- "if sgai_api_key:\n",
- " print(\"SGAI_API_KEY found in environment.\")\n",
- "else:\n",
- " print(\"SGAI_API_KEY not found in environment.\")\n",
- " # Prompt the user to input the API key securely (hidden input)\n",
- " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
- " if sgai_api_key:\n",
- " # Set the API key in the environment\n",
- " os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n",
- " print(\"SGAI_API_KEY has been set in the environment.\")\n",
- " else:\n",
- " print(\"No API key entered. Please set the API key to continue.\")\n"
- ]
+ "metadata": {
+ "id": "sffqFG2EJ8bI",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "c588274d-64ce-4d13-f12d-0458d9c4839d"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n",
+ "outputs": [],
"metadata": {
"id": "jnqMB2-xVYQ7"
},
- "source": [
- "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n\n\n \n
\n",
- " \n",
- "
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Houses Listing with Official Scrapegraph SDK"]},{"cell_type":"markdown","metadata":{"id":"8vZBkAWLq9C1"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` API key"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","outputId":"18dfce64-db37-4825-d316-fabd064100d0"},"outputs":[{"name":"stdout","output_type":"stream","text":["SGAI_API_KEY found in environment.\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | price | \n","bedrooms | \n","bathrooms | \n","square_feet | \n","address | \n","city | \n","state | \n","zip_code | \n","tags | \n","agent_name | \n","agency | \n","
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","549000 | \n","1 | \n","1 | \n","477 | \n","380 14th St Unit 405 | \n","San Francisco | \n","CA | \n","94103 | \n","[New construction] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 1 | \n","1799000 | \n","4 | \n","2 | \n","2735 | \n","123 Grattan St | \n","San Francisco | \n","CA | \n","94117 | \n","[] | \n","Sean Engmann | \n","eXp Realty of Northern CA Inc. | \n","
| 2 | \n","1995000 | \n","7 | \n","3 | \n","3330 | \n","1590 Washington St | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 3 | \n","549000 | \n","0 | \n","1 | \n","477 | \n","240 Lombard St Unit 835 | \n","San Francisco | \n","CA | \n","94111 | \n","[] | \n","Tim Gullicksen | \n","Corcoran Icon Properties | \n","
| 4 | \n","5495000 | \n","10 | \n","7 | \n","6505 | \n","1057 Steiner St | \n","San Francisco | \n","CA | \n","94115 | \n","[] | \n","Bonnie Spindler | \n","Corcoran Icon Properties | \n","
| 5 | \n","925000 | \n","2 | \n","1 | \n","779 | \n","2 Fallon Place Unit 57 | \n","San Francisco | \n","CA | \n","94133 | \n","[] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 6 | \n","898000 | \n","2 | \n","2 | \n","1175 | \n","5160 Diamond Heights Blvd Unit 208C | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Joe Polyak | \n","Rise Homes | \n","
| 7 | \n","1700000 | \n","4 | \n","2 | \n","1950 | \n","1351 26th Ave | \n","San Francisco | \n","CA | \n","94122 | \n","[] | \n","Glenda Queensbury | \n","Referral Realty-BV | \n","
| 8 | \n","1899000 | \n","3 | \n","2 | \n","1560 | \n","340 Yerba Buena Ave | \n","San Francisco | \n","CA | \n","94127 | \n","[] | \n","Jeannie Anderson | \n","Coldwell Banker Realty | \n","
| 9 | \n","850000 | \n","2 | \n","2 | \n","1055 | \n","588 Minna Unit 604 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Mohamed Lakdawala | \n","Remax Prestigious Properties | \n","
| 10 | \n","1990000 | \n","3 | \n","1 | \n","1280 | \n","1450 Diamond St | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Mary Anne Villamil | \n","Kinetic Real Estate | \n","
| 11 | \n","849000 | \n","1 | \n","1 | \n","855 | \n","81 Lansing St Unit 401 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Kristen Haenggi | \n","Compass | \n","
| 12 | \n","1080000 | \n","2 | \n","2 | \n","936 | \n","451 Kansas St Unit 466 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Maureen DeBoer | \n","LKJ Realty | \n","
| 13 | \n","1499000 | \n","4 | \n","2 | \n","2145 | \n","486 Yale St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Alicia Atienza | \n","Statewide Realty | \n","
| 14 | \n","1140000 | \n","2 | \n","2 | \n","998 | \n","588 Minna Unit 801 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Milan Jezdimirovic | \n","Compass | \n","
| 15 | \n","1988000 | \n","2 | \n","1 | \n","3800 | \n","183 19th Ave | \n","San Francisco | \n","CA | \n","94121 | \n","[Amazing Property, Marina Style, Needs TLC] | \n","Leo Cheung | \n","eXp Realty of California, Inc | \n","
| 16 | \n","1218000 | \n","2 | \n","2 | \n","1275 | \n","1998 Pacific Ave Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[Light-filled, Freshly painted, Walker's parad... | \n","Grace Sun | \n","Compass | \n","
| 17 | \n","895000 | \n","1 | \n","1 | \n","837 | \n","425 1st St Unit 2501 | \n","San Francisco | \n","CA | \n","94105 | \n","[Unobstructed bay bridge views, Open layout] | \n","Matt Fuller | \n","Jackson Fuller Real Estate | \n","
| 18 | \n","1499000 | \n","3 | \n","1 | \n","1500 | \n","Unlisted Address | \n","San Francisco | \n","CA | \n","NA | \n","[Contractor's Special, Fixer-upper] | \n","Jaymee Faith Sagisi | \n","IMPACT | \n","
| 19 | \n","900000 | \n","1 | \n","1 | \n","930 | \n","1101 Green St Unit 302 | \n","San Francisco | \n","CA | \n","94109 | \n","[Historic Art Deco, Iconic views] | \n","NA | \n","NA | \n","
| 20 | \n","858000 | \n","1 | \n","1 | \n","1104 | \n","260 King St Unit 557 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Miyuki Takami | \n","eXp Realty of California, Inc | \n","
| 21 | \n","945000 | \n","2 | \n","1 | \n","767 | \n","307 Page St Unit 1 | \n","San Francisco | \n","CA | \n","94102 | \n","[] | \n","NA | \n","NA | \n","
| 22 | \n","1099000 | \n","2 | \n","2 | \n","1330 | \n","1080 Sutter St Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Annette Liberty | \n","Coldwell Banker Realty | \n","
| 23 | \n","950000 | \n","4 | \n","3 | \n","2090 | \n","3328 26th St Unit 3330 | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Isaac Munene | \n","Coldwell Banker Realty | \n","
| 24 | \n","1088000 | \n","2 | \n","2 | \n","1065 | \n","1776 Sacramento St Unit 503 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Marilyn Becklehimer | \n","Dio Real Estate | \n","
| 25 | \n","1788888 | \n","4 | \n","3 | \n","1856 | \n","2317 15th St | \n","San Francisco | \n","CA | \n","94114 | \n","[] | \n","Joel Gile | \n","Sequoia Real Estate | \n","
| 26 | \n","1650000 | \n","3 | \n","2 | \n","1547 | \n","2475 47th Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Lucy Goldenshteyn | \n","Redfin | \n","
| 27 | \n","998000 | \n","2 | \n","2 | \n","1202 | \n","50 Lansing St Unit 201 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Tracey Broadman | \n","Vanguard Properties | \n","
| 28 | \n","1595000 | \n","3 | \n","5 | \n","1995 | \n","15 Joy St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Mike Stack | \n","Vanguard Properties | \n","
| 29 | \n","1028000 | \n","2 | \n","2 | \n","1065 | \n","50 Lansing St Unit 403 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Robyn Kaufman | \n","Vivre Real Estate | \n","
| 30 | \n","999000 | \n","1 | \n","1 | \n","1021 | \n","338 Spear St Unit 6J | \n","San Francisco | \n","CA | \n","94105 | \n","[Spacious, Balcony, Bright courtyard views] | \n","Paul Hwang | \n","Skybox Realty | \n","
| 31 | \n","799800 | \n","2 | \n","2 | \n","1109 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 32 | \n","529880 | \n","1 | \n","1 | \n","740 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 33 | \n","489000 | \n","1 | \n","1 | \n","741 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 34 | \n","1359000 | \n","4 | \n","2 | \n","1845 | \n","170 Thrift St | \n","San Francisco | \n","CA | \n","94112 | \n","[Updated, Single-family home] | \n","Cristal Wright | \n","Vanguard Properties | \n","
| 35 | \n","1295000 | \n","3 | \n","1 | \n","1214 | \n","1922 43rd Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Mila Romprey | \n","Premier Realty Associates | \n","
| 36 | \n","1098000 | \n","3 | \n","1 | \n","1006 | \n","150 Putnam St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Genie Mantzoros | \n","Epic Real Estate & Asso. Inc. | \n","
| 37 | \n","1189870 | \n","3 | \n","2 | \n","1436 | \n","327 Ordway St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Shawn Zahraie | \n","Affinity Enterprises, Inc | \n","
| 38 | \n","899000 | \n","2 | \n","1 | \n","1118 | \n","272 Farallones St | \n","San Francisco | \n","CA | \n","94112 | \n","[] | \n","Janice Lee | \n","Coldwell Banker Realty | \n","
| 39 | \n","30000 | \n","0 | \n","0 | \n","0 | \n","0 Evans Ave | \n","San Francisco | \n","CA | \n","94124 | \n","[Land, 0.12 Acre, $251,467 per Acre] | \n","Heidy Carrera | \n","Berkshire Hathaway HomeService | \n","
\n"," \n","
| \n", + " | price | \n", + "bedrooms | \n", + "bathrooms | \n", + "square_feet | \n", + "address | \n", + "city | \n", + "state | \n", + "zip_code | \n", + "tags | \n", + "agent_name | \n", + "agency | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "549000 | \n", + "1 | \n", + "1 | \n", + "477 | \n", + "380 14th St Unit 405 | \n", + "San Francisco | \n", + "CA | \n", + "94103 | \n", + "[New construction] | \n", + "Eddie O'Sullivan | \n", + "Elevation Real Estate | \n", + "
| 1 | \n", + "1799000 | \n", + "4 | \n", + "2 | \n", + "2735 | \n", + "123 Grattan St | \n", + "San Francisco | \n", + "CA | \n", + "94117 | \n", + "[] | \n", + "Sean Engmann | \n", + "eXp Realty of Northern CA Inc. | \n", + "
| 2 | \n", + "1995000 | \n", + "7 | \n", + "3 | \n", + "3330 | \n", + "1590 Washington St | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Eddie O'Sullivan | \n", + "Elevation Real Estate | \n", + "
| 3 | \n", + "549000 | \n", + "0 | \n", + "1 | \n", + "477 | \n", + "240 Lombard St Unit 835 | \n", + "San Francisco | \n", + "CA | \n", + "94111 | \n", + "[] | \n", + "Tim Gullicksen | \n", + "Corcoran Icon Properties | \n", + "
| 4 | \n", + "5495000 | \n", + "10 | \n", + "7 | \n", + "6505 | \n", + "1057 Steiner St | \n", + "San Francisco | \n", + "CA | \n", + "94115 | \n", + "[] | \n", + "Bonnie Spindler | \n", + "Corcoran Icon Properties | \n", + "
| 5 | \n", + "925000 | \n", + "2 | \n", + "1 | \n", + "779 | \n", + "2 Fallon Place Unit 57 | \n", + "San Francisco | \n", + "CA | \n", + "94133 | \n", + "[] | \n", + "Eddie O'Sullivan | \n", + "Elevation Real Estate | \n", + "
| 6 | \n", + "898000 | \n", + "2 | \n", + "2 | \n", + "1175 | \n", + "5160 Diamond Heights Blvd Unit 208C | \n", + "San Francisco | \n", + "CA | \n", + "94131 | \n", + "[] | \n", + "Joe Polyak | \n", + "Rise Homes | \n", + "
| 7 | \n", + "1700000 | \n", + "4 | \n", + "2 | \n", + "1950 | \n", + "1351 26th Ave | \n", + "San Francisco | \n", + "CA | \n", + "94122 | \n", + "[] | \n", + "Glenda Queensbury | \n", + "Referral Realty-BV | \n", + "
| 8 | \n", + "1899000 | \n", + "3 | \n", + "2 | \n", + "1560 | \n", + "340 Yerba Buena Ave | \n", + "San Francisco | \n", + "CA | \n", + "94127 | \n", + "[] | \n", + "Jeannie Anderson | \n", + "Coldwell Banker Realty | \n", + "
| 9 | \n", + "850000 | \n", + "2 | \n", + "2 | \n", + "1055 | \n", + "588 Minna Unit 604 | \n", + "San Francisco | \n", + "CA | \n", + "94103 | \n", + "[] | \n", + "Mohamed Lakdawala | \n", + "Remax Prestigious Properties | \n", + "
| 10 | \n", + "1990000 | \n", + "3 | \n", + "1 | \n", + "1280 | \n", + "1450 Diamond St | \n", + "San Francisco | \n", + "CA | \n", + "94131 | \n", + "[] | \n", + "Mary Anne Villamil | \n", + "Kinetic Real Estate | \n", + "
| 11 | \n", + "849000 | \n", + "1 | \n", + "1 | \n", + "855 | \n", + "81 Lansing St Unit 401 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[] | \n", + "Kristen Haenggi | \n", + "Compass | \n", + "
| 12 | \n", + "1080000 | \n", + "2 | \n", + "2 | \n", + "936 | \n", + "451 Kansas St Unit 466 | \n", + "San Francisco | \n", + "CA | \n", + "94107 | \n", + "[] | \n", + "Maureen DeBoer | \n", + "LKJ Realty | \n", + "
| 13 | \n", + "1499000 | \n", + "4 | \n", + "2 | \n", + "2145 | \n", + "486 Yale St | \n", + "San Francisco | \n", + "CA | \n", + "94134 | \n", + "[] | \n", + "Alicia Atienza | \n", + "Statewide Realty | \n", + "
| 14 | \n", + "1140000 | \n", + "2 | \n", + "2 | \n", + "998 | \n", + "588 Minna Unit 801 | \n", + "San Francisco | \n", + "CA | \n", + "94103 | \n", + "[] | \n", + "Milan Jezdimirovic | \n", + "Compass | \n", + "
| 15 | \n", + "1988000 | \n", + "2 | \n", + "1 | \n", + "3800 | \n", + "183 19th Ave | \n", + "San Francisco | \n", + "CA | \n", + "94121 | \n", + "[Amazing Property, Marina Style, Needs TLC] | \n", + "Leo Cheung | \n", + "eXp Realty of California, Inc | \n", + "
| 16 | \n", + "1218000 | \n", + "2 | \n", + "2 | \n", + "1275 | \n", + "1998 Pacific Ave Unit 202 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[Light-filled, Freshly painted, Walker's parad... | \n", + "Grace Sun | \n", + "Compass | \n", + "
| 17 | \n", + "895000 | \n", + "1 | \n", + "1 | \n", + "837 | \n", + "425 1st St Unit 2501 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[Unobstructed bay bridge views, Open layout] | \n", + "Matt Fuller | \n", + "Jackson Fuller Real Estate | \n", + "
| 18 | \n", + "1499000 | \n", + "3 | \n", + "1 | \n", + "1500 | \n", + "Unlisted Address | \n", + "San Francisco | \n", + "CA | \n", + "NA | \n", + "[Contractor's Special, Fixer-upper] | \n", + "Jaymee Faith Sagisi | \n", + "IMPACT | \n", + "
| 19 | \n", + "900000 | \n", + "1 | \n", + "1 | \n", + "930 | \n", + "1101 Green St Unit 302 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[Historic Art Deco, Iconic views] | \n", + "NA | \n", + "NA | \n", + "
| 20 | \n", + "858000 | \n", + "1 | \n", + "1 | \n", + "1104 | \n", + "260 King St Unit 557 | \n", + "San Francisco | \n", + "CA | \n", + "94107 | \n", + "[] | \n", + "Miyuki Takami | \n", + "eXp Realty of California, Inc | \n", + "
| 21 | \n", + "945000 | \n", + "2 | \n", + "1 | \n", + "767 | \n", + "307 Page St Unit 1 | \n", + "San Francisco | \n", + "CA | \n", + "94102 | \n", + "[] | \n", + "NA | \n", + "NA | \n", + "
| 22 | \n", + "1099000 | \n", + "2 | \n", + "2 | \n", + "1330 | \n", + "1080 Sutter St Unit 202 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Annette Liberty | \n", + "Coldwell Banker Realty | \n", + "
| 23 | \n", + "950000 | \n", + "4 | \n", + "3 | \n", + "2090 | \n", + "3328 26th St Unit 3330 | \n", + "San Francisco | \n", + "CA | \n", + "94110 | \n", + "[] | \n", + "Isaac Munene | \n", + "Coldwell Banker Realty | \n", + "
| 24 | \n", + "1088000 | \n", + "2 | \n", + "2 | \n", + "1065 | \n", + "1776 Sacramento St Unit 503 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Marilyn Becklehimer | \n", + "Dio Real Estate | \n", + "
| 25 | \n", + "1788888 | \n", + "4 | \n", + "3 | \n", + "1856 | \n", + "2317 15th St | \n", + "San Francisco | \n", + "CA | \n", + "94114 | \n", + "[] | \n", + "Joel Gile | \n", + "Sequoia Real Estate | \n", + "
| 26 | \n", + "1650000 | \n", + "3 | \n", + "2 | \n", + "1547 | \n", + "2475 47th Ave | \n", + "San Francisco | \n", + "CA | \n", + "94116 | \n", + "[] | \n", + "Lucy Goldenshteyn | \n", + "Redfin | \n", + "
| 27 | \n", + "998000 | \n", + "2 | \n", + "2 | \n", + "1202 | \n", + "50 Lansing St Unit 201 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[] | \n", + "Tracey Broadman | \n", + "Vanguard Properties | \n", + "
| 28 | \n", + "1595000 | \n", + "3 | \n", + "5 | \n", + "1995 | \n", + "15 Joy St | \n", + "San Francisco | \n", + "CA | \n", + "94110 | \n", + "[] | \n", + "Mike Stack | \n", + "Vanguard Properties | \n", + "
| 29 | \n", + "1028000 | \n", + "2 | \n", + "2 | \n", + "1065 | \n", + "50 Lansing St Unit 403 | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[] | \n", + "Robyn Kaufman | \n", + "Vivre Real Estate | \n", + "
| 30 | \n", + "999000 | \n", + "1 | \n", + "1 | \n", + "1021 | \n", + "338 Spear St Unit 6J | \n", + "San Francisco | \n", + "CA | \n", + "94105 | \n", + "[Spacious, Balcony, Bright courtyard views] | \n", + "Paul Hwang | \n", + "Skybox Realty | \n", + "
| 31 | \n", + "799800 | \n", + "2 | \n", + "2 | \n", + "1109 | \n", + "10 Innes Ct | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[New Construction] | \n", + "Lennar | \n", + "Lennar | \n", + "
| 32 | \n", + "529880 | \n", + "1 | \n", + "1 | \n", + "740 | \n", + "10 Innes Ct | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[New Construction] | \n", + "Lennar | \n", + "Lennar | \n", + "
| 33 | \n", + "489000 | \n", + "1 | \n", + "1 | \n", + "741 | \n", + "10 Innes Ct | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[New Construction] | \n", + "Lennar | \n", + "Lennar | \n", + "
| 34 | \n", + "1359000 | \n", + "4 | \n", + "2 | \n", + "1845 | \n", + "170 Thrift St | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[Updated, Single-family home] | \n", + "Cristal Wright | \n", + "Vanguard Properties | \n", + "
| 35 | \n", + "1295000 | \n", + "3 | \n", + "1 | \n", + "1214 | \n", + "1922 43rd Ave | \n", + "San Francisco | \n", + "CA | \n", + "94116 | \n", + "[] | \n", + "Mila Romprey | \n", + "Premier Realty Associates | \n", + "
| 36 | \n", + "1098000 | \n", + "3 | \n", + "1 | \n", + "1006 | \n", + "150 Putnam St | \n", + "San Francisco | \n", + "CA | \n", + "94110 | \n", + "[] | \n", + "Genie Mantzoros | \n", + "Epic Real Estate & Asso. Inc. | \n", + "
| 37 | \n", + "1189870 | \n", + "3 | \n", + "2 | \n", + "1436 | \n", + "327 Ordway St | \n", + "San Francisco | \n", + "CA | \n", + "94134 | \n", + "[] | \n", + "Shawn Zahraie | \n", + "Affinity Enterprises, Inc | \n", + "
| 38 | \n", + "899000 | \n", + "2 | \n", + "1 | \n", + "1118 | \n", + "272 Farallones St | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[] | \n", + "Janice Lee | \n", + "Coldwell Banker Realty | \n", + "
| 39 | \n", + "30000 | \n", + "0 | \n", + "0 | \n", + "0 | \n", + "0 Evans Ave | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[Land, 0.12 Acre, $251,467 per Acre] | \n", + "Heidy Carrera | \n", + "Berkshire Hathaway HomeService | \n", + "
\n \n
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "# Scraping agent for ecommerce\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/crewai-integration) [](https://colab.research.google.com/drive/1oqGzVer_XTWOJq88tpcPyQ0B72l38YUL?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "# Scraping agent for ecommerce\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/crewai-integration) [](https://colab.research.google.com/drive/1oqGzVer_XTWOJq88tpcPyQ0B72l38YUL?usp=sharing)",
+ "text/markdown": "# Scraping agent for ecommerce\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/crewai-integration) [](https://colab.research.google.com/drive/1oqGzVer_XTWOJq88tpcPyQ0B72l38YUL?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "XjbB5NvzdB0j"
},
- "source": [
- "# Scraping agent for ecommerce"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:08:56.908Z",
+ "executionStartTime": "2026-03-26T00:08:56.908Z"
},
{
- "cell_type": "markdown",
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "3DHEIAp316DR"
},
- "source": [
- ""
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "## Installation of the required libraries",
+ "outputs": [],
"metadata": {
"id": "wPQ3ZmBjc-Mz"
},
- "source": [
- "## Installation of the required libraries"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "%%capture\n!pip install crewai\n!pip install crewai-tools",
+ "outputs": [],
"metadata": {
"id": "m9mtKbuacveq"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install crewai\n",
- "!pip install crewai-tools"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "## Environment configuration",
+ "outputs": [],
"metadata": {
"id": "ws1ueZZ9djul"
},
- "source": [
- "## Environment configuration"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "_XTTnMmZdnhz",
- "outputId": "dcc56a3b-7bc3-4e0e-d057-310e1e98b9d8"
- },
+ "source": "import os\nfrom getpass import getpass\n\n# Check if the API key is already set in the environment\nsgai_api_key = os.getenv(\"SCRAPEGRAPH_API_KEY\")\n\nif sgai_api_key:\n print(\"SCRAPEGRAPH_API_KEY found in environment.\")\nelse:\n print(\"SCRAPEGRAPH_API_KEY not found in environment.\")\n # Prompt the user to input the API key securely (hidden input)\n sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n if sgai_api_key:\n # Set the API key in the environment\n os.environ[\"SCRAPEGRAPH_API_KEY\"] = sgai_api_key\n print(\"SCRAPEGRAPH_API_KEY has been set in the environment.\")\n else:\n print(\"No API key entered. Please set the API key to continue.\")",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"SCRAPEGRAPH_API_KEY found in environment.\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "import os\n",
- "from getpass import getpass\n",
- "\n",
- "# Check if the API key is already set in the environment\n",
- "sgai_api_key = os.getenv(\"SCRAPEGRAPH_API_KEY\")\n",
- "\n",
- "if sgai_api_key:\n",
- " print(\"SCRAPEGRAPH_API_KEY found in environment.\")\n",
- "else:\n",
- " print(\"SCRAPEGRAPH_API_KEY not found in environment.\")\n",
- " # Prompt the user to input the API key securely (hidden input)\n",
- " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
- " if sgai_api_key:\n",
- " # Set the API key in the environment\n",
- " os.environ[\"SCRAPEGRAPH_API_KEY\"] = sgai_api_key\n",
- " print(\"SCRAPEGRAPH_API_KEY has been set in the environment.\")\n",
- " else:\n",
- " print(\"No API key entered. Please set the API key to continue.\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
"metadata": {
+ "id": "_XTTnMmZdnhz",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "pbWasAqodoZG",
- "outputId": "06f75acf-c0be-4fdd-e7b7-9f19ce4a156a"
+ "outputId": "dcc56a3b-7bc3-4e0e-d057-310e1e98b9d8"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
+ },
+ {
+ "source": "# Check if the API key is already set in the environment\nsgai_api_key = os.getenv(\"OPENAI_API_KEY\")\n\nif sgai_api_key:\n print(\"OPENAI_API_KEY found in environment.\")\nelse:\n print(\"OPENAI_API_KEY not found in environment.\")\n # Prompt the user to input the API key securely (hidden input)\n sgai_api_key = getpass(\"Please enter your OPENAI_API_KEY: \").strip()\n if sgai_api_key:\n # Set the API key in the environment\n os.environ[\"OPENAI_API_KEY\"] = sgai_api_key\n print(\"OPENAI_API_KEY has been set in the environment.\")\n else:\n print(\"No API key entered. Please set the API key to continue.\")",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"OPENAI_API_KEY found in environment.\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "# Check if the API key is already set in the environment\n",
- "sgai_api_key = os.getenv(\"OPENAI_API_KEY\")\n",
- "\n",
- "if sgai_api_key:\n",
- " print(\"OPENAI_API_KEY found in environment.\")\n",
- "else:\n",
- " print(\"OPENAI_API_KEY not found in environment.\")\n",
- " # Prompt the user to input the API key securely (hidden input)\n",
- " sgai_api_key = getpass(\"Please enter your OPENAI_API_KEY: \").strip()\n",
- " if sgai_api_key:\n",
- " # Set the API key in the environment\n",
- " os.environ[\"OPENAI_API_KEY\"] = sgai_api_key\n",
- " print(\"OPENAI_API_KEY has been set in the environment.\")\n",
- " else:\n",
- " print(\"No API key entered. Please set the API key to continue.\")"
- ]
+ "metadata": {
+ "id": "pbWasAqodoZG",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "06f75acf-c0be-4fdd-e7b7-9f19ce4a156a"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "## Imports",
+ "outputs": [],
"metadata": {
"id": "Dpb7akE3dUjX"
},
- "source": [
- "## Imports"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "from crewai import Agent, Crew, Process, Task\nfrom crewai_tools import ScrapegraphScrapeTool",
+ "outputs": [],
"metadata": {
"id": "2vOrmP3GdWJt"
},
- "outputs": [],
- "source": [
- "from crewai import Agent, Crew, Process, Task\n",
- "from crewai_tools import ScrapegraphScrapeTool"
- ]
- },
- {
"cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
"execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "lyC3gQWxeGVh",
- "outputId": "764eed43-2558-4e77-8fe5-e7280edda2d2"
- },
+ "executionEndTime": null,
+ "executionStartTime": null
+ },
+ {
+ "source": "website = \"https://www.ebay.com/sch/i.html?_nkw=keyboard&_sacat=0&_from=R40&_trksid=p4432023.m570.l1313\"\ntool = ScrapegraphScrapeTool()\n\nagent = Agent(\n role=\"Web Researcher\",\n goal=\"Research and extract accurate information from websites\",\n backstory=\"You are an expert web researcher with experience in extracting and analyzing information from various websites.\",\n tools=[tool],\n)\n\ntask = Task(\n name=\"scraping task\",\n description=f\"Visit the website {website} and extract detailed information about all the keyboards available.\",\n expected_output=\"A file with the informations extracted from the website.\",\n agent=agent,\n)\n\ncrew = Crew(\n agents=[agent],\n tasks=[task],\n)\n\nres = crew.kickoff()",
"outputs": [
{
"name": "stderr",
- "output_type": "stream",
"text": [
"WARNING:opentelemetry.trace:Overriding of current TracerProvider is not allowed\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "website = \"https://www.ebay.com/sch/i.html?_nkw=keyboard&_sacat=0&_from=R40&_trksid=p4432023.m570.l1313\"\n",
- "tool = ScrapegraphScrapeTool()\n",
- "\n",
- "agent = Agent(\n",
- " role=\"Web Researcher\",\n",
- " goal=\"Research and extract accurate information from websites\",\n",
- " backstory=\"You are an expert web researcher with experience in extracting and analyzing information from various websites.\",\n",
- " tools=[tool],\n",
- ")\n",
- "\n",
- "task = Task(\n",
- " name=\"scraping task\",\n",
- " description=f\"Visit the website {website} and extract detailed information about all the keyboards available.\",\n",
- " expected_output=\"A file with the informations extracted from the website.\",\n",
- " agent=agent,\n",
- ")\n",
- "\n",
- "crew = Crew(\n",
- " agents=[agent],\n",
- " tasks=[task],\n",
- ")\n",
- "\n",
- "res = crew.kickoff()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
"metadata": {
+ "id": "lyC3gQWxeGVh",
"colab": {
- "base_uri": "https://localhost:8080/",
- "height": 137
+ "base_uri": "https://localhost:8080/"
},
- "id": "fiPH4Ta8JnCO",
- "outputId": "4b9d7fb0-511e-4158-de52-e5b549ca8f5e"
+ "outputId": "764eed43-2558-4e77-8fe5-e7280edda2d2"
},
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
+ },
+ {
+ "source": "res.raw",
"outputs": [
{
"data": {
- "application/vnd.google.colaboratory.intrinsic+json": {
- "type": "string"
- },
"text/plain": [
"'{\\n \"keyboards\": [\\n {\\n \"title\": \"Teclado para juegos SteelSeries Apex Pro TKL Gen 3 con cable certificado restaurado\",\\n \"price\": \"USD 164.99\",\\n \"seller_information\": {\\n \"seller_name\": \"gamesngadgetsplus\",\\n \"ratings\": 3580,\\n \"positive_feedback\": \"99.1%\"\\n },\\n \"item_specifics\": {\\n \"condition\": \"Restored\",\\n \"shipping_cost\": \"USD 63.74\",\\n \"shipping_origin\": \"United States\"\\n },\\n \"link\": \"https://www.ebay.com/itm/146284292952?_skw=keyboard&epid=14072039453&itmmeta=01JHZ2XD60HFYG3H2SGW3ZPD20&hash=item220f392b58:g:5NoAAOSw7ytnZb8d&itmprp=enc%3AAQAJAAAA4HoV3kP08IDx%2BKZ9MfhVJKnJnAISV9LpbsB4qVE8BPDpZj1d0167Ko%2FPBy31lCqIBYnInLHGEdjeLgwAL02glnIeEEqKoS8V3y38FHo2UIjJyTsghnHunyefi%2FEBvMC2cLNkeLXhfAORbJw3MAn4akA698gi%2FDHZRVsyYFpHdTlbNnmDWCY0Q9P9Fwen%2BFtKEK%2BFBGll8l%2F%2BfDJ1U5bOAA8iV65K2VaglTrW4CbrLvIABLQFhJAgq4wik7AiQpnRDahukAMx%2BX3VRfl8iGPeGPZ0WXJ93qe41x8cGWgH0ljL%7Ctkp%3ABFBMytP14o9l\"\\n },\\n {\\n \"title\": \"Teclado electrónico portátil de 61 teclas para todos los niveles (sin soporte)\",\\n \"price\": \"Not specified\",\\n \"seller_information\": {\\n \"seller_name\": \"Not specified\",\\n \"ratings\": \"Not specified\",\\n \"positive_feedback\": \"Not specified\"\\n },\\n \"item_specifics\": {\\n \"condition\": \"Not specified\",\\n \"shipping_cost\": \"Not specified\",\\n \"shipping_origin\": \"Not specified\"\\n },\\n \"link\": \"https://www.ebay.com/itm/116374921423?_skw=keyboard&itmmeta=01JHZ2XD60HFBCXZGCTNGMSD5V&hash=item1b187c60cf:g:PosAAOSwO~pnIUCg&itmprp=enc%3AAQAJAAAA4HoV3kP08IDx%2BKZ9MfhVJKmMPWqDzISoc%2Fy0NEb29gCBnD2SpusvA4MSMxa35GGERgjG9%2F6sPB22PedK5%2F1LDxZYJxeqhSv646gq6mzx0ZBojH%2BEibOOCTplRqmprWLDLC6HUF16BntMg6fgfm0Q6F4OvFJ1CqEgvY6uz2HQ759uu%2F2yNkR1WiVurWxkd1HRZZ1kvI6r3XrsTvwVQnSa%2F6SWbqbe0IYP8%2FN8ohPwBzLez7IxzcWbVbFWk31XtVDH0%2Bio5dT6ZWgofYotfUUQUuYjCwnuLCXR0BDwXX1BX4\"\\n },\\n {\\n \"title\": \"Teclado electrónico portátil de 61 teclas\",\\n \"price\": \"USD 49.99 (or Best Offer)\",\\n \"seller_information\": {\\n \"seller_name\": \"elanza771\",\\n \"ratings\": 4076,\\n \"positive_rating_percentage\": 95.4\\n },\\n \"item_specifics\": \"Totally new\"\\n },\\n // More items similar to above...\\n ]\\n}'"
- ]
+ ],
+ "application/vnd.google.colaboratory.intrinsic+json": {
+ "type": "string"
+ }
},
- "execution_count": 23,
"metadata": {},
- "output_type": "execute_result"
+ "output_type": "execute_result",
+ "execution_count": 23
}
],
- "source": [
- "res.raw"
- ]
+ "metadata": {
+ "id": "fiPH4Ta8JnCO",
+ "colab": {
+ "height": 137,
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "4b9d7fb0-511e-4158-de52-e5b549ca8f5e"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
}
],
"metadata": {
@@ -237,8 +224,8 @@
"provenance": []
},
"kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
+ "name": "python3",
+ "display_name": "Python 3"
},
"language_info": {
"name": "python"
@@ -246,4 +233,4 @@
},
"nbformat": 4,
"nbformat_minor": 0
-}
+}
\ No newline at end of file
diff --git a/cookbook/research-agent/llama_index_agent.ipynb b/cookbook/research-agent/llama_index_agent.ipynb
index bb22c21..8b5f926 100644
--- a/cookbook/research-agent/llama_index_agent.ipynb
+++ b/cookbook/research-agent/llama_index_agent.ipynb
@@ -1,264 +1,239 @@
{
"cells": [
{
- "cell_type": "markdown",
- "metadata": {
- "id": "ReBHQ5_834pZ"
- },
- "source": [
- "\n",
- " \n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Extract Keyboard prices with llama-index and ScrapegraphAI APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/llama-index-agent) [](https://colab.research.google.com/drive/1oqGzVer_XTWOJq88tpcPyQ0B72l38YUL?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Extract Keyboard prices with llama-index and ScrapegraphAI APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/llama-index-agent) [](https://colab.research.google.com/drive/1oqGzVer_XTWOJq88tpcPyQ0B72l38YUL?usp=sharing)",
+ "text/markdown": "## 🕷️ Extract Keyboard prices with llama-index and ScrapegraphAI APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/llama-index-agent) [](https://colab.research.google.com/drive/1oqGzVer_XTWOJq88tpcPyQ0B72l38YUL?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Extract Keyboard prices with llama-index and ScrapegraphAI APIs"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:14:21.547Z",
+ "executionStartTime": "2026-03-26T00:14:21.540Z"
},
{
- "cell_type": "markdown",
+ "source": "\n \n
\n",
- " \n",
- "
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, and `tavily`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-langgraph-tavily) [](https://colab.research.google.com/drive/1lk4cjuRLDN0w71kXm9uAPQM6jH03yGW9?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, and `tavily`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-langgraph-tavily) [](https://colab.research.google.com/drive/1lk4cjuRLDN0w71kXm9uAPQM6jH03yGW9?usp=sharing)",
+ "text/markdown": "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, and `tavily`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-langgraph-tavily) [](https://colab.research.google.com/drive/1lk4cjuRLDN0w71kXm9uAPQM6jH03yGW9?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, and `tavily`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:07:22.280Z",
+ "executionStartTime": "2026-03-26T00:07:22.280Z"
},
{
- "cell_type": "markdown",
- "source": [
- ""
- ],
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "cJrlyZbLwQek"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 1,
+ "source": "%%capture\n!pip install langgraph langchain-scrapegraph langchain-openai \"langchain-community>=0.2.11\" tavily-python",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install langgraph langchain-scrapegraph langchain-openai \"langchain-community>=0.2.11\" tavily-python"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 1,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph`, `Tavily` and `OpenAI` API keys",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph`, `Tavily` and `OpenAI` API keys"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "id": "sffqFG2EJ8bI",
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "outputId": "7e47aa4d-edca-48f2-df6e-bd7fafabfc9c"
- },
+ "source": "import getpass\nimport os\n\nif not os.environ.get(\"SGAI_API_KEY\"):\n os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n\nif not os.environ.get(\"TAVILY_API_KEY\"):\n os.environ[\"TAVILY_API_KEY\"] = getpass.getpass(\"Tavily API key:\\n\")\n\nif not os.environ.get(\"OPENAI_API_KEY\"):\n os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"Scrapegraph API key:\n",
"··········\n",
@@ -89,322 +105,227 @@
"··········\n",
"OpenAI API key:\n",
"··········\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "import getpass\n",
- "import os\n",
- "\n",
- "if not os.environ.get(\"SGAI_API_KEY\"):\n",
- " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n",
- "\n",
- "if not os.environ.get(\"TAVILY_API_KEY\"):\n",
- " os.environ[\"TAVILY_API_KEY\"] = getpass.getpass(\"Tavily API key:\\n\")\n",
- "\n",
- "if not os.environ.get(\"OPENAI_API_KEY\"):\n",
- " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"
- ]
+ "metadata": {
+ "id": "sffqFG2EJ8bI",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "7e47aa4d-edca-48f2-df6e-bd7fafabfc9c"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 2,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n",
+ "outputs": [],
"metadata": {
"id": "jnqMB2-xVYQ7"
},
- "source": [
- "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n\n\n \n
\n",
- " \n",
- "
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, `tavily` and `MySQL`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-langgraph-tavily-mysql) [](https://colab.research.google.com/drive/1uFyXOLLibGQKqvr8K7jaG_7S3V6oits2?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, `tavily` and `MySQL`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-langgraph-tavily-mysql) [](https://colab.research.google.com/drive/1uFyXOLLibGQKqvr8K7jaG_7S3V6oits2?usp=sharing)",
+ "text/markdown": "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, `tavily` and `MySQL`\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-langgraph-tavily-mysql) [](https://colab.research.google.com/drive/1uFyXOLLibGQKqvr8K7jaG_7S3V6oits2?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, `tavily` and `MySQL`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:13:02.867Z",
+ "executionStartTime": "2026-03-26T00:13:02.866Z"
},
{
- "cell_type": "markdown",
- "source": [
- ""
- ],
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "-GmCGaVI3kcT"
- }
+ },
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 1,
+ "source": "%%capture\n!pip install langgraph langchain-scrapegraph langchain-openai \"langchain-community>=0.2.11\" tavily-python",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install langgraph langchain-scrapegraph langchain-openai \"langchain-community>=0.2.11\" tavily-python"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 1,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph`, `Tavily` and `OpenAI` API keys",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph`, `Tavily` and `OpenAI` API keys"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "sffqFG2EJ8bI",
- "outputId": "132bdfb9-45c8-4c26-8ac2-d13593ba392a"
- },
+ "source": "import getpass\nimport os\n\nif not os.environ.get(\"SGAI_API_KEY\"):\n os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n\nif not os.environ.get(\"TAVILY_API_KEY\"):\n os.environ[\"TAVILY_API_KEY\"] = getpass.getpass(\"Tavily API key:\\n\")\n\nif not os.environ.get(\"OPENAI_API_KEY\"):\n os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")",
"outputs": [
{
"name": "stdout",
- "output_type": "stream",
"text": [
"Scrapegraph API key:\n",
"··········\n",
@@ -89,314 +105,227 @@
"··········\n",
"OpenAI API key:\n",
"··········\n"
- ]
+ ],
+ "output_type": "stream"
}
],
- "source": [
- "import getpass\n",
- "import os\n",
- "\n",
- "if not os.environ.get(\"SGAI_API_KEY\"):\n",
- " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n",
- "\n",
- "if not os.environ.get(\"TAVILY_API_KEY\"):\n",
- " os.environ[\"TAVILY_API_KEY\"] = getpass.getpass(\"Tavily API key:\\n\")\n",
- "\n",
- "if not os.environ.get(\"OPENAI_API_KEY\"):\n",
- " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"
- ]
+ "metadata": {
+ "id": "sffqFG2EJ8bI",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "132bdfb9-45c8-4c26-8ac2-d13593ba392a"
+ },
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": 2,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n",
+ "outputs": [],
"metadata": {
"id": "jnqMB2-xVYQ7"
},
- "source": [
- "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n\n\n \n
\n",
- " \n",
- "
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Wired Science News with `langchain-scrapegraph` and `langgraph`"]},{"cell_type":"markdown","source":[""],"metadata":{"id":"FxtXj1Qtx3zH"}},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"os_vm0MkIxr9","executionInfo":{"status":"ok","timestamp":1734786986883,"user_tz":-60,"elapsed":13752,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}}},"outputs":[],"source":["%%capture\n","!pip install langgraph langchain-scrapegraph langchain-openai"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` and `OpenAI` API keys"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":67210,"status":"ok","timestamp":1734787067908,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"},"user_tz":-60},"id":"sffqFG2EJ8bI","outputId":"8c3dd34f-22d5-4557-c562-97728738726b"},"outputs":[{"name":"stdout","output_type":"stream","text":["Scrapegraph API key:\n","··········\n","OpenAI API key:\n","··········\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n","\n","if not os.environ.get(\"OPENAI_API_KEY\"):\n"," os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | title | \n","link | \n","author | \n","
|---|---|---|---|
| 0 | \n","December Wildfires Are Now a Thing | \n","https://www.wired.com/story/december-wildfires... | \n","Kylie Mohr | \n","
| 1 | \n","How to Manage Food Anxiety Over the Holidays | \n","https://www.wired.com/story/how-to-cope-with-f... | \n","Alison Fixsen | \n","
| 2 | \n","A Spacecraft Is About to Fly Into the Sun’s At... | \n","https://www.wired.com/story/parker-solar-probe... | \n","Eric Berger, Ars Technica | \n","
| 3 | \n","To Improve Your Gut Microbiome, Spend More Tim... | \n","https://www.wired.com/story/to-improve-your-gu... | \n","Kathy Willis | \n","
| 4 | \n","This Tropical Virus Is Spreading Out of the Am... | \n","https://www.wired.com/story/this-tropical-viru... | \n","Geraldine Castro | \n","
| 5 | \n","CDC Confirms First US Case of Severe Bird Flu | \n","https://www.wired.com/story/cdc-confirms-first... | \n","Emily Mullin | \n","
| 6 | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 7 | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
| 10 | \n","Environmental Sensing Is Here, Tracking Everyt... | \n","https://www.wired.com/story/environmental-sens... | \n","Sabrina Weiss | \n","
| 11 | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 12 | \n","Climate Change Is Destroying Monarch Butterfli... | \n","https://www.wired.com/story/global-warming-thr... | \n","Andrea J. Arratibel | \n","
| 13 | \n","More Humanitarian Organizations Will Harness A... | \n","https://www.wired.com/story/humanitarian-organ... | \n","David Miliband | \n","
| 14 | \n","Chocolate Has a Sustainability Problem. Scienc... | \n","https://www.wired.com/story/chocolate-has-a-su... | \n","Eve Thomas | \n","
| 15 | \n","We’ve Never Been Closer to Finding Life Outsid... | \n","https://www.wired.com/story/james-webb-space-t... | \n","Lisa Kaltenegger | \n","
| 16 | \n","The End Is Near for NASA’s Voyager Probes | \n","https://www.wired.com/story/the-end-is-near-fo... | \n","Luca Nardi | \n","
| 17 | \n","Why Can’t You Switch Seats in an Empty Airplane? | \n","https://www.wired.com/story/why-cant-you-switc... | \n","Rhett Allain | \n","
| 18 | \n","The Simple Math Behind Public Key Cryptography | \n","https://www.wired.com/story/how-public-key-cry... | \n","John Pavlus | \n","
| 19 | \n","Everyone Is Capable of Mathematical Thinking--... | \n","https://www.wired.com/story/everyone-is-capabl... | \n","Kelsey Houston-Edwards | \n","
| 20 | \n","The Physics of the Macy’s Thanksgiving Day Par... | \n","https://www.wired.com/story/the-physics-of-the... | \n","Rhett Allain | \n","
| 21 | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 22 | \n","Muscle Implants Could Allow Mind-Controlled Pr... | \n","https://www.wired.com/story/amputees-could-con... | \n","Emily Mullin | \n","
| 23 | \n","Combining AI and Crispr Will Be Transformational | \n","https://www.wired.com/story/combining-ai-and-c... | \n","Jennifer Doudna | \n","
| 24 | \n","Neuralink Plans to Test Whether Its Brain Impl... | \n","https://www.wired.com/story/neuralink-robotic-... | \n","Emily Mullin | \n","
| 25 | \n","Eight Scientists, a Billion Dollars, and the M... | \n","https://www.wired.com/story/aria-moonshot-darp... | \n","Matt Reynolds | \n","
| 26 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","NA | \n","
| 27 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","Carlton Reid | \n","
| 28 | \n","Meet the Next Generation of Doctors--and Their... | \n","https://www.wired.com/story/next-generation-do... | \n","Neha Mukherjee | \n","
| 29 | \n","AI Is Building Highly Effective Antibodies Tha... | \n","https://www.wired.com/story/labgenius-antibody... | \n","Amit Katwala | \n","
| 30 | \n","An Uncertain Future Requires Uncertain Predict... | \n","https://www.wired.com/story/embrace-uncertaint... | \n","David Spiegelhalter | \n","
| 31 | \n","These Rats Learned to Drive--and They Love It | \n","https://www.wired.com/story/these-rats-learned... | \n","Kelly Lambert | \n","
| 32 | \n","Scientists Are Unlocking the Secrets of Your ‗... | \n","https://www.wired.com/story/cerebellum-brain-m... | \n","R Douglas Fields | \n","
| 33 | \n","Meet the Designer Behind Neuralink’s Surgical ... | \n","https://www.wired.com/story/designer-behind-ne... | \n","Emily Mullin | \n","
| 34 | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 35 | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 36 | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 37 | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 38 | \n","Designer Babies Are Teenagers Now--and Some of... | \n","https://www.wired.com/story/your-next-job-desi... | \n","Emi Nietfeld | \n","
| 39 | \n","US Meat, Milk Prices Should Spike if Donald Tr... | \n","https://www.wired.com/story/us-meat-milk-price... | \n","Matt Reynolds | \n","
| 40 | \n","An Augmented Reality Program Can Help Patients... | \n","https://www.wired.com/story/lining-up-tech-to-... | \n","Grace Browne | \n","
| 41 | \n","Meet the Plant Hacker Creating Flowers Never S... | \n","https://www.wired.com/story/meet-the-plant-hac... | \n","Matt Reynolds | \n","
| 42 | \n","A Mysterious Respiratory Disease Has the Democ... | \n","https://www.wired.com/story/drc-mysterious-res... | \n","Marta Musso | \n","
| 43 | \n","Skip the Sea Kelp Supplements | \n","https://www.wired.com/story/pass-on-sea-kelp-s... | \n","Boutayna Chokrane | \n","
| 44 | \n","Why Soccer Players Are Training in the Dark | \n","https://www.wired.com/story/why-soccer-players... | \n","RM Clark | \n","
| 45 | \n","A Parasite That Eats Cattle Alive Is Creeping ... | \n","https://www.wired.com/story/a-parasite-that-ea... | \n","Geraldine Castro | \n","
| 46 | \n","Lasers Are Making It Easier to Find Buried Lan... | \n","https://www.wired.com/story/this-laser-system-... | \n","Ritsuko Kawai | \n","
| 47 | \n","Mark Cuban’s War on Drug Prices: ‖How Much Fuc... | \n","https://www.wired.com/story/big-interview-mark... | \n","Marah Eakin | \n","
| 48 | \n","Can Artificial Rain, Drones, or Satellites Cle... | \n","https://www.wired.com/story/artificial-rain-dr... | \n","Arunima Kar | \n","
| 49 | \n","These Stem Cell Treatments Are Worth Millions.... | \n","https://www.wired.com/story/stem-cells-cost-ri... | \n","Matt Reynolds | \n","
| 50 | \n","The Mystery of How Supermassive Black Holes Merge | \n","https://www.wired.com/story/how-do-merging-sup... | \n","Jonathan O’Callaghan | \n","
| 51 | \n","The $60 Billion Potential Hiding in Your Disca... | \n","https://www.wired.com/story/a-dollar60-billion... | \n","Vince Beiser | \n","
| 52 | \n","Tune In to the Healing Powers of a Decent Play... | \n","https://www.wired.com/story/music-therapy-heal... | \n","Daniel Levitin | \n","
| 53 | \n","Returning the Amazon Rainforest to Its True Ca... | \n","https://www.wired.com/story/amazon-rainforest-... | \n","Nemonte Nenquimo and Mitch Anderson | \n","
\n"," \n","
| \n", + " | title | \n", + "link | \n", + "author | \n", + "
|---|---|---|---|
| 0 | \n", + "December Wildfires Are Now a Thing | \n", + "https://www.wired.com/story/december-wildfires... | \n", + "Kylie Mohr | \n", + "
| 1 | \n", + "How to Manage Food Anxiety Over the Holidays | \n", + "https://www.wired.com/story/how-to-cope-with-f... | \n", + "Alison Fixsen | \n", + "
| 2 | \n", + "A Spacecraft Is About to Fly Into the Sun’s At... | \n", + "https://www.wired.com/story/parker-solar-probe... | \n", + "Eric Berger, Ars Technica | \n", + "
| 3 | \n", + "To Improve Your Gut Microbiome, Spend More Tim... | \n", + "https://www.wired.com/story/to-improve-your-gu... | \n", + "Kathy Willis | \n", + "
| 4 | \n", + "This Tropical Virus Is Spreading Out of the Am... | \n", + "https://www.wired.com/story/this-tropical-viru... | \n", + "Geraldine Castro | \n", + "
| 5 | \n", + "CDC Confirms First US Case of Severe Bird Flu | \n", + "https://www.wired.com/story/cdc-confirms-first... | \n", + "Emily Mullin | \n", + "
| 6 | \n", + "The Study That Called Out Black Plastic Utensi... | \n", + "https://www.wired.com/story/black-plastic-uten... | \n", + "Beth Mole, Ars Technica | \n", + "
| 7 | \n", + "How Christmas Trees Could Become a Source of L... | \n", + "https://www.wired.com/story/how-christmas-tree... | \n", + "Alexa Phillips | \n", + "
| 8 | \n", + "Creating a Global Package to Solve the Problem... | \n", + "https://www.wired.com/story/global-plastics-tr... | \n", + "Susan Solomon | \n", + "
| 9 | \n", + "These 3 Things Are Standing in the Way of a Gl... | \n", + "https://www.wired.com/story/these-3-things-are... | \n", + "Steve Fletcher and Samuel Winton | \n", + "
| 10 | \n", + "Environmental Sensing Is Here, Tracking Everyt... | \n", + "https://www.wired.com/story/environmental-sens... | \n", + "Sabrina Weiss | \n", + "
| 11 | \n", + "Generative AI and Climate Change Are on a Coll... | \n", + "https://www.wired.com/story/true-cost-generati... | \n", + "Sasha Luccioni | \n", + "
| 12 | \n", + "Climate Change Is Destroying Monarch Butterfli... | \n", + "https://www.wired.com/story/global-warming-thr... | \n", + "Andrea J. Arratibel | \n", + "
| 13 | \n", + "More Humanitarian Organizations Will Harness A... | \n", + "https://www.wired.com/story/humanitarian-organ... | \n", + "David Miliband | \n", + "
| 14 | \n", + "Chocolate Has a Sustainability Problem. Scienc... | \n", + "https://www.wired.com/story/chocolate-has-a-su... | \n", + "Eve Thomas | \n", + "
| 15 | \n", + "We’ve Never Been Closer to Finding Life Outsid... | \n", + "https://www.wired.com/story/james-webb-space-t... | \n", + "Lisa Kaltenegger | \n", + "
| 16 | \n", + "The End Is Near for NASA’s Voyager Probes | \n", + "https://www.wired.com/story/the-end-is-near-fo... | \n", + "Luca Nardi | \n", + "
| 17 | \n", + "Why Can’t You Switch Seats in an Empty Airplane? | \n", + "https://www.wired.com/story/why-cant-you-switc... | \n", + "Rhett Allain | \n", + "
| 18 | \n", + "The Simple Math Behind Public Key Cryptography | \n", + "https://www.wired.com/story/how-public-key-cry... | \n", + "John Pavlus | \n", + "
| 19 | \n", + "Everyone Is Capable of Mathematical Thinking--... | \n", + "https://www.wired.com/story/everyone-is-capabl... | \n", + "Kelsey Houston-Edwards | \n", + "
| 20 | \n", + "The Physics of the Macy’s Thanksgiving Day Par... | \n", + "https://www.wired.com/story/the-physics-of-the... | \n", + "Rhett Allain | \n", + "
| 21 | \n", + "A Third Person Has Received a Transplant of a ... | \n", + "https://www.wired.com/story/a-third-person-has... | \n", + "Emily Mullin | \n", + "
| 22 | \n", + "Muscle Implants Could Allow Mind-Controlled Pr... | \n", + "https://www.wired.com/story/amputees-could-con... | \n", + "Emily Mullin | \n", + "
| 23 | \n", + "Combining AI and Crispr Will Be Transformational | \n", + "https://www.wired.com/story/combining-ai-and-c... | \n", + "Jennifer Doudna | \n", + "
| 24 | \n", + "Neuralink Plans to Test Whether Its Brain Impl... | \n", + "https://www.wired.com/story/neuralink-robotic-... | \n", + "Emily Mullin | \n", + "
| 25 | \n", + "Eight Scientists, a Billion Dollars, and the M... | \n", + "https://www.wired.com/story/aria-moonshot-darp... | \n", + "Matt Reynolds | \n", + "
| 26 | \n", + "The Atlas Robot Is Dead. Long Live the Atlas R... | \n", + "https://www.wired.com/story/the-atlas-robot-is... | \n", + "NA | \n", + "
| 27 | \n", + "The Atlas Robot Is Dead. Long Live the Atlas R... | \n", + "https://www.wired.com/story/the-atlas-robot-is... | \n", + "Carlton Reid | \n", + "
| 28 | \n", + "Meet the Next Generation of Doctors--and Their... | \n", + "https://www.wired.com/story/next-generation-do... | \n", + "Neha Mukherjee | \n", + "
| 29 | \n", + "AI Is Building Highly Effective Antibodies Tha... | \n", + "https://www.wired.com/story/labgenius-antibody... | \n", + "Amit Katwala | \n", + "
| 30 | \n", + "An Uncertain Future Requires Uncertain Predict... | \n", + "https://www.wired.com/story/embrace-uncertaint... | \n", + "David Spiegelhalter | \n", + "
| 31 | \n", + "These Rats Learned to Drive--and They Love It | \n", + "https://www.wired.com/story/these-rats-learned... | \n", + "Kelly Lambert | \n", + "
| 32 | \n", + "Scientists Are Unlocking the Secrets of Your ‗... | \n", + "https://www.wired.com/story/cerebellum-brain-m... | \n", + "R Douglas Fields | \n", + "
| 33 | \n", + "Meet the Designer Behind Neuralink’s Surgical ... | \n", + "https://www.wired.com/story/designer-behind-ne... | \n", + "Emily Mullin | \n", + "
| 34 | \n", + "Antibodies Could Soon Help Slow the Aging Process | \n", + "https://www.wired.com/story/antibodies-could-s... | \n", + "Andrew Steele | \n", + "
| 35 | \n", + "Good at Reading? Your Brain May Be Structured ... | \n", + "https://www.wired.com/story/good-at-reading-yo... | \n", + "Mikael Roll | \n", + "
| 36 | \n", + "Mega-Farms Are Driving the Threat of Bird Flu | \n", + "https://www.wired.com/story/mega-farms-are-dri... | \n", + "Georgina Gustin | \n", + "
| 37 | \n", + "RFK Plans to Take on Big Pharma. It’s Easier S... | \n", + "https://www.wired.com/story/rfks-plan-to-take-... | \n", + "Emily Mullin | \n", + "
| 38 | \n", + "Designer Babies Are Teenagers Now--and Some of... | \n", + "https://www.wired.com/story/your-next-job-desi... | \n", + "Emi Nietfeld | \n", + "
| 39 | \n", + "US Meat, Milk Prices Should Spike if Donald Tr... | \n", + "https://www.wired.com/story/us-meat-milk-price... | \n", + "Matt Reynolds | \n", + "
| 40 | \n", + "An Augmented Reality Program Can Help Patients... | \n", + "https://www.wired.com/story/lining-up-tech-to-... | \n", + "Grace Browne | \n", + "
| 41 | \n", + "Meet the Plant Hacker Creating Flowers Never S... | \n", + "https://www.wired.com/story/meet-the-plant-hac... | \n", + "Matt Reynolds | \n", + "
| 42 | \n", + "A Mysterious Respiratory Disease Has the Democ... | \n", + "https://www.wired.com/story/drc-mysterious-res... | \n", + "Marta Musso | \n", + "
| 43 | \n", + "Skip the Sea Kelp Supplements | \n", + "https://www.wired.com/story/pass-on-sea-kelp-s... | \n", + "Boutayna Chokrane | \n", + "
| 44 | \n", + "Why Soccer Players Are Training in the Dark | \n", + "https://www.wired.com/story/why-soccer-players... | \n", + "RM Clark | \n", + "
| 45 | \n", + "A Parasite That Eats Cattle Alive Is Creeping ... | \n", + "https://www.wired.com/story/a-parasite-that-ea... | \n", + "Geraldine Castro | \n", + "
| 46 | \n", + "Lasers Are Making It Easier to Find Buried Lan... | \n", + "https://www.wired.com/story/this-laser-system-... | \n", + "Ritsuko Kawai | \n", + "
| 47 | \n", + "Mark Cuban’s War on Drug Prices: ‖How Much Fuc... | \n", + "https://www.wired.com/story/big-interview-mark... | \n", + "Marah Eakin | \n", + "
| 48 | \n", + "Can Artificial Rain, Drones, or Satellites Cle... | \n", + "https://www.wired.com/story/artificial-rain-dr... | \n", + "Arunima Kar | \n", + "
| 49 | \n", + "These Stem Cell Treatments Are Worth Millions.... | \n", + "https://www.wired.com/story/stem-cells-cost-ri... | \n", + "Matt Reynolds | \n", + "
| 50 | \n", + "The Mystery of How Supermassive Black Holes Merge | \n", + "https://www.wired.com/story/how-do-merging-sup... | \n", + "Jonathan O’Callaghan | \n", + "
| 51 | \n", + "The $60 Billion Potential Hiding in Your Disca... | \n", + "https://www.wired.com/story/a-dollar60-billion... | \n", + "Vince Beiser | \n", + "
| 52 | \n", + "Tune In to the Healing Powers of a Decent Play... | \n", + "https://www.wired.com/story/music-therapy-heal... | \n", + "Daniel Levitin | \n", + "
| 53 | \n", + "Returning the Amazon Rainforest to Its True Ca... | \n", + "https://www.wired.com/story/amazon-rainforest-... | \n", + "Nemonte Nenquimo and Mitch Anderson | \n", + "
\n \n
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
+ "source": "## 🕷️ Extract Wired Science News Info with llama-index and ScrapegraphAI's APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-3) [](https://colab.research.google.com/drive/10rkKL0Z50If1LWSR0ulgoseWMmd5gb6A?usp=sharing)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "## 🕷️ Extract Wired Science News Info with llama-index and ScrapegraphAI's APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-3) [](https://colab.research.google.com/drive/10rkKL0Z50If1LWSR0ulgoseWMmd5gb6A?usp=sharing)",
+ "text/markdown": "## 🕷️ Extract Wired Science News Info with llama-index and ScrapegraphAI's APIs\n\n[](https://www.runalph.ai/notebooks/scrapegraphai/scrapegraph-llama-index-3) [](https://colab.research.google.com/drive/10rkKL0Z50If1LWSR0ulgoseWMmd5gb6A?usp=sharing)"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
"metadata": {
"id": "jEkuKbcRrPcK"
},
- "source": [
- "## 🕷️ Extract Wired Science News Info with llama-index and ScrapegraphAI's APIs"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "completed",
+ "execution_count": null,
+ "executionEndTime": "2026-03-26T00:02:49.233Z",
+ "executionStartTime": "2026-03-26T00:02:49.233Z"
},
{
- "cell_type": "markdown",
+ "source": "",
+ "outputs": [],
"metadata": {
"id": "oQa0HZo7nuB_"
},
- "source": [
- ""
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔧 Install `dependencies`",
+ "outputs": [],
"metadata": {
"id": "IzsyDXEWwPVt"
},
- "source": [
- "### 🔧 Install `dependencies`"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "%%capture\n!pip install llama-index-tools-scrapegraphai\n!pip install llama-index",
+ "outputs": [],
"metadata": {
"id": "os_vm0MkIxr9"
},
- "outputs": [],
- "source": [
- "%%capture\n",
- "!pip install llama-index-tools-scrapegraphai\n",
- "!pip install llama-index"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 🔑 Import `ScrapeGraph` API key",
+ "outputs": [],
"metadata": {
"id": "apBsL-L2KzM7"
},
- "source": [
- "### 🔑 Import `ScrapeGraph` API key"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)",
+ "outputs": [],
"metadata": {
"id": "ol9gQbAFkh9b"
},
- "source": [
- "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "code",
- "execution_count": null,
+ "source": "import getpass\nimport os\n\nif not os.environ.get(\"SGAI_API_KEY\"):\n os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")",
+ "outputs": [],
"metadata": {
"id": "sffqFG2EJ8bI"
},
- "outputs": [],
- "source": [
- "import getpass\n",
- "import os\n",
- "\n",
- "if not os.environ.get(\"SGAI_API_KEY\"):\n",
- " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"
- ]
+ "cell_type": "code",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n",
+ "outputs": [],
"metadata": {
"id": "jnqMB2-xVYQ7"
},
- "source": [
- "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
- ]
+ "cell_type": "markdown",
+ "isExecuting": false,
+ "stdinRequest": null,
+ "executionState": "idle",
+ "execution_count": null,
+ "executionEndTime": null,
+ "executionStartTime": null
},
{
- "cell_type": "markdown",
+ "source": "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n\n\n \n
\n",
- " \n",
- "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Wired Science News with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"6Rgz5T4eHBVz"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734531168564,"user_tz":-60,"elapsed":17535,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"bcdd9ae1-151e-41d6-df36-8835c660460f"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | category | \n","title | \n","link | \n","author | \n","
|---|---|---|---|---|
| 0 | \n","Science | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 1 | \n","Environment | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 2 | \n","Xenotransplantation | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 3 | \n","Health | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 4 | \n","Science | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 5 | \n","Health | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 6 | \n","Health | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 7 | \n","Environment | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Environment | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","Environment | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
\n"," \n","
| \n", + " | category | \n", + "title | \n", + "link | \n", + "author | \n", + "
|---|---|---|---|---|
| 0 | \n", + "Science | \n", + "The Study That Called Out Black Plastic Utensi... | \n", + "https://www.wired.com/story/black-plastic-uten... | \n", + "Beth Mole, Ars Technica | \n", + "
| 1 | \n", + "Environment | \n", + "Generative AI and Climate Change Are on a Coll... | \n", + "https://www.wired.com/story/true-cost-generati... | \n", + "Sasha Luccioni | \n", + "
| 2 | \n", + "Xenotransplantation | \n", + "A Third Person Has Received a Transplant of a ... | \n", + "https://www.wired.com/story/a-third-person-has... | \n", + "Emily Mullin | \n", + "
| 3 | \n", + "Health | \n", + "Antibodies Could Soon Help Slow the Aging Process | \n", + "https://www.wired.com/story/antibodies-could-s... | \n", + "Andrew Steele | \n", + "
| 4 | \n", + "Science | \n", + "Good at Reading? Your Brain May Be Structured ... | \n", + "https://www.wired.com/story/good-at-reading-yo... | \n", + "Mikael Roll | \n", + "
| 5 | \n", + "Health | \n", + "Mega-Farms Are Driving the Threat of Bird Flu | \n", + "https://www.wired.com/story/mega-farms-are-dri... | \n", + "Georgina Gustin | \n", + "
| 6 | \n", + "Health | \n", + "RFK Plans to Take on Big Pharma. It’s Easier S... | \n", + "https://www.wired.com/story/rfks-plan-to-take-... | \n", + "Emily Mullin | \n", + "
| 7 | \n", + "Environment | \n", + "How Christmas Trees Could Become a Source of L... | \n", + "https://www.wired.com/story/how-christmas-tree... | \n", + "Alexa Phillips | \n", + "
| 8 | \n", + "Environment | \n", + "Creating a Global Package to Solve the Problem... | \n", + "https://www.wired.com/story/global-plastics-tr... | \n", + "Susan Solomon | \n", + "
| 9 | \n", + "Environment | \n", + "These 3 Things Are Standing in the Way of a Gl... | \n", + "https://www.wired.com/story/these-3-things-are... | \n", + "Steve Fletcher and Samuel Winton | \n", + "
\n \n