chore: sync upstream firecrawl/main (2026-04-13)

CLAUDE.md

@@ -3,6 +3,26 @@
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
 Firecrawl is a web scraper API. The directory you have access to is a monorepo:
+ - `apps/api` has the actual API and worker code
+ - `apps/js-sdk`, `apps/python-sdk`, `apps/rust-sdk` and `apps/java-sdk` are various SDKs
+
+For local development and PowerShell instructions, see `LOCAL_DEVELOPMENT_GUIDE.md`.
+
+When making changes to the API, here are the general steps you should take:
+1. Write some end-to-end tests that assert your win conditions, if they don't already exist
+  - 1 happy path (more is encouraged if there are multiple happy paths with significantly different code paths taken)
+  - 1+ failure path(s)
+  - Generally, E2E (called `snips` in the API) is always preferred over unit testing.
+  - In the API, always use `scrapeTimeout` from `./lib` to set the timeout you use for scrapes.
+  - These tests will be ran on a variety of configurations. You should gate tests in the following manner:
+    - If it requires fire-engine: `!process.env.TEST_SUITE_SELF_HOSTED`
+    - If it requires AI: `!process.env.TEST_SUITE_SELF_HOSTED || process.env.OPENAI_API_KEY || process.env.OLLAMA_BASE_URL`
+2. Write code to achieve your win conditions
+3. Run your tests using `pnpm harness jest ...`
+  - `pnpm harness` is a command that gets the API server and workers up for you to run the tests. Don't try to `pnpm start` manually.
+  - The full test suite takes a long time to run, so you should try to only execute the relevant tests locally, and let CI run the full test suite.
+4. Push to a branch, open a PR, and let CI run to verify your win condition.
+Keep these steps in mind while building your TODO list.
 - `apps/api` — the API server, queue workers, and scraping engines (TypeScript, the only place most changes land)
 - `apps/*-sdk` — language SDKs (js, python, go, java, ruby, rust, php, dot-net, elixir)
 - `apps/playwright-service-ts` — headless browser sidecar used by the API

PROGRESS_SUMMARY.md

@@ -0,0 +1,36 @@
+# Firecrawl Setup Progress Summary - January 6, 2026
+
+## Work Completed
+
+### 1. Environment Configuration
+- **AI Provider**: Configured Firecrawl to use **OpenRouter** (`MODEL_NAME=xiaomi/mimo-v2-flash:free`).
+- **Database**: Switched to **Supabase** as the persistent backend (`USE_DB_AUTHENTICATION=true`).
+- **Stability**: Set `DISABLE_BLOCKLIST=true` and `DISABLE_ENGPICKER=true` to prevent API crashes caused by missing advanced cloud-only tables.
+- **Docker**: Optimized `docker-compose.yaml` to pass all Supabase environment variables (`SUPABASE_URL`, `SUPABASE_SERVICE_TOKEN`, etc.) to the containers.
+
+### 2. Database Integration (Shared Project Safety)
+- **Safety Zone**: Established a naming convention where Firecrawl only interacts with **lowercase** tables (`scrapes`, `crawls`, `api_keys`). This ensures zero conflict with the other application's **CamelCase** tables (`Chat`, `User`, `Document`).
+- **Initialization**: 
+    - Initialized the `nuq` schema for job queuing.
+    - Created the `public` schema tables required for logging and authentication.
+- **Self-Hosted Mocks**: Created a full set of mock RPCs (Stored Procedures) in Supabase to bypass cloud billing logic while remaining compatible with the "production" auth path.
+- **Persistent Storage Fix**: Successfully dropped and recreated the Firecrawl tables to add a `content` column (for markdown/HTML) and a `result` column (for AI extractions). This resolved "upstream timeout" issues caused by locks from other applications.
+
+### 3. API Logic Updates
+- **Content Saving**: Modified `apps/api/src/services/logging/log_job.ts` to actively save the full markdown/HTML output into the Supabase `scrapes` table.
+- **Extraction Saving**: Updated the `/v1/extract` workflow to save the final AI JSON result directly into the Supabase `extracts` table.
+- **SQL Template**: Updated `apps/nuq-postgres/supabase-setup.sql` to include these persistent storage columns for any future deployments.
+
+### 4. Functional Verification
+- **Scrape Test**: Confirmed that `Invoke-RestMethod` to `/v1/scrape` successfully renders markdown and saves it to the `content` column in Supabase.
+- **Extract Test**: Verified that the `/v1/extract` endpoint processes prompts correctly and logs the activity to the `extracts` table.
+
+### 5. Access Information
+- **Local API**: `http://localhost:3002`
+- **Authenticated Token**: `fc-31dba252482749989356775a972cd48a`
+- **Supabase Project ID**: `fhqycqubkkrdgzswccwd`
+
+## 🛡️ Important Rules & Safety
+- **DO NOT** modify any tables starting with Capital Letters.
+- **DO NOT** run destructive SQL (DROP/TRUNCATE) on non-Firecrawl tables.
+- **ALWAYS** ensure new logging columns (`content`, `result`) are included in database schema updates.

test-runs/tulsa-cre-companies/Bauer-Associates/About---Bauer-and-Associates.md

@@ -0,0 +1,46 @@
+About
+=====
+
+In 1979, Grover Bauer founded Bauer & Associates in Tulsa, Oklahoma. Since that time, our team of qualified professionals have been helping grow Tulsa and the surrounding area. We have a proven track record in the completion of commercial, industrial, and investment property transactions, including sales and/or leases of office buildings, retain centers, warehouses, manufacturing plants, self-storage facilities and land for the uses referenced above. Our diversified organization has the expertise to be your full service brokerage and property management representative.
+
+Your property is important to you, and oftentimes, a primary source of income. We understand how important it is to keep your property at a high occupancy level, and we work hard to make sure we get you there! Our team offers a level of service and expertise unmatched in the Tulsa Metro area commercial, industrial, and investment real estate market.
+
+Our tenants and buyers also experience a unique level of quality service when they are looking to find property in the Tulsa Metro area. Whether you need a space for your growing business to occupy, or you are looking for your next investment property in the Tulsa Metro area, we know the market, and we are just as eager as you are to find the perfect space or property!  
+Our ability to solve and overcome problems has been proven time after time in Tulsa commercial property for lease or sale transactions. Continuous monitoring of the Tulsa Metro area commercial properties for sale and/or lease, market trends, as well as the implementation of a state-of-the-art property database, allows us to give you the very best possible service in the always dynamic Tulsa Metro area market place.
+
+Let us help you in your search for:
+-----------------------------------
+
+* * *
+
+*   Tulsa Commercial Property for Sale
+*   Tulsa Commercial Property for Lease
+*   Tulsa Office Space for Lease
+*   Tulsa Office Space for Sale
+*   Tulsa Office Buildings for Lease
+
+*   Tulsa Industrial Property for Sale
+*   Tulsa Commercial Real Estate
+*   Tulsa Warehouse Space for Lease
+*   Tulsa Investment Property for Sale
+*   Tulsa Retail Space for Lease
+
+*   Tulsa Shopping Centers for Sale
+*   Property for Sale in Oklahoma
+*   Property for Lease in Oklahoma
+*   Tulsa Commercial Properties for Sale
+*   Tulsa Commercial Properties for Lease
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Available-Properties---Bauer-and-Associates.md

@@ -0,0 +1,42 @@
+Available Properties
+====================
+
+Location
+
+Type
+
+Price/Size
+
+More
+
+Reset
+
+[Map](https://bauertulsa.com/available-properties/#/search/map " Map")
+
+Sort
+
+*   
+*   0
+*   
+
+Loading
+
+*   
+*   0
+*   
+
+©2025 Moody's Analytics, Inc. All rights reserved. By searching this site, you agree to Moody's [Terms of Use](https://members.moodyscre.com/terms/)
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Brad-Baldwin---Bauer-and-Associates.md

@@ -0,0 +1,18 @@
+Brad Baldwin
+============
+
+[![](https://bauertulsa.com/wp-content/uploads/Brad-Baldwin-1080x1327.jpg)](https://bauertulsa.com/wp-content/uploads/Brad-Baldwin-e1502450056575.jpg)
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Contact---Bauer-and-Associates.md

@@ -0,0 +1,76 @@
+Contact
+=======
+
+Bauer & Associates engages in the sale and leasing of commercial, industrial, and investment properties as well as property management. For further information, please contact us at:
+
+**Bauer & Associates, Realtors**  
+6846 S. Canton Ave.  
+Suite 100  
+Tulsa, OK 74136  
+[info@bauertulsa.com](mailto:info@bauertulsa.com)
+
+Phone: [(918) 665-1210](tel:(918)%20665-1210)
+
+Fax:      (918) 665-6462
+
+Bauer & Associates specializes in the sale, lease and management of commercial real estate industrial property and warehouse space in Tulsa, Oklahoma. Our property management team services shopping centers, office buildings, and industrial facilities.
+
+"\*" indicates required fields
+
+Comments
+
+This field is for validation purposes and should be left unchanged.
+
+Name\*
+
+Email\*
+
+Phone\*
+
+Your Message
+
+
+
+“GREAT EXPERIENCE, ALWAYS LOVE WORKING WITH THESE GUYS.”
+
+— Adam Stockdall
+
+Let us help you in your search for:
+-----------------------------------
+
+* * *
+
+*   Tulsa Commercial Property for Sale
+*   Tulsa Commercial Property for Lease
+*   Tulsa Office Space for Lease
+*   Tulsa Office Space for Sale
+*   Tulsa Office Buildings for Lease
+
+*   Tulsa Industrial Property for Sale
+*   Tulsa Commercial Real Estate
+*   Tulsa Warehouse Space for Lease
+*   Tulsa Investment Property for Sale
+*   Tulsa Retail Space for Lease
+
+*   Tulsa Shopping Centers for Sale
+*   Property for Sale in Oklahoma
+*   Property for Lease in Oklahoma
+*   Tulsa Commercial Properties for Sale
+*   Tulsa Commercial Properties for Lease
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)
+
+
+Notifications

test-runs/tulsa-cre-companies/Bauer-Associates/Difference---Bauer-and-Associates.md

@@ -0,0 +1,18 @@
+Difference
+==========
+
+[![Difference](https://bauertulsa.com/wp-content/uploads/Difference.png)](https://bauertulsa.com/wp-content/uploads/Difference.png)
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Doug-Bauer---Bauer-and-Associates.md

@@ -0,0 +1,18 @@
+Doug Bauer
+==========
+
+[![](https://bauertulsa.com/wp-content/uploads/Doug-Bauer-1080x1308.jpg)](https://bauertulsa.com/wp-content/uploads/Doug-Bauer-e1502449629963.jpg)
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Duane-Suchy---Bauer-and-Associates.md

@@ -0,0 +1,18 @@
+Duane Suchy
+===========
+
+[![](https://bauertulsa.com/wp-content/uploads/Duane-Suchy-1080x1350.jpg)](https://bauertulsa.com/wp-content/uploads/Duane-Suchy-e1502449817541.jpg)
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Grover-Bauer---Bauer-and-Associates.md

@@ -0,0 +1,18 @@
+Grover Bauer
+============
+
+[![](https://bauertulsa.com/wp-content/uploads/Grover-Bauer-1080x1350.jpg)](https://bauertulsa.com/wp-content/uploads/Grover-Bauer-e1502449529502.jpg)
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)

test-runs/tulsa-cre-companies/Bauer-Associates/Hello-world----Bauer-and-Associates.md

@@ -0,0 +1,27 @@
+Hello world!
+============
+
+by [admin](https://bauertulsa.com/author/admin/ "Posts by admin")
+ | Jun 20, 2017 | [Uncategorized](https://bauertulsa.com/category/uncategorized/)
+ | [0 comments](https://bauertulsa.com/hello-world/#respond)
+
+Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!
+
+### Submit a Comment
+
+You must be [logged in](https://bauertulsa.com/wp-login.php?redirect_to=https%3A%2F%2Fbauertulsa.com%2Fhello-world%2F)
+ to post a comment.
+
+*   [Home](https://bauertulsa.com/)
+
+*   [About](https://bauertulsa.com/about/)
+
+*   [Available Properties](https://bauertulsa.com/available-properties/)
+
+*   [Our Team](https://bauertulsa.com/our-team/)
+
+*   [Testimonials](https://bauertulsa.com/testimonials/)
+
+*   [Contact](https://bauertulsa.com/contact/)
+
+*   [Newsletter](https://bauertulsa.com/newsletter/)