Skip to content

fix(docs): strengthen crawl + indexing signals for the blog#472

Merged
rejifald merged 1 commit into
mainfrom
claude/search-console-indexing-7e7840
Jul 8, 2026
Merged

fix(docs): strengthen crawl + indexing signals for the blog#472
rejifald merged 1 commit into
mainfrom
claude/search-console-indexing-7e7840

Conversation

@rejifald

@rejifald rejifald commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Why

Google Search Console reports ~107 blog/docs URLs as "Discovered — currently not indexed."

I checked every technical cause and there's no blocker: blog posts are statically generated, robots/canonical/sitemap are clean, and there's no noindex in the HTML or response headers (proxy.ts, next.config headers(), security-headers.mjs all verified). This is the classic young-domain crawl-budget pattern — Google discovered the whole sitemap at once and is metering how fast it crawls because the domain has little authority yet.

The dominant fix is non-code (see Follow-up below), but this PR maximizes the on-site signals that help Google prioritize crawling the URLs it has already discovered — and fixes one real anti-pattern.

Changes

  • Truthful sitemap <lastmod> (app/sitemap.ts) — was stamping every URL with the build timestamp, so each deploy told Google all ~107 pages changed simultaneously. Google treats an always-current lastmod as untrustworthy and ignores it, discarding the signal it uses to schedule crawls. Now each blog URL reports its real frontmatter date; the blog index uses the newest post's date. (Docs/home stay on build time — no per-page date available.)
  • Per-post BlogPosting JSON-LD (app/(home)/blog/[slug]/page.tsx) — the root layout emits site-wide WebSite/SoftwareApplication, but posts had none. Adds headline, datePublished, author, image, mainEntityOfPage so crawlers model each post as an Article (aids prioritization; unlocks Article rich results).
  • Per-post OG image route (app/og/blog/[slug]/route.tsx, new) — mirrors the existing docs OG route, wired into openGraph/twitter/JSON-LD. Posts previously declared a summary_large_image card with no image, so social cards rendered blank.
  • "From the blog" home section (app/(home)/components/from-the-blog.tsx, new) — links the 4 latest posts from the home page. The home page is the most-crawled URL, so direct links from it are the strongest crawl-priority signal we can give the posts.

Verification

  • /sitemap.xml served live returns distinct per-post <lastmod> dates (2026-07-08, 2026-07-02, 2026-06-30, 2026-06-29…) instead of one build timestamp.
  • Docs typecheck passes (tsc --noEmit after next typegen) — the new typed RouteContext<'/og/blog/[slug]'> route resolves.
  • Home page renders the section with 4 crawlable /blog/* links + a "Read the blog" CTA (confirmed in the served HTML and via DOM inspect).
  • No existing test asserts homepage structure, sitemap contents, or JSON-LD.

Follow-up (not code — the dominant lever)

These get pages indexed faster than any code change on a young domain:

  1. Request indexing for priority URLs in GSC (URL Inspection → Request Indexing; rate-limited ~10–15/day) — spend them on /blog + top posts + key docs.
  2. Confirm the sitemap is submitted (GSC → Sitemaps → "Success").
  3. Keep building authority/backlinks (awesome-lists, registries, writeups) — each external link Google follows raises crawl priority.
  4. Patience — the "Validate Fix" started 7/3 re-runs automatically; a young domain indexes gradually over weeks.

🤖 Generated with Claude Code

Google Search Console reported ~107 blog/docs URLs as "Discovered —
currently not indexed". There is no technical blocker (pages are statically
generated; robots/canonical/sitemap are clean; no noindex in HTML or headers)
— it's the young-domain crawl-budget pattern. These changes maximize the
signals that help Google prioritize crawling the URLs it has already
discovered.

- sitemap: emit each blog URL's real frontmatter date as <lastmod> instead of
  the build timestamp. Stamping every URL with the build time told Google that
  all ~107 pages changed on every deploy, so it treated lastmod as noise and
  ignored it — discarding the signal it uses to schedule crawls. Docs/home stay
  on the build time (no per-page date); the blog index uses the newest post.

- blog posts: add per-post BlogPosting JSON-LD (headline, datePublished,
  author, image, mainEntityOfPage) so crawlers can model each post as an
  Article instead of guessing — a content signal that aids prioritization and
  unlocks Article rich results.

- blog posts: add a per-post OG image route (/og/blog/[slug]) mirroring the
  docs one, wired into openGraph/twitter/JSON-LD. Posts previously declared a
  summary_large_image card with no image, so social cards rendered blank.

- home page: add a "From the blog" section linking the 4 latest posts. The
  home page is the most-crawled URL, so direct links from it are the strongest
  crawl-priority signal we can give a young blog's posts.

Verified: /sitemap.xml serves distinct per-post <lastmod> dates; docs
typecheck passes (the new typed OG route resolves); the home page renders the
section with 4 crawlable /blog links.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@rejifald rejifald merged commit f396c11 into main Jul 8, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant