fix(docs): strengthen crawl + indexing signals for the blog#472
Merged
Conversation
Google Search Console reported ~107 blog/docs URLs as "Discovered — currently not indexed". There is no technical blocker (pages are statically generated; robots/canonical/sitemap are clean; no noindex in HTML or headers) — it's the young-domain crawl-budget pattern. These changes maximize the signals that help Google prioritize crawling the URLs it has already discovered. - sitemap: emit each blog URL's real frontmatter date as <lastmod> instead of the build timestamp. Stamping every URL with the build time told Google that all ~107 pages changed on every deploy, so it treated lastmod as noise and ignored it — discarding the signal it uses to schedule crawls. Docs/home stay on the build time (no per-page date); the blog index uses the newest post. - blog posts: add per-post BlogPosting JSON-LD (headline, datePublished, author, image, mainEntityOfPage) so crawlers can model each post as an Article instead of guessing — a content signal that aids prioritization and unlocks Article rich results. - blog posts: add a per-post OG image route (/og/blog/[slug]) mirroring the docs one, wired into openGraph/twitter/JSON-LD. Posts previously declared a summary_large_image card with no image, so social cards rendered blank. - home page: add a "From the blog" section linking the 4 latest posts. The home page is the most-crawled URL, so direct links from it are the strongest crawl-priority signal we can give a young blog's posts. Verified: /sitemap.xml serves distinct per-post <lastmod> dates; docs typecheck passes (the new typed OG route resolves); the home page renders the section with 4 crawlable /blog links. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Google Search Console reports ~107 blog/docs URLs as "Discovered — currently not indexed."
I checked every technical cause and there's no blocker: blog posts are statically generated,
robots/canonical/sitemap are clean, and there's nonoindexin the HTML or response headers (proxy.ts,next.configheaders(),security-headers.mjsall verified). This is the classic young-domain crawl-budget pattern — Google discovered the whole sitemap at once and is metering how fast it crawls because the domain has little authority yet.The dominant fix is non-code (see Follow-up below), but this PR maximizes the on-site signals that help Google prioritize crawling the URLs it has already discovered — and fixes one real anti-pattern.
Changes
<lastmod>(app/sitemap.ts) — was stamping every URL with the build timestamp, so each deploy told Google all ~107 pages changed simultaneously. Google treats an always-currentlastmodas untrustworthy and ignores it, discarding the signal it uses to schedule crawls. Now each blog URL reports its real frontmatter date; the blog index uses the newest post's date. (Docs/home stay on build time — no per-page date available.)BlogPostingJSON-LD (app/(home)/blog/[slug]/page.tsx) — the root layout emits site-wideWebSite/SoftwareApplication, but posts had none. Addsheadline,datePublished,author,image,mainEntityOfPageso crawlers model each post as an Article (aids prioritization; unlocks Article rich results).app/og/blog/[slug]/route.tsx, new) — mirrors the existing docs OG route, wired intoopenGraph/twitter/JSON-LD. Posts previously declared asummary_large_imagecard with no image, so social cards rendered blank.app/(home)/components/from-the-blog.tsx, new) — links the 4 latest posts from the home page. The home page is the most-crawled URL, so direct links from it are the strongest crawl-priority signal we can give the posts.Verification
/sitemap.xmlserved live returns distinct per-post<lastmod>dates (2026-07-08,2026-07-02,2026-06-30,2026-06-29…) instead of one build timestamp.typecheckpasses (tsc --noEmitafternext typegen) — the new typedRouteContext<'/og/blog/[slug]'>route resolves./blog/*links + a "Read the blog" CTA (confirmed in the served HTML and via DOM inspect).Follow-up (not code — the dominant lever)
These get pages indexed faster than any code change on a young domain:
/blog+ top posts + key docs.🤖 Generated with Claude Code