Skip to content

feat(content): case-study post with full Claude transcripts — GEO/SEO Phase 2.1 #2971

@matheuspoleza

Description

@matheuspoleza

Description

Write + publish the flagship v1 content piece: a "radical transparency case study" comparing Claude building 3–5 apps in Vertz vs Next.js + Drizzle + tRPC. Publish full transcripts as the core artifact. Manifesto-aligned (no over-claim), adversarially defensible (no methodology to dispute when readers can see every tool call).

This REPLACES the original "statistical benchmark" plan which was killed by adversarial review (n=5 too small, p-hacking risk, budget impossible at $200/mo).

Context

Part of GEO/SEO strategy v1 (28-day scope).

  • Design doc: plans/geo-seo-strategy.md
  • Full task spec: plans/geo-seo-strategy/phase-02-ignition-content.md → Task 1 (rewritten v2)
  • Budget: ~$80 one-time API cost (fits $200/mo cap)

Acceptance Criteria

BDD scenarios

describe('Feature: Radical transparency case-study post', () => {
  describe('Given preregistration is required before running', () => {
    describe('When preregistration.md is committed', () => {
      it('then it is committed to main ≥5 days before first session', () => {});
      it('then it contains: task list, exact prompts, framework versions, model version, success criteria', () => {});
      it('then a public gist is announced on X for pre-comment', () => {});
    });
  });

  describe('Given the case-study sessions run', () => {
    describe('When each task runs 5 times per framework', () => {
      it('then raw JSON transcripts are committed to `benchmarks/case-study/transcripts/`', () => {});
      it('then no curation, no "best of 5" cherry-picking', () => {});
      it('then total API spend ≤$80', () => {});
    });
  });

  describe('Given the final post is published', () => {
    describe('When a reader loads it', () => {
      it('then the headline contains NO statistic (no "89%", no "4x", etc.)', () => {});
      it('then a "Limitations" section acknowledges n=5, 2 frameworks, 1 LLM', () => {});
      it('then ≥1 section documents where Vertz lost or performed worse', () => {});
      it('then the reproduction steps are exact and runnable', () => {});
      it('then an external (non-Vertz) engineer has reviewed methodology', () => {});
    });
  });
});

Checklist

  • Decision made on task count: 3 or 5 (default: 3)
  • External reviewer contacted + accepted (candidates: Sebastian Markbåge, Jarred Sumner, Theo Browne, Daniel Ehrenberg, Shawn "swyx" Wang)
  • Preregistration committed 5+ days before first run
  • 30–50 sessions actually run. Real transcripts. No synthetic data.
  • Post published REGARDLESS of outcome (even if Next.js wins tasks)
  • Cross-post drafts ready: dev.to, Hashnode (with canonical back)
  • HN title drafted: "Show HN: Full Claude transcripts of building N apps in Vertz vs Next.js"

Dependencies

  • Blocked by: issue (SSR head injection) for the post to be SEO-useful on publish
  • Blocked by: issue (MCP server) if we want Claude to use it during sessions (methodology decision — to be made during preregistration)

Progress

  • 2026-04-22: Opened

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions