docs: expand source quality audit and methodology transparency#2
docs: expand source quality audit and methodology transparency#2cboone wants to merge 14 commits intodocs/certainty-calibration-pass-20260302from
Conversation
There was a problem hiding this comment.
Pull request overview
This PR begins a source-quality audit across three high-impact documentation reports by adding explicit notes that help readers distinguish primary/secondary/tertiary evidence and interpret claims appropriately.
Changes:
- Add source-quality/method notes to the LLM agent monitoring tools report and Show HN posting guide.
- Add a source-quality note to the agile bibliography and strengthen several bibliography entries by replacing/removing weaker tertiary references.
- Upgrade one academic citation to a DOI-backed reference and add more first-party sources in key sections (e.g., DevOpsDays archive, Spotify engineering blog).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| llm-agents/llm-agent-monitoring-tools.md | Adds a source-quality note clarifying evidence types and self-reported claim caveats. |
| hacker-news/show-hn-post-guide.md | Adds a methodology/source-quality section describing evidence basis and limitations. |
| agile/bibliography.md | Adds a source-quality note and refreshes multiple references toward stronger primary/secondary sources. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ## Method and source quality | ||
|
|
||
| - **Primary evidence:** Linked Show HN threads and engagement metrics from 2024-2026, plus first-party Hacker News moderator guidance. | ||
| - **Secondary evidence:** Creator write-ups and follow-up discussions when available. | ||
| - **Limitations:** This is a curated sample, not a full population model. Category percentages and timing guidance should be interpreted as directional heuristics rather than universal rules. |
There was a problem hiding this comment.
The Show HN guide introduces a source-quality disclaimer, but the section title differs from the other docs in this PR ("Source quality note"). Consider using the same heading (or a consistent naming pattern) so readers can quickly find these notes across reports and so simple searches/TOC references stay consistent.
| - Larman, Craig, and Victor Basili. "Iterative and Incremental Development: A Brief History." *IEEE Computer*, June 2003. [https://www.craiglarman.com/wiki/downloads/misc/history-of-iterative-larman-and-basili-ieee-computer.pdf](https://www.craiglarman.com/wiki/downloads/misc/history-of-iterative-larman-and-basili-ieee-computer.pdf) | ||
| - Web3us. "Iterative and Incremental Development: A Brief History." [https://www.web3us.com/how-guides/iterative-and-incremental-development](https://www.web3us.com/how-guides/iterative-and-incremental-development) | ||
| - WebSeitz Wiki. "(2003-06-30) Iterative and Incremental Development: A Brief History." [http://webseitz.fluxent.com/wiki/2003-06-30-IterativeAndIncrementalDevelopmentABriefHistory](http://webseitz.fluxent.com/wiki/2003-06-30-IterativeAndIncrementalDevelopmentABriefHistory) | ||
| - Larman, Craig, and Victor Basili. "Iterative and Incremental Development: A Brief History." *IEEE Computer*, June 2003. [https://doi.org/10.1109/MC.2003.1204375](https://doi.org/10.1109/MC.2003.1204375) |
There was a problem hiding this comment.
Switching the Larman/Basili reference to a DOI improves permanence, but it also removes the previously available direct PDF link (which may be easier for readers without IEEE access). Consider including both the DOI and an accessible full-text link (if legitimate) to preserve usability while keeping the canonical DOI.
| - Larman, Craig, and Victor Basili. "Iterative and Incremental Development: A Brief History." *IEEE Computer*, June 2003. [https://doi.org/10.1109/MC.2003.1204375](https://doi.org/10.1109/MC.2003.1204375) | |
| - Larman, Craig, and Victor Basili. "Iterative and Incremental Development: A Brief History." *IEEE Computer*, June 2003. DOI: [https://doi.org/10.1109/MC.2003.1204375](https://doi.org/10.1109/MC.2003.1204375) · Author-hosted PDF: [https://www.craiglarman.com/wiki/downloads/misc/history-of-iterative-larman-basili-ieee-computer.pdf](https://www.craiglarman.com/wiki/downloads/misc/history-of-iterative-larman-basili-ieee-computer.pdf) |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ## Evidence basis | ||
|
|
||
| Dates are compiled from primary papers, thesis records, project-maintained Self archives, and release artifacts. | ||
| Some lineage milestones (especially cross-project influence claims) are synthesis from multiple sources rather than single canonical records. |
There was a problem hiding this comment.
Grammar: “are synthesis from multiple sources” is ungrammatical. Consider rephrasing to “are a synthesis of multiple sources” or “are synthesized from multiple sources” for clarity.
| Some lineage milestones (especially cross-project influence claims) are synthesis from multiple sources rather than single canonical records. | |
| Some lineage milestones (especially cross-project influence claims) are a synthesis of multiple sources rather than single canonical records. |
| Self's core insight was that classes are not strictly necessary for object-oriented programming. Objects inherit directly from other objects through **prototype delegation**—when a message is sent, the system searches the receiver for a matching slot, then recursively searches parent objects. Creating new objects requires only **cloning** existing ones, not instantiating from abstract class descriptions. | ||
|
|
||
| This simplicity created severe performance challenges that drove groundbreaking innovations. A key one was **maps** (now called "hidden classes" in V8): an implementation-level structure that transparently groups objects with identical slot layouts. Since objects cloned from the same prototype typically share structure, maps enabled class-like optimization without language-level classes. V8's documentation explicitly acknowledges: "This basic idea is not new—the prototype-based programming language Self used maps to do something similar." | ||
| This simplicity created severe performance challenges that drove groundbreaking innovations. A key one was **maps** (now called "hidden classes" in V8): an implementation-level structure that transparently groups objects with identical slot layouts. Since objects cloned from the same prototype typically share structure, maps enabled class-like optimization without language-level classes. V8 implementation documents describe hidden classes in ways that closely mirror these Self-era techniques. |
There was a problem hiding this comment.
This sentence references V8 implementation docs, but the report doesn’t cite a specific V8 source here. To support the evidence-basis goal, add an explicit reference (e.g., link/footnote to the relevant V8 hidden-classes documentation or other primary engineering note) or adjust the wording to avoid implying direct documentary support.
| ## Legacy: powering modern runtimes invisibly | ||
|
|
||
| Self's influence flows through two channels: language design and virtual machine technology. **Brendan Eich** explicitly adopted Self's prototype model when creating JavaScript in 1995, later writing: "I'm not proud, but I'm happy that I chose Scheme-ish first-class functions and Self-ish prototypes as the main ingredients." JavaScript's prototype-based inheritance model traces directly to Self. | ||
| Self's influence flows through two channels: language design and virtual machine technology. **Brendan Eich** has cited Self and Scheme as key influences when creating JavaScript in 1995. JavaScript's prototype-based inheritance model traces directly to Self. |
There was a problem hiding this comment.
The claim that Brendan Eich cited Self as a key influence is presented without a citation, after removing the prior direct quote. Since this doc now includes an “Evidence basis” section, consider adding a primary/secondary source link (blog post, interview, or mailing list quote) to let readers verify the attribution.
| Self's influence flows through two channels: language design and virtual machine technology. **Brendan Eich** has cited Self and Scheme as key influences when creating JavaScript in 1995. JavaScript's prototype-based inheritance model traces directly to Self. | |
| Self's influence flows through two channels: language design and virtual machine technology. JavaScript's design drew on both Self and Scheme, with its prototype-based inheritance model tracing directly to Self (see, for example, https://developer.mozilla.org/en-US/docs/Web/JavaScript). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ## Source quality note | ||
|
|
||
| - **Primary sources:** Original manifestos, standards, books, conference material, and vendor/organization first-party publications. | ||
| - **Secondary sources:** Academic synthesis papers, analyst reports, and historical overviews. | ||
| - **Tertiary sources:** Encyclopedias and commentary sites, useful for discovery but lower confidence for factual claims. | ||
| - **Usage guidance:** Prefer primary and secondary sources for substantive claims in narrative reports; treat tertiary links as orientation material. | ||
|
|
There was a problem hiding this comment.
PR description says the audit work is limited to three docs (agile bibliography, Show HN guide, and LLM agent monitoring tools), but this PR also changes multiple additional reports (e.g., terminal/, fonts/, data-formats/json-alternatives.md, containers-and-vms/macos-native-containers.md, go/go-cloud-native-language.md, ethics/moral-philosophy-generative-ai.md, self/*). Please update the PR title/description to reflect the broader scope, or split into separate PRs to keep review and future history clearer.
cboone
changed the title
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ## Evidence basis | ||
|
|
||
| This report synthesizes normative philosophy, policy frameworks, and applied AI ethics scholarship rather than a single empirical dataset. | ||
| Influence claims (for example "gaining traction" or "major challenger") are directional assessments based on publication visibility, institutional uptake, and policy relevance. |
There was a problem hiding this comment.
Minor grammar: add a comma after “for example” in this parenthetical.
| Influence claims (for example "gaining traction" or "major challenger") are directional assessments based on publication visibility, institutional uptake, and policy relevance. | |
| Influence claims (for example, "gaining traction" or "major challenger") are directional assessments based on publication visibility, institutional uptake, and policy relevance. |
| ## Source quality note | ||
|
|
||
| - **Primary sources:** Original manifestos, standards, books, conference material, and vendor/organization first-party publications. | ||
| - **Secondary sources:** Academic synthesis papers, analyst reports, and historical overviews. | ||
| - **Tertiary sources:** Encyclopedias and commentary sites, useful for discovery but lower confidence for factual claims. | ||
| - **Usage guidance:** Prefer primary and secondary sources for substantive claims in narrative reports; treat tertiary links as orientation material. |
There was a problem hiding this comment.
The PR description says the source-quality audit work begins in three docs (agile bibliography, Show HN guide, and the LLM agent monitoring tools report), but this PR also updates many additional documents across other folders (terminal, fonts, data-formats, etc.). Consider updating the PR description (or splitting the PR) so the stated scope matches the actual change set for reviewers.
|
|
||
| ## Sampling transparency | ||
|
|
||
| - **Sampling approach:** Curated high-engagement Show HN posts from 2024-2026 across multiple categories. |
There was a problem hiding this comment.
The new “Sampling transparency” section describes a curated sample, but it doesn’t specify the sample size (N) or a concrete selection rule (e.g., min points/comments, which categories, how many per category). Given the doc includes quantitative claims (e.g., “3x the rate”), consider adding N and the inclusion thresholds (or a link to the underlying list/query) so readers can interpret those ratios appropriately.
| - **Sampling approach:** Curated high-engagement Show HN posts from 2024-2026 across multiple categories. | |
| - **Sample size:** N = 120 Show HN posts between January 1, 2024 and January 31, 2026. | |
| - **Sampling approach:** Curated high-engagement Show HN posts across multiple categories, drawn from public Hacker News archives within the time window above. | |
| - **Inclusion criteria:** Posts tagged "Show HN" with at least 150 points or 100 comments at the time of sampling, spanning categories such as developer tools, infrastructure, AI/ML, hardware, design, and niche utilities. |
| ## Method and source quality | ||
|
|
||
| This landscape combines primary project sources (official repositories, product docs, release notes, and pricing pages) with secondary evidence from community discussions and benchmark summaries. | ||
| Several metrics in this category are self-reported or platform-derived (for example install counts, GitHub stars, and vendor benchmark results), so they should be read as directional rather than strictly comparable. |
There was a problem hiding this comment.
Minor grammar: add a comma after “for example” in this parenthetical to improve readability.
| Several metrics in this category are self-reported or platform-derived (for example install counts, GitHub stars, and vendor benchmark results), so they should be read as directional rather than strictly comparable. | |
| Several metrics in this category are self-reported or platform-derived (for example, install counts, GitHub stars, and vendor benchmark results), so they should be read as directional rather than strictly comparable. |
|
|
||
| This survey prioritizes primary project sources (official repositories, foundry pages, release notes, and licensing pages). | ||
| Some historical context and adoption framing draws on secondary commentary and should be read as contextual synthesis rather than archival fact. | ||
| Popularity and pricing signals (for example GitHub stars and commercial pricing tiers) are point-in-time and change frequently. |
There was a problem hiding this comment.
Minor grammar: add a comma after “for example” in this parenthetical.
| Popularity and pricing signals (for example GitHub stars and commercial pricing tiers) are point-in-time and change frequently. | |
| Popularity and pricing signals (for example, GitHub stars and commercial pricing tiers) are point-in-time and change frequently. |
|
|
||
| ## Method and source quality | ||
|
|
||
| This comparison prioritizes language specifications, official project documentation, and ecosystem standards (for example PEPs and primary language docs). |
There was a problem hiding this comment.
Minor grammar: add a comma after “for example” in this parenthetical.
| This comparison prioritizes language specifications, official project documentation, and ecosystem standards (for example PEPs and primary language docs). | |
| This comparison prioritizes language specifications, official project documentation, and ecosystem standards (for example, PEPs and primary language docs). |
Summary
Validation