Analyze delta between pptools and PPWB

[tangledhelix]

The DP team already provides the PPWB tools, which already support HTML5 and have a big overlap with pptools. It may be the better strategy to discern what PPWB doesn't have which pptools does, and investigate adding those features to PPWB.

At this point, 2 of 3 major parts of pptools still have no HTML5 support; and the HTML5 support currently provided was more or less copied out of PPWB by a contributor, meaning that portion is largely identical in code and function already.

See also #12 which discusses a Python core module removal that this codebase relies upon.

[tangledhelix]

PPtools tools / modes

• upload txt/html files, they persist in system (up to (n) files?)

Compare two files

• compare 1 to 2, or 2 to 1 (dwdiff mode)
• transforms for html file:
  • extract/proc footnotes separately
  • ignore case during compare
  • transform smallcaps case [upper,lower,title] before compare
  • add [Illustration: or [Sidenote: to reduce diffs shown
  • greek transliteration (less useful these days)
  • has a default transformation (CSS) applied for compare
    • default mostly does some whitespace twiddling and removes page numbers
  • can suppress nbsp (U+00A0) between numbers (why? ad prices?)
  • can suppress zero-width space (U+200B)
  • can downgrade smart quotes
  • custom transformational CSS
    • example given: surround footnote marker with brackets
    • example given: hide thought breaks
    • known use case: surround bold with =
• transforms for text file:
  • suppress [Footnote:
  • suppress [Illustration:
  • suppress [Sidenote:
• general:
  • suppress [**notes]
  • regroup split wo-* *rds
  • "text cleaning" - have to look in code for what that means
  • "silence formatting differences" - have to look at what it means

Checks (marked as "BETA")

• text
  • duplicated lines
    • in my test it detected one which did exist twice in the book ...
  • footnote checker
  • block spacing (not sure how much people use this?? it's non-intuitive)
  • list of text headers (e.g. what epub should have as TOC)
• html
  • main doc language, and other found languages
  • look for inline styles
  • show title & author of book
  • TOC as ebookmaker would build it
  • images
  • footnotes
  • page numbering (arabic vs roman)
  • suspect dot where comma expected
  • quotes (on test project it told me to use curlies)

Find Languages

• This seems to have the columns out of order
• shows an HTML tag, language, and the content tagged, for review presumably

[tangledhelix]

PPWB tools / modes

pptext

• can enable/disable individual tests
• supports goodwords file
• supports 12 languages wordlist (spellcheck), can check as many as desired
• test list:
  • spellcheck
  • edit distance
  • text checks
    • hyphen consistency
    • hyphen-space consistency
  • jeebies
  • smart quote check
• offers easily downloaded results for storage
• shows file encoding
• shows paragraph count
• scans smart quotes for suspects
• spellcheck report has results grouped by word, then line number; word is highlighted
• edit distance shows similar but slightly different words even if not proximate in file
  • shows words, count of each, lines found (not all; but maybe in verbose shows all?)
• text analysis report
  • hyphenated vs non-hyphenated of same word pair (wordpair, word-pair)
  • hyphenated vs spaced of same word pair (word pair, word-pair)
• checks for any asterisks in file
• checks for multiple adjacent spaces
• checks for trailing spaces
• checks for certain chars & * [ ], also ligatures, accented letters...
• spacing pattern report; i think this is to show if the 421... or 4121... pattern breaks
  • formatted more usably than the pptools variant of this.
• reports short lines
• reports long lines
• reports repeated words (like they were very very kind or they had had lunch)
• reports suspected ellipsis issues
• reports adjacent dashes (which weren't turned to — I assume)
• scanno check (bad/had, band/hand, and so on)
• curly quote check
• mixed-case within a word (Anglo-Saxons)
• "book level check" - not sure what it means
• paragraph level checks (suspect of improper end of paragraph and stuff)
• jeebies report

pphtml

• as with pptext, offers a downloadable link if you want to save
• image tests (all are used; all targeted by HTML file exist... file sizes, dimensions
• link checks (no externals; cover is linked, links resolve properly
• PPV tests
  • compare title to h1
  • no <pre> tags
  • charset validation
  • doctype header
  • alt tag tests
  • lang code, asks to confirm
  • document outline (toc)
• check class=chapter and h2 count up to the same
• CSS tests: show defined & used for comparison

ppmsq

• converts straight quotes to curly quotes
• out of scope for pptools compare; not a validator / checker

ppcomp

• options
  • ignore case
  • extra/process footnotes separately
  • html only
    • add [Illustration: or [Sidenote:
    • use greek transliteration in title
    • suppress nbsp between numbers
    • suppress NoBreak (U+2060)
    • suppress zero width space (U+200B)
    • convert smallcaps to uppercase
  • text only
    • suppress [Footnote or [Illustration
  • general
    • remove [**notes]
    • regroup split wo-* *rds
    • silence formatting differences
    • text cleaning: best effort / none / proofers
• allows report download
  • the report itself is a dwdiff style diff with above options applied

Misc

• ppwb page also links to online tools, just noting here; they're external things
• PG's online ebookmaker
• W3C validator (not Nu...)
• W3C CSS validator
• (note to self: should Nu be linked there too?)

[tangledhelix]

Legend:

• T Text file
• H HTML file
• R compare with file from the Rounds

ppcomp ✅

Feature	PPtools	PPWB	Propose adding to PPWB?
Compare two files (T, H, R)	✅	✅
Enable/disable specific checks	✅	✅
Process footnotes separately	✅	✅
Ignore case when comparing	✅	✅
H Small-caps transform	✅	✅
H Add [Illustration:	✅	✅
H Add [Sidenote:	✅	✅
H Greek transliteration in title	✅	✅
H Default CSS transform (whitespace, page numbers)	✅	❎	DistributedProofreaders/ppwb#45
H Suppress nbsp (U+00A0) between numbers	✅	✅
H Suppress zero-width space (U+200B)	✅	✅
H Suppress word join (NoBreak) (U+2060)	❌	✅
H Downgrade smart quotes	✅	❌	Ignoring; smart quotes are strongly encouraged
H Custom CSS transformation	✅	❎	DistributedProofreaders/ppwb#44
T Suppress [Footnote:	✅	✅
T Suppress [Illustration:	✅	✅
T Suppress [Sidenote:	✅	❎	DistributedProofreaders/ppwb#42
R Ignore [**notes]	✅	✅
R Regroup split wo-* *rds	✅	✅
R Text cleaning	✅	✅
R Silence formatting diffs	✅	✅

pptext ✅

Feature	PPtools	PPWB	Propose adding to PPWB?
Support "good words" file	❌	✅
Verbose mode	❌	✅
Spell-check	❌	✅
LOTE spell-check	❌	✅
Edit distance	❌	✅
Enable/disable specific tests	❌	✅
Duplicate lines	✅	❎	DistributedProofreaders/pptext#17 DistributedProofreaders/pptext#19
Footnotes	✅	❌	The pptools footnote analysis doesn't seem to work with commonly used footnote formatting? Ignoring.
Block spacing	✅	✅
Short lines	❌	✅
Long lines	❌	✅
Repeated words (e.g., "very very")	❌	✅
Suspect ellipses	❌	✅
Adjacent dashes	❌	✅
Scannos (had/bad, hear/bear, ...)	❌	✅
Chapter / Section headers	✅	❎	DistributedProofreaders/pptext#16
Hyphenated/non consistency	❌	✅
Hyphenated/spaced consistency	❌	✅
Jeebies	❌	✅
Mixed-case within word (Anglo-Saxon)	❌	✅
Improper paragraph end	❌	✅
Smart quotes check	❌	✅
Paragraph count	❌	✅
Asterisk check	❌	✅
Adjacent spaces check	❌	✅
Trailing spaces check	❌	✅
Check special characters (&, brackets, accents, ...)	❌	✅
Standalone digits	❌	✅
Mixed letter/number words	❌	✅
Today/To-day, northeast/north-east, etc... "book level checks"	❌	✅
Unexpected paragraph end / missing para break	❌	✅

pphtml

Feature	PPtools	PPWB	Propose adding to PPWB?
Main document language	✅	✅
Other languages	✅	❌
Inline style tags	✅	❌	Declining to propose; they're used pretty commonly e.g. by Guiguts illo placement tooling. They're not an "error".
Extract title/author	✅	✅
ToC (EPUB-style)	✅	✅
Image checks	✅	✅
Link checker	❌	✅
PPV checks	❌	✅
Structural check (H2 & .chapter have same count)	❌	✅
CSS report	❌	✅
Footnote check	✅	❌	pptools check doesn't really work anyway.
Page numbers (Arabic v. Roman)	✅	⏳	DistributedProofreaders/pphtml#23 (pending)
Suspect . where , expected	✅	❌	Yes?
Check quote marks	✅	❌	Maybe; check what it does

Language checks (pptools only)

Feature	PPtools	PPWB	Propose adding to PPWB?
Language check	✅	❌
Find tagged languages, display each phrase	✅	❌