update to trafilatura 2.0.0

[rahulbot]

We should evaluate upgrading to the latest trafilatura. I see a relevant note:

"trafilatura==1.8.", # must stay below v1.11. to allow easy extraction of canonical_url

But I see that 2.0.0 has improvements to the URL extraction listed in their release notes, so we should evaluate upgrading as part of our regular maintenance. Is this "easy extraction of canonical_url" issue still something we're worried about? What was that for?

[philbudne]

Is this "easy extraction of canonical_url" issue still something we're worried about? What was that for?

We used it to process HTML "blindly" fetched from S3 (without an accompanying CSV file indicating the (original; ie; not final redirected) URL (or a CSV file that has the downloads_id, but not the URL). It is not required for new news, or for processing of past years (unless we discover more CSV files that lack a URL column). NOTE: we ARE currently saving the extracted canonical_url (when available) in WARC files, but it isn't used. earlier epochs of WARCs don't have it. And if someone had the need to extract it at some later point, they could do so, since the WARC file contains the original HTML. ISTR: The new(er) minor version of trafilatura I looked at added an argument (defaulting to off) to do additional extraction steps (rather than doing the compatible thing, and defaulting to old behavior). I only looked at the new trafilatura code, and did not: 1. test that it works for us with EITHER setting 2. find out how much cheaper the default is, 3. or conversely if the non-default setting is any more expensive than the current default. My gut feeling: The package maintainer made a breaking change in a minor release means even minor version upgrade need to be rigorously tested. I'd suggest starting with a daily synthetic RSS file, running a dev stack that fetches and processes everything, and then reset ES and do a run with the new trafilatura, and comparing the results (maybe by reading the old WARC files, and comparing with the new ES contents?). IMO: Upgrading for improvements in extracted URL quality alone isn't worth the bother/risk, at least not at this moment.