Handling of pre-releases and removed packages

[ts-mk]

There seems to be an issue that packages without any normal release are added to the list. This is the case with flake8-bbs (BBS). The package had only few first alpha versions published to PyPI before I removed them together with the entire package (it got replaced by flake8-bas).

I know that for example pip would install a pre-release version even if --pre flag is not used if there is no other version but a pre-release one, but I don't think that's a good behaviour worth copying.

[bskinn]

Hm. Thanks for pointing this out, but a key part of the goal of this repo is to try to provide as complete a picture as possible of all of the error codes that have been defined in the flake8 ecosystem, in order to guide new extension authors toward choosing as-yet-unclaimed codes/code families.

I specifically don't want to exclude pre-release-only projects from the list, because those projects are staking what may become a stable, durable claim to codes...and other early-stage projects would, all told, be best served by staying away from them.

Projects that are no longer up on PyPI, though -- yeah, those it makes no sense to include. I hadn't realized that the PyPI Simple API kept around projects even after their deletion.

I would think a check for a 404 at https://pypi.org/project/{project_name} should suffice to screen for deleted projects... what do you think, @ts-mk?

[ts-mk]

I specifically don't want to exclude pre-release-only projects from the list, because those projects are staking what may become a stable, durable claim to codes...and other early-stage projects would, all told, be best served by staying away from them.

I see. That reasoning makes sense.

I would think a check for a 404 at https://pypi.org/project/{project_name} should suffice to screen for deleted projects...

Yeah, that sounds good.

[bskinn]

A clean way to exclude deleted packages is to use the new JSON API. See #22 (comment)

[bskinn]

A clean way to exclude deleted packages is to use the new JSON API. See #22 (comment)

Looks like this isn't 100% reliable, there were still some deleted packages that showed up. Checking the 404 at the individual-package JSON endpoint seems better.

[bskinn]

Also hitting some problems where new packages are capitalized/punctuated differently than old packages. Not sure if it's a rename, or a destroy/recreate.

In any event, due to PEP 503 name normalization, packages keying on simple string matching of names isn't enough. If a package exists in the old data with a normalized name that matches the normalized name of a package in the new data, then those two packages should be treated as the same package.

We need:

• The package name in the new data must be retained as-is
• If any package in the old data has a name that, after normalization, matches the normalized name of a given package in the new data, then those two packages should be checked for version differences.
  • In this case, when the new data is saved for later use as the old data, the name of the new package should be stored as-is (although this will be less important, with the normalized matching implemented)

[bskinn]

We need:

* The package name in the new data must be retained _as-is_

* If any package in the old data has a name that, after normalization, matches the normalized name of a given package in the new data, then those two packages should be checked for version differences.
  
  * In this case, when the new data is saved for later use as the old data, the name of the new package should be stored _as-is_ (although this will be less important, with the normalized matching implemented)

Believe this is implemented correctly in cc8fac9 for the new and updated package detection in prep for the RSS feed and the tweet generation.

For the initial package screening in f8_list.py, the normalization was implemented for new package detection in c5d9bdd, and for updated package detection in 8c95577.

That said there's a lot more logic cleanup that needs to happen here. What lists of packages have what semantic meanings, what packages are and are not carried through the different data forms....