Produce both filled and unfilled files?

[sipa]

Unsure if this is the right place to discuss/report, but I want to write it up so it isn't forgotten.

I notice that for the latest collaboratively-produced asmap file, the output was switched to use the --fill option to asmap-tool.py to produce a smaller binary. I think it makes sense that the eventually-included dump in the Bitcoin Core binary is a filled one, but for auditing purposes it is probably better that the collaboratively-produced file isn't, so the ability to compare/inspect isn't hurt by the filling.

Note that it's possible to perform the filling step afterwards, converting an unfilled binary .dat file to a filled one (asmap-tool.py encode --fill asmap_unfilled.dat asmap_filled.dat) without needing the original text format data. I also believe that the conversion is deterministic (as long as the asmap-tool.py encoding algorithm doesn't change) so with the same unfilled .dat file one would be able to recreate the same corresponding filled one.

[fjahr]

Yeah, we can certainly do that, it's just one extra step in the process and a bit more data in the repo. I think we should explain the difference between filled and not filled in the readme here then as well (copy+paste from the new docs in core by @jurraca should be easy I think) but then we can start doing that with the next run.

FWIW, users could currently also produce the missing unfilled file from the raw output uploaded in the issues (for example #22 (comment)) but I think it's good if we confirm the hash of the result of both files by multiple people and add them to the repo. It's almost no extra work for us and saves users of the data and researchers some hassle.

[fjahr]

We have included both the filled and the unfilled file for the latest asmap we built last week (#30) and we'll continue with this going forward, so closing this now.