Crashes with "KeyError: 'entries'" #9
Description
axeman pulled down a few TBs of CSVs from various CTLs and then crashed:
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72408700-72408766.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72407530-72407596.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72410455-72410521.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72410195-72410261.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72412535-72412601.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72414160-72414226.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72415460-72415526.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72415850-72415916.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72417410-72417476.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72419880-72419946.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72419360-72419426.csv written!
[INFO:root] 2019-10-17 06:47:18,526 - Completed DigiCert Log Server 2, stored at /tmp/ct2.digicert-ct.com_log.csv!
[INFO:root] 2019-10-17 06:47:18,526 - Finished downloading and processing ct2.digicert-ct.com/log
[INFO:root] 2019-10-17 06:47:18,526 - Downloading certificates for DigiCert Yeti2019 Log
[INFO:root] 2019-10-17 06:47:30,526 - Starting processing coro and process pool
[1] Making dir...
[INFO:root] 2019-10-17 06:47:31,057 - Getting things to process...
[INFO:root] 2019-10-17 06:47:31,058 - Queue Status: Processing Queue Size:0 Downloaded blocks:50/1130984 (0.0044%)
Traceback (most recent call last):
File "/usr/local/bin/axeman", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 295, in main
loop.run_until_complete(retrieve_certificates(loop, concurrency_count=args.concurrency_count))
File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 112, in retrieve_certificates
await download_tasks
File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 53, in download_worker
for index, entry in zip(range(start, end + 1), entry_list['entries']):
KeyError: 'entries'