Prevent duplicate commit insertion on facade re-processing

[MoralCode]

the commits table has a primary key that is derived only from a sequence (i.e. an autoincrementing integer)

When facade analyze_commits_in_parallel runs (to insert all new commit change information into the commits table), it does not utilize Augur's existing upsert (bulk_insert_dicts or similar) logic. Instead it simply inserts always, causing new IDs to be generated and new rows to be added.

If the run of analyzing commits is a rerun (i.e. the repo previously was fully collected, but the admin reset the last collection date to force recollection, meaning many of the commits are already in the table), this will simply generate duplicate rows, contributing to the size growth of one of the largest tables in Augur.

In order to use upserts for the commits table, we need a compound primary key based on the actual data. Given this table is actually more accurately described as commit_changes (#3682), i propose this constraint UniqueConstraint("repo_id", "cmt_commit_hash", "cmt_filename", name="commit-changes-unique"),.

[MoralCode]

This is blocked by the linked commit because the linked commit causes any commit with multiple file deletions to have two rows containing the same repo id, commit hash, and filename, breaking this the uniqueness constraint.

There may also be other reasons for the uniqueness violation that have yet to be uncovered

[sgoggins]

I ran this query on the giant Augur instance I maintain:

select * from commits where repo_id=1
order by cmt_author_date, cmt_commit_hash, cmt_filename; 

And I believe I am not seeing duplicate rows. The results are attached to this comment. I am open to being proven wrong, but where and how are you determining that duplicate rows are being inserted?

commit-test.csv.zip