Restore _ordered fields handling

[kristjaneerik]

Here I'm

• restoring the functionality required to fix Clinical Significance Order Lost in Allele Grouping #48
• adding tests for group_by_allele that I think are a bit easier to follow (had them in [WIP] Propagating ordered fields #33)
• skipping chrY in add_gnomad_fields.py -- I had previously done this in [WIP] Propagating ordered fields #33 because it led to issues. See comments below.
• fixing misc PEP8 issues/typos that were on the way
• not fetching normalize.py if it exists

[kristjaneerik]

I checked in results from the newest, 2017-12 dataset.

I also ran master on that dataset and diff-ed the output for b37/single using the src/comparing_clinvar_alleles.py script I added. Only differences are in the _ordered fields for most files, as expected.

The clinvar_alleles_with_gnomad_exomes files differ for two variants (variation_ids 9741 and 9754), which end up getting nan values for some columns for the run with this branch. They are both variants in SRY (e.g. https://www.ncbi.nlm.nih.gov/clinvar/variation/9741/), so I suspect it's because of the check I added for ignoring chrY for gnomad. Happy to revert that. I added it because there were issues when I ran the code many months ago. I used the r2.0.2 release for gnomAD now, but was using an older release previously, which didn't have any chrY variants while this one does. Thoughts?

One additional note is that I used normalize.py from here: ericminikel/minimal_representation#1

[kristjaneerik]

@bw2 any comments on this? Happy to run it on the 2018-02 release.

[holtgrewe]

Any news on this?

[bw2]

@holtgrewe You can use this branch, but we have deprecated the overall repo and are no longer maintaining it. (please see the note in the README)

[holtgrewe]

Hi @bw2. I'm sorry to hear this. Are you aware of some alternative source for a "digested" ClinVar database?