Skip to content

RefGenie - add assets currently used from AWS-iGenomes #1086

@ewels

Description

@ewels

Split from #592, related directly to #1084


We want to transition away from using AWS-iGenomes to using the central RefGenie server(s) to host reference genome assets for the nf-core pipelines. To do this, we need to make sure that everything we currently have is available on a AWS refgenie server somewhere. Once all assets are mirrored we can do a clean swap in the config.

Going from igenomes.config, here is a check-list:

Summary of genomes
  • GRCh37
  • GRCh38
  • GRCm38
  • TAIR10
  • EB2
  • UMD3.1
  • WBcel235
  • CanFam3.1
  • GRCz10
  • BDGP6
  • EquCab2
  • EB1
  • Galgal4
  • Gm01
  • Mmul_1
  • IRGSP-1.0
  • CHIMP2.1.4
  • Rnor_6.0
  • R64-1-1
  • EF2
  • Sbi1
  • Sscrofa10.2
  • AGPv3
  • hg38
  • hg19
  • mm10
  • bosTau8
  • ce10
  • canFam3
  • danRer10
  • dm6
  • equCab2
  • galGal4
  • panTro4
  • rn6
  • sacCer3
  • susScr3
Detailed version with asset types
  • GRCh37
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • GRCh38
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
    • blacklist
  • GRCm38
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • TAIR10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • EB2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • UMD3.1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • WBcel235
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • CanFam3.1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • GRCz10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • BDGP6
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • EquCab2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • EB1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • Galgal4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • Gm01
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • Mmul_1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • IRGSP-1.0
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • CHIMP2.1.4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • Rnor_6.0
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • R64-1-1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • EF2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
  • Sbi1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • Sscrofa10.2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • AGPv3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • hg38
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
    • blacklist
  • hg19
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • mm10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • bosTau8
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • ce10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
  • canFam3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • danRer10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • dm6
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • equCab2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • galGal4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • panTro4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • rn6
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • sacCer3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • readme
    • mito_name
    • macs_gsize
  • susScr3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name

From @nsheff:

For updating: We drive the server instances off a git repository, here: https://github.com/refgenie/refgenomes.databio.org

RIght now, it's semi-automated. Eventually we want it to be that you just update the repo with a PR, and when merged it will deploy automatically. For now, though, we have to build the things manually, but it's all scripted from that repository. The asses are all just annotated with a PEP. So, to add a genome you'd add it to this CSV file: https://github.com/refgenie/refgenomes.databio.org/blob/master/asset_pep/genome_descriptions.csv

Then you'd add whatever inputs are required to build the assets to this file: https://github.com/refgenie/refgenomes.databio.org/blob/master/asset_pep/recipe_inputs.csv

See also:

Metadata

Metadata

Labels

templatenf-core pipeline/component template

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions