Using custom objects with paths

[nh13]

Now that we can pass around custom (value/metadata) classes and have them cached, it would be wonderful if Nextflow could recognize Path members/properties so they are localized and available for the process that uses the class. For example:

import nextflow.io.ValueObject

@ValueObject
class SampleData {
    /** The sample name */
    String name
    /** The path to the read one FASTQ */
    Path r1
    /** The path to the read two FASTQ */
    Path r2
}

There could be a way annotate r1 and r2 (or return the list of paths etc) for nextflow to localize them.

[mjhipp]

I wonder if it would be possible by using a different type as a member of the ValueObject? I am not very familiar with the source code, so I assume this wouldn't actually work, but something like:

import nextflow.io.ValueObject
import nextflow.processor.TaskPath

@ValueObject
class SampleData {
    /** The sample name */
    String name
    /** The path to the read one FASTQ */
    TaskPath r1
    /** The path to the read two FASTQ */
    TaskPath r2
}

Or maybe nextflow.file.FileHolder?

[pditommaso]

Would just String be enough for paths? internal classes are not meant to be used in pipeline scripts.

[nh13]

I'd love to have things properly typed (eg. Path).

[pditommaso]

In principle Path should work as well. What's the problem with it?

[nh13]

Does nextflow inspect the object for members of type Path and localize already?

[pditommaso]

What do you mean for localize?

[nh13]

If I pass in a SampleData class in the script, the r1

process foo {
  input:
     val sample // Type: SampleData
     path fasta
  output:
    ???
  script:
    """
    bwa mem ${fasta} ${sample.r1} ${sample.r2}
    """
}

[mjhipp]

Files being localized from the working directory to the analysis node (mainly for cloud workflows)

[pditommaso]

we are entering in a minefield .. 😄 no, the file staging (ie localisation) is determined by the path declaration. The pattern you are suggesting is interesting but not supported.

[ryankelley-lm]

There's a variety of situations where this capability would make it less error prone to deal with collections of genomic files, including VCF (+ index), BAM (+ index), FASTA (+ index). In the above example, you could imagine additional behaviors such as "is_paired" to allow scripts to handle single- or paired-end data in a more expressive way. Is this something that would completely break the process handling design or is there an opportunity for a user contribution in this area?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nh13]

Still relevant, here’s what I do:

import groovy.transform.Immutable
import nextflow.io.ValueObject
import nextflow.util.KryoHelper

@ValueObject
@Immutable(copyWith=true, knownImmutables = ['path'])
class Metadata {
    static { 
        // Register this class with the Kryo framework that serializes and deserializes objects
        // that pass through channles. This allows for caching when this object is used.
        KryoHelper.register(Metadata)
    }

    Path path
}