-
Notifications
You must be signed in to change notification settings - Fork 263
Draft: Advanced side quests for custom training #756
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
pinin4fjords
wants to merge
147
commits into
master
Choose a base branch
from
custom-training-advanced-topics
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Three new side quests for advanced Nextflow topics: 1. Workflow Output Definitions (45 mins) - New publish/output block syntax - Dynamic paths based on metadata - Index file generation 2. Running on Cloud Executors (30 mins) - Executor concepts and options - AWS Batch configuration patterns - S3 integration and cost management - Conceptual/config-focused (no cloud account needed) 3. Plugin Development (90 mins) - Full hands-on plugin creation - Custom functions with @function annotation - Build, test, and use locally - Complete working nf-greeting plugin solution Also adds Gradle to devcontainer for plugin development support. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
✅ Deploy Preview for nextflow-training ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
Nextflow linting complete! ❌ 5 files had 22 errors 💡 Tip: Click filename locations to go directly to that code. View all 22 issues
View formatting changes
|
- Align markdown table columns - Expand devcontainer JSON to multi-line format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Makefiles require tabs for recipe indentation (Make syntax requirement). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add new Section 1 covering plugin usage before development content
- Include examples of installing plugins via plugins {} block
- Show how to import and use plugin functions with include syntax
- Reference nf-schema as practical example (from Hello nf-core)
- Add table of popular community plugins
- Update learning goals to include plugin usage
- Renumber subsequent sections (1→2, 2→3, etc.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The gradle-sdkman feature fails with SDKMAN permission errors during build. Plugin projects include a Gradle wrapper (./gradlew) that downloads Gradle automatically, so we only need Java pre-installed (provided by java:1 feature). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add prominent warning that this is an advanced topic - Add info box explaining Java, Groovy, and Gradle for newcomers - Highlight that "Using existing plugins" section is most important - Add detailed breakdown of Groovy syntax with comparisons to Python - Explain what unit tests are and why they matter - Add troubleshooting tips for build failures - Mark "Going further" section as reference material - Add note that Gradle wrapper downloads Gradle automatically These changes make the tutorial more accessible to participants without a software engineering background. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The command requires both plugin name and organization name arguments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update tree output to show actual generated files - Change package from nextflow.greeting to training.plugin - Rename classes to NfGreetingExtension, NfGreetingPlugin, etc. - Update all code examples and file paths to match - Update test class names in examples and expected output - Fix build.gradle example to show correct extensionPoints 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use correct plugin version (0.0.1-alpha4) - Use correct nextflowVersion (24.10.0) - Remove NfGreetingObserver from extensionPoints (not included by default) - Add publishing block with registry configuration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Show the actual generated template code and the changes needed, making it clearer what participants need to modify. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The generated Makefile uses 'assemble' not 'build' for compilation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Shows the Gradle download on first run and sets expectations for subsequent faster builds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reassure beginners that deprecation warnings are safe to ignore and explain what each part of the output means. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Clarify that we're creating a new test file (not editing existing) - Use Spock framework syntax to match generated project style - Add note explaining Spock's given/expect syntax - Update test output to show Spock-style test names 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Gradle hides test details on success, which can confuse beginners. Added explanation that BUILD SUCCESSFUL means tests passed, with optional --info flag for verbose output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The --info flag produces verbose Gradle output, not clean test results. Instead, point users to the HTML test report for detailed results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Show how to use python http.server to view the Gradle test report in VS Code, with screenshot showing all tests passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The starter files already have the plugin configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Shows the starter file and the changes needed to use plugin functions, with highlighted lines and explanation of key changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The starter config has the plugin commented out. Users need to uncomment it AND add the @0.1.0 version for local plugins. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Added note explaining the startup/complete messages come from the NfGreetingObserver included in the plugin template. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Added section 1.6 with a practical exercise using nf-hello plugin. Users create a simple workflow that uses the sayHello function, seeing plugin download and usage in action. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Added 'Why use plugins instead of local functions?' section - Restructured exercise to show local function first, then plugin - Before/After tabs show the code change clearly - Explains the value proposition: versioned, shareable code 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Connect trace observers to the messages users already saw - Add Before/After exercise to customize observer messages - Add table of available observer hooks - Mark operators as reference (more complex) - Update publishing section with clearer steps 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ion types - Add progress table showing coverage of extension types from table 2.1 - Add TaskCounterObserver creation exercise (new file, not editing existing) - Add custom operator (shoutAll) with full Before/After convention - Add configurable decorator exercise with Before/After - Add conceptual coverage of Factories, Executors, and Filesystems - Update publishing section with complete instructions - Clean up solution main.nf formatting Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix duplicate 5.4 numbering (install locally → 5.5) - Split monolithic section 7 into focused sections: - Section 7: Trace observers - Section 8: Custom operators and factories - Section 9: Configuration - Section 10: Publishing your plugin - Add Takeaway/What's next transitions between new sections Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove extra features from solution files that aren't covered in the tutorial: - Remove unused prefix/suffix config options from nextflow.config - Simplify output message in main.nf to match tutorial - Remove extra comment block from random_id_example.nf Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Code block fixes - Fix 6 incorrect hl_lines values in 02_joint_calling.md - Fix wrong title in 03_modules.md (tests/main.nf.test → genomics-3.nf) - Fix wrong language in 04_testing.md (console → groovy) ## Syntax modernization - Convert 25 Before/After blocks to Material for MkDocs tabbed syntax (=== "After" / === "Before") across all 4 parts Found during tutorial walkthrough validation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add sections on EKS fundamentals and configuration (sections 4-5) - Document PVC-based approach with Amazon EFS for shared storage - Include Kubernetes RBAC configuration example - Add eks.config example configuration file - Update profiles to include EKS option - Add documentation source links for verification - Mention Fusion as optional Platform enhancement (not core requirement) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Convert AWS Batch and EKS architecture diagrams from ASCII art to Mermaid flowcharts for better maintainability and rendering. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add warning that EKS is more complex than AWS Batch - Explain Kubernetes concepts (pods, namespaces, PVCs) for newcomers - Clarify when to choose EKS vs AWS Batch - Focus on Nextflow config, not infrastructure setup - Simplify: "ask your admin for namespace, service account, storage" - Remove detailed YAML examples (link to docs instead) - Update summary checklist to match simplified approach Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Explain how Nextflow authenticates with AWS: - Environment variables - AWS credentials file (~/.aws/credentials) - IAM instance profile Clarify distinction between credentials for Nextflow (to submit jobs) vs jobRole for jobs themselves (to access S3). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add inline comment explaining container is a fallback - Explain in docs that most pipelines define per-process containers - Fallback is useful for simple bash processes without explicit containers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add links to Platform pricing/info - Update section 7.4 title to indicate Platform feature - Simplify Fusion code example - Update takeaway to mention Platform requirement Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Content was generic and not particularly useful for this guide. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Changed from incorrect executor block syntax to correct process scope: - executor.name -> process.executor - executor.queueName -> process.queue - Consolidated container setting in process block These corrections align with the official Nextflow documentation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove make comparison for Gradle (advanced concept) - Replace reverseString() with samplesheetToList() as example (more recognizable) - Reduce hyperbole throughout, get to the point faster - Add Slack as example for extending pipelines - Clarify that community can add features without modifying Nextflow core - Simplify tip about using plugins section - Add notes about latest version default and runtime download - Restructure "Try it" section: remove local function comparison, use config-only approach, remove CLI flag diversion - Add note about development burden being on plugin developer and Nextflow handling plugin management Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The plugin registry already lists available plugins, so this section was redundant. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace em-dash elaborations with periods or semicolons - Use colons instead of hyphens for list item explanations - Remove unnecessary trailing explanations - Simplify sentences to be more direct Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix section 2.7 -> 2.6 after removing Popular plugins section - Run prettier to fix table formatting Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Removing plugin files so they can be cleanly merged from the plugin-development-side-quest branch, which has the reviewed and updated version.
- Remove LLM-style "Let's" phrases throughout - Add note explaining -output-dir is a native Nextflow feature (vs nf-core's params.outdir convention) - Add hl_lines to code blocks in section 1.1 highlighting publishDir and meta map pattern Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update the starter code to use publishDir closures with meta.language for a fairer comparison with workflow output blocks. Both approaches now demonstrate dynamic paths, so the lesson focuses on the real benefits of workflow outputs (centralization, index files) rather than implying publishDir can't do dynamic paths. - Starter modules/greetings.nf now uses publishDir closures - Section 1.1 shows dynamic publishDir from the start - Section 1.2 shows language-organized output - Section 2 introduces workflow outputs with static paths - Section 3 restores dynamic organization in the output block Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Tree provides a clearer visual representation of the directory structure. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove "dynamic paths" from key benefits (publishDir can do this too) - Replace comparison table with note that publishDir is legacy - Recommend workflow outputs for all new pipeline development Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Links to official docs for defining custom config as a proper class with annotations for IDE and linter support. Addresses review feedback from @adamrtalbot. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add warning admonition in section 1.3 (Limitations) where readers decide whether to learn this. Remove redundant note from summary. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
05053e4 to
3caaad8
Compare
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- aws.config: Replace invalid executor.queueName with process.queue - aws.config: Use process.executor instead of executor.name - local.config: Separate executor selection from resource limits Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains custom training content for specific teaching sessions. It is not intended to be merged into the main training materials at this time.
Summary
Three new side quests covering advanced Nextflow topics:
1. Workflow Output Definitions (45 mins)
publish:andoutput {}block syntax (replacingpublishDir)2. Running on Cloud Executors (30 mins)
3. Plugin Development (90 mins)
@Functionannotationnf-greetingplugin solution includedAdditional Changes
Testing
Notes
Keep on branch for custom training sessions. May be adapted for general availability later.
🤖 Generated with Claude Code