diff --git a/PACKAGE_INSTALLATION_GUIDE.md b/PACKAGE_INSTALLATION_GUIDE.md new file mode 100644 index 00000000..8f36cf00 --- /dev/null +++ b/PACKAGE_INSTALLATION_GUIDE.md @@ -0,0 +1,189 @@ +# Pre-installing Packages in Workbench Apps + +Users often want specific packages pre-installed in their apps to avoid running `pip install` or `install.packages()` every time. This guide shows two simple approaches. + +--- + +## Approach 1: Natural Language with Claude (Easiest!) + +Just ask Claude what you need, and it will help you install packages: + +**Examples:** +- "I need ggplot2" +- "Set me up for machine learning" +- "I want tools for genomics analysis" +- "Install deep learning packages" + +Claude understands natural language and will either: +- Generate an installation command for you to run +- Pre-configure your app with the packages + +**Supported domains:** +- Machine Learning, Deep Learning, NLP +- Data Visualization, Dashboards +- Bioinformatics, Genomics, Single-cell +- BigQuery/GCP, Statistics, Time Series +- And more... + +--- + +## Approach 2: Pre-install with common-packages Feature + +Use the built-in `common-packages` feature in your `.devcontainer.json`: + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "tensorflow scikit-learn pandas numpy matplotlib", + "rPackages": "Seurat,DESeq2,ggplot2,tidyverse" + } + } +} +``` + +**Python packages:** Space-separated +**R packages:** Comma-separated (NO SPACES) + +**When to use this:** +- You know exactly which packages you need +- You want packages available immediately at app startup +- You're creating a new app + +See [`features/src/common-packages/README.md`](features/src/common-packages/README.md) for details. + +--- + +## Complete Examples + +### Example 1: Jupyter with Machine Learning Packages + +**With Claude:** +Just say: "Create a Jupyter app for machine learning" + +**Manual configuration:** +```json +{ + "name": "Jupyter - Machine Learning", + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "scikit-learn xgboost tensorflow torch pandas numpy matplotlib" + } + } +} +``` + +### Example 2: R Analysis for Genomics + +**With Claude:** +Just say: "I need R packages for genomics analysis" + +**Manual configuration:** +```json +{ + "name": "R Analysis - Genomics", + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "rPackages": "Seurat,DESeq2,GenomicRanges,tidyverse,ggplot2" + } + } +} +``` + +### Example 3: On-Demand Installation (Already in Jupyter) + +If you're already working and realize you need more packages: + +**Ask Claude:** "I need plotly for visualization" + +**Claude will show you:** +```python +!pip install plotly +``` + +Or for R: +```python +%%R +install.packages("plotly") +``` + +--- + +## Comparison + +| Approach | When Available | Requires Rebuild | Natural Language | +|----------|---------------|------------------|------------------| +| **Ask Claude** | On-demand or pre-build | No (on-demand) / Yes (pre-build) | ✅ Yes | +| **common-packages feature** | At app startup | Yes | ❌ No | + +**Best practice:** +- Use `common-packages` for packages you know you'll need +- Ask Claude when you discover new needs while working + +--- + +## Common Package Lists + +### Python + +**Data Science:** pandas, numpy, scipy, matplotlib, seaborn +**Machine Learning:** scikit-learn, xgboost, lightgbm +**Deep Learning:** tensorflow, torch, transformers, keras +**NLP:** transformers, spacy, nltk, gensim +**Bioinformatics:** biopython, scanpy, anndata +**BigQuery:** google-cloud-bigquery, google-cloud-storage, db-dtypes +**Visualization:** matplotlib, seaborn, plotly, dash, streamlit + +### R + +**Core Data Science:** tidyverse, dplyr, tidyr, readr, ggplot2 +**Visualization:** ggplot2, plotly, shiny, shinydashboard +**Machine Learning:** caret, randomForest, xgboost +**Bioinformatics:** Seurat, DESeq2, edgeR, limma +**Genomics:** GenomicRanges, AnnotationDbi, biomaRt +**BigQuery:** bigrquery, googleCloudStorageR + +--- + +## FAQ + +### Q: How do I install packages I discover I need while working? + +**A:** Just ask Claude! Say "I need " or "I need tools for ", and Claude will generate the installation command for you. + +### Q: Can I use both Python and R packages? + +**A:** Yes! Just specify both: +```json +{ + "pythonPackages": "pandas numpy", + "rPackages": "ggplot2,tidyverse" +} +``` + +### Q: What if I need a package that's not in the common lists? + +**A:** Just ask Claude or add it explicitly to the `pythonPackages` or `rPackages` field. For example: +- "I need the 'polars' package" +- Or add: `"pythonPackages": "polars duckdb"` + +### Q: How do I install Bioconductor packages? + +**A:** Ask Claude "I need Bioconductor packages for RNA-seq", or use: +```json +{ + "rPackages": "BiocManager,DESeq2,edgeR,limma" +} +``` + +The `common-packages` feature automatically handles Bioconductor installation. + +--- + +## Summary + +**Easiest way:** Ask Claude in natural language +**For known needs:** Use `common-packages` feature in `.devcontainer.json` +**Both work together:** Pre-install common packages, ask Claude for additional ones + +No more manual `pip install` or `install.packages()` every time! 🎉 diff --git a/features/src/common-packages/README.md b/features/src/common-packages/README.md new file mode 100644 index 00000000..17aeb4b4 --- /dev/null +++ b/features/src/common-packages/README.md @@ -0,0 +1,77 @@ +# Pre-install Packages Feature + +Pre-install your Python and R packages so you don't have to run `pip install` or `install.packages()` every time you create an app. + +## Usage + +Just list the packages you want in your `.devcontainer.json`: + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "pandas numpy scikit-learn matplotlib", + "rPackages": "tidyverse,ggplot2,dplyr,plotly,shiny" + } + } +} +``` + +That's it! Packages will be pre-installed when the app is built. + +## Examples + +### R Analysis with 15 packages + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "rPackages": "tidyverse,ggplot2,dplyr,tidyr,readr,plotly,shiny,DT,data.table,caret,randomForest,bigrquery,googleCloudStorageR,arrow,lubridate" + } + } +} +``` + +### Jupyter with Python packages + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "pandas numpy matplotlib seaborn scikit-learn google-cloud-bigquery google-cloud-storage" + } + } +} +``` + +### Both Python and R + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "pandas numpy", + "rPackages": "ggplot2,dplyr" + } + } +} +``` + +## Format + +- **Python packages:** Space-separated (e.g., `"pandas numpy scikit-learn"`) +- **R packages:** Comma-separated (e.g., `"tidyverse,ggplot2,dplyr"`) + +## How It Works + +- Packages install during container build (one-time) +- Apps launch instantly with packages ready +- Users can still install more packages at runtime if needed +- Much simpler than creating custom app configs + +## Performance + +- **First build:** Takes time to install packages +- **Every app after:** Instant - packages already there +- **vs. manual install every time:** Saves 5-10 minutes per app launch diff --git a/features/src/common-packages/devcontainer-feature.json b/features/src/common-packages/devcontainer-feature.json new file mode 100644 index 00000000..24a71fc9 --- /dev/null +++ b/features/src/common-packages/devcontainer-feature.json @@ -0,0 +1,19 @@ +{ + "id": "common-packages", + "version": "1.0.0", + "name": "Pre-install Packages", + "description": "Pre-install Python and R packages so users don't have to run pip install or install.packages() every time", + "documentationURL": "https://github.com/verily-src/workbench-app-devcontainers/tree/main/features/src/common-packages", + "options": { + "pythonPackages": { + "type": "string", + "default": "", + "description": "Python packages to install (space-separated, e.g., 'pandas numpy scikit-learn')" + }, + "rPackages": { + "type": "string", + "default": "", + "description": "R packages to install (comma-separated, e.g., 'tidyverse,ggplot2,dplyr')" + } + } +} diff --git a/features/src/common-packages/install.sh b/features/src/common-packages/install.sh new file mode 100755 index 00000000..99f51c7a --- /dev/null +++ b/features/src/common-packages/install.sh @@ -0,0 +1,18 @@ +#!/bin/bash +set -e + +echo "Installing user-specified packages..." + +# Install Python packages +if [ -n "${PYTHONPACKAGES}" ] && command -v pip &> /dev/null; then + echo "Installing Python packages: ${PYTHONPACKAGES}" + pip install --no-cache-dir ${PYTHONPACKAGES} +fi + +# Install R packages +if [ -n "${RPACKAGES}" ] && command -v R &> /dev/null; then + echo "Installing R packages: ${RPACKAGES}" + R --quiet -e "install.packages(strsplit('${RPACKAGES}', ',')[[1]], repos='https://cran.rstudio.com/', quiet=TRUE)" +fi + +echo "Package installation complete!" diff --git a/features/src/llm-context/skills/CREATE_CUSTOM_APP_WITH_PACKAGES.md b/features/src/llm-context/skills/CREATE_CUSTOM_APP_WITH_PACKAGES.md new file mode 100644 index 00000000..dcf44788 --- /dev/null +++ b/features/src/llm-context/skills/CREATE_CUSTOM_APP_WITH_PACKAGES.md @@ -0,0 +1,135 @@ +# Create Custom App with Pre-installed Packages + +**When to use:** User wants any app type (R Analysis, Jupyter, VSCode, etc.) with specific packages pre-installed. + +**Examples:** +- "I want R with tidyverse and ggplot2" +- "I want Jupyter with pandas, numpy, and scikit-learn" +- "I want VSCode with tensorflow and torch" + +**Goal:** Generate a complete devcontainer directory that pre-installs the requested packages. + +--- + +## Package Format + +**Python packages:** Space-separated +```json +"pythonPackages": "pandas numpy scikit-learn" +``` + +**R packages:** Comma-separated (NO SPACES) +```json +"rPackages": "tidyverse,ggplot2,dplyr" +``` + +**Both:** +```json +{ + "pythonPackages": "pandas numpy", + "rPackages": "ggplot2,dplyr" +} +``` + +--- + +## Key Points + +- **Works for ANY app type**: R Analysis, Jupyter, VSCode, RStudio, etc. +- **Any packages**: Users specify their own list - not limited to presets +- **During build**: Packages install once, available instantly after +- **User's repo**: Output can go anywhere, not just workbench-app-devcontainers + +--- + +## How to Generate + +Use the `common-packages` feature in `.devcontainer.json`: + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "USER_PACKAGES_HERE", + "rPackages": "USER_PACKAGES_HERE" + } + } +} +``` + +--- + +## Example: Jupyter with Python Packages + +**User:** "I want Jupyter with pandas, numpy, and scikit-learn" + +**Generate:** + +`.devcontainer.json`: +```json +{ + "name": "Jupyter - Custom Packages", + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "pandas numpy scikit-learn" + } + } +} +``` + +--- + +## Example: R Analysis with R Packages + +**User:** "I want R with tidyverse and ggplot2" + +**Generate:** + +`.devcontainer.json`: +```json +{ + "name": "R Analysis - Custom Packages", + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "rPackages": "tidyverse,ggplot2" + } + } +} +``` + +--- + +## Example: VSCode with Both + +**User:** "I want VSCode with Python and R packages" + +**Generate:** + +`.devcontainer.json`: +```json +{ + "name": "VSCode - Custom Packages", + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "pandas numpy", + "rPackages": "ggplot2,dplyr" + } + } +} +``` + +--- + +## Common Packages Reference + +**Python:** +- Data: pandas, numpy, scipy +- ML: scikit-learn, tensorflow, torch, transformers, xgboost +- Viz: matplotlib, seaborn, plotly +- Cloud: google-cloud-bigquery, google-cloud-storage + +**R:** +- Core: tidyverse, ggplot2, dplyr, tidyr, readr +- Viz: plotly, shiny, shinydashboard +- ML: caret, randomForest, xgboost +- Cloud: bigrquery, googleCloudStorageR diff --git a/features/src/llm-context/skills/INSTALL_PACKAGES.md b/features/src/llm-context/skills/INSTALL_PACKAGES.md new file mode 100644 index 00000000..c1fbd1fe --- /dev/null +++ b/features/src/llm-context/skills/INSTALL_PACKAGES.md @@ -0,0 +1,411 @@ +# Install Packages (Natural Language) + +**When to use:** User requests packages in natural language within a Jupyter, RStudio, or VSCode environment. + +**Examples:** +- "I need ggplot2" +- "Install machine learning packages" +- "I want tools for genomics analysis" +- "Can you set up deep learning packages?" +- "Set me up for single-cell RNA-seq analysis" + +--- + +## What This Skill Does + +When a user requests packages, this skill: + +1. **Parses** the natural language to identify packages +2. **Maps** domains (e.g., "machine learning") to package lists +3. **Generates** the appropriate installation command +4. **Provides** usage examples + +--- + +## Two Approaches + +### Approach 1: Pre-install Packages (Best for Known Needs) + +Use the `common-packages` feature in `.devcontainer.json`: + +```json +{ + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "tensorflow scikit-learn pandas numpy matplotlib", + "rPackages": "Seurat,DESeq2,ggplot2,tidyverse" + } + } +} +``` + +**When to use:** User knows packages upfront when creating the app. + +### Approach 2: On-Demand Installation (Best for Exploratory Work) + +Generate installation commands that users run in their environment: + +**Python (in Jupyter):** +```python +!pip install scikit-learn xgboost pandas numpy matplotlib +``` + +**R (in Jupyter with R kernel or RStudio):** +```python +%%R +install.packages(c("ggplot2", "tidyverse"), repos="https://cran.rstudio.com/") +``` + +**When to use:** User realizes they need packages while working. + +--- + +## Package Domain Mappings + +### Python Domains + +``` +machine learning → scikit-learn xgboost lightgbm pandas numpy matplotlib +deep learning → tensorflow torch transformers keras pandas numpy matplotlib +nlp → transformers spacy nltk gensim pandas numpy +visualization → matplotlib seaborn plotly pandas numpy +bioinformatics → biopython scanpy anndata pandas numpy matplotlib +genomics → biopython pysam scanpy pandas numpy +single-cell → scanpy anndata leidenalg pandas numpy matplotlib +bigquery → google-cloud-bigquery google-cloud-storage db-dtypes pandas numpy +statistics → scipy statsmodels pingouin pandas numpy +time series → prophet statsmodels pmdarima pandas numpy matplotlib +geospatial → geopandas shapely folium pandas numpy +computer vision → opencv-python pillow scikit-image numpy +web scraping → beautifulsoup4 requests pandas +``` + +### R Domains + +``` +data science → tidyverse,dplyr,tidyr,readr,ggplot2 +visualization → ggplot2,plotly,shiny,shinydashboard +machine learning → caret,randomForest,xgboost,mlr3 +bioinformatics → Seurat,DESeq2,edgeR,limma +genomics → GenomicRanges,AnnotationDbi,biomaRt,Seurat +single-cell → Seurat,SingleCellExperiment,scater +statistics → lme4,nlme,survival,MASS +time series → forecast,zoo,tseries +bigquery → bigrquery,googleCloudStorageR +``` + +--- + +## How to Use This Skill + +### Step 1: Identify Language + +**Python indicators:** +- Mentions: "python", "pip", "jupyter", "numpy", "pandas", "tensorflow" +- Context: In `.ipynb` file or Jupyter environment +- Default for Jupyter apps + +**R indicators:** +- Mentions: "R", "ggplot", "tidyverse", "Seurat", "rstudio" +- Explicitly says "R packages" + +### Step 2: Map Natural Language to Packages + +**Example 1:** "I need ggplot2" +- Language: R +- Packages: `tidyverse,ggplot2` + +**Example 2:** "I need machine learning packages" +- Language: Python (default) +- Domain: machine learning +- Packages: `scikit-learn xgboost lightgbm pandas numpy matplotlib` + +**Example 3:** "Install tools for single-cell analysis" +- Ask: "Python (scanpy) or R (Seurat)?" +- Python: `scanpy anndata leidenalg pandas numpy matplotlib` +- R: `Seurat,SingleCellExperiment,scater,tidyverse` + +### Step 3: Generate Installation Command + +**For Python (in Jupyter):** +```python +!pip install scikit-learn xgboost lightgbm pandas numpy matplotlib +``` + +**For R (in Jupyter with %%R magic):** +```python +%%R +install.packages(c("ggplot2", "tidyverse"), repos="https://cran.rstudio.com/") +``` + +**For R (in RStudio):** +```r +install.packages(c("ggplot2", "tidyverse"), repos="https://cran.rstudio.com/") +``` + +**For R Bioconductor packages:** +```python +%%R +if (!requireNamespace("BiocManager", quietly = TRUE)) + install.packages("BiocManager") +BiocManager::install(c("DESeq2", "edgeR", "limma")) +``` + +### Step 4: Provide Usage Examples + +After showing the installation command, provide import/usage examples: + +**Python:** +```python +import pandas as pd +import numpy as np +from sklearn import ... +``` + +**R:** +```r +library(ggplot2) +library(tidyverse) +``` + +--- + +## Complete Examples + +### Example 1: User says "I need ggplot2" + +**Response:** + +```markdown +I'll help you install ggplot2. Run this in a notebook cell: + +\`\`\`python +%%R +install.packages(c("ggplot2", "tidyverse"), repos="https://cran.rstudio.com/") +\`\`\` + +After installation, use it in R cells: +\`\`\`r +library(ggplot2) +ggplot(data, aes(x=x, y=y)) + geom_point() +\`\`\` +``` + +### Example 2: User says "Set me up for deep learning" + +**Response:** + +```markdown +I'll install deep learning packages. Run this: + +\`\`\`python +!pip install tensorflow torch transformers keras pandas numpy matplotlib +\`\`\` + +This installs: +- TensorFlow and PyTorch (frameworks) +- Transformers (NLP models) +- pandas, numpy (data handling) +- matplotlib (visualization) + +Import them: +\`\`\`python +import tensorflow as tf +import torch +import pandas as pd +import numpy as np +\`\`\` +``` + +### Example 3: User says "I need packages for genomics analysis" + +**Response:** + +```markdown +For genomics analysis, which would you prefer? + +**Python (scanpy)** - best for single-cell RNA-seq: +\`\`\`python +!pip install scanpy anndata biopython pysam pandas numpy matplotlib +\`\`\` + +**R (Seurat/Bioconductor)** - best for bulk RNA-seq and single-cell: +\`\`\`python +%%R +if (!requireNamespace("BiocManager", quietly = TRUE)) + install.packages("BiocManager") +BiocManager::install(c("Seurat", "DESeq2", "GenomicRanges", "AnnotationDbi")) +\`\`\` + +Let me know which you prefer, or I can set up both! +``` + +### Example 4: User says "I want BigQuery packages" + +**Response:** + +```markdown +I'll install Google Cloud BigQuery packages: + +\`\`\`python +!pip install google-cloud-bigquery google-cloud-storage db-dtypes pandas numpy +\`\`\` + +Example usage: +\`\`\`python +from google.cloud import bigquery +import pandas as pd + +client = bigquery.Client() +query = "SELECT * FROM \`project.dataset.table\` LIMIT 100" +df = client.query(query).to_dataframe() +\`\`\` +``` + +--- + +## Special Cases + +### Bioconductor Packages (R) + +Packages like DESeq2, edgeR, limma require BiocManager: + +```python +%%R +if (!requireNamespace("BiocManager", quietly = TRUE)) + install.packages("BiocManager") +BiocManager::install(c("DESeq2", "edgeR", "limma")) +``` + +### GPU/CUDA Packages + +If user mentions GPU: + +```python +# PyTorch with CUDA 11.8 +!pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 + +# TensorFlow GPU +!pip install tensorflow-gpu +``` + +### Package Versions + +If user specifies versions: + +```python +!pip install tensorflow==2.13.0 pandas==1.5.0 +``` + +### Package Aliases + +Handle common aliases: +- "sklearn" → "scikit-learn" +- "cv2" → "opencv-python" +- "tf" → "tensorflow" +- "bs4" → "beautifulsoup4" + +--- + +## Creating Apps with Pre-installed Packages + +When user wants to **create a new app** with packages, generate a `.devcontainer.json`: + +**Example:** "Create a Jupyter app with machine learning packages" + +```json +{ + "name": "Jupyter - Machine Learning", + "dockerComposeFile": "docker-compose.yaml", + "service": "app", + "features": { + "ghcr.io/verily-src/workbench-app-devcontainers/common-packages": { + "pythonPackages": "scikit-learn xgboost lightgbm tensorflow torch pandas numpy matplotlib seaborn" + } + }, + "customizations": { + "vscode": { + "extensions": [ + "ms-python.python", + "ms-toolsai.jupyter" + ] + } + } +} +``` + +--- + +## Error Handling + +### Unknown Package + +```markdown +I couldn't find a package called "xyzabc". Did you mean: +- xgboost (machine learning) +- ... + +Or describe what you're trying to do (e.g., "analyze genomics data") +``` + +### Ambiguous Request + +```markdown +I can help install packages! What type of analysis? + +- Machine learning / AI +- Data visualization +- Bioinformatics / genomics +- Statistics +- (or tell me specific packages) +``` + +### Installation Failure + +```markdown +Installation failed. Try: +1. Install packages one at a time +2. Check write permissions +3. Try: \`!pip install --user \` +``` + +--- + +## Decision Tree + +``` +User requests packages +│ +├─ Creating new app? +│ └─ Generate .devcontainer.json with common-packages feature +│ +├─ Working in existing environment? +│ │ +│ ├─ Explicit packages? (e.g., "ggplot2", "tensorflow") +│ │ └─ Generate pip/R install command +│ │ +│ └─ Domain mentioned? (e.g., "machine learning") +│ ├─ Map to package list +│ ├─ Identify language (Python/R) +│ └─ Generate install command +│ +└─ Too vague? + └─ Ask clarifying question +``` + +--- + +## Summary + +This skill enables **natural language package installation** by: + +1. **Parsing** user intent (domain/task identification) +2. **Mapping** to curated package lists +3. **Generating** installation commands +4. **Providing** usage examples + +**Key principle:** Make it easy to request packages without knowing exact names, while supporting explicit package specifications. + +**Two modes:** +- **Pre-install** (common-packages feature) - for app creation +- **On-demand** (pip/R commands) - for exploratory work