Major development to include MassIVE; iPROX and PRoteomeXchange

.codacy/codacy.yaml

@@ -0,0 +1,15 @@
+runtimes:
+    - dart@3.7.2
+    - go@1.22.3
+    - java@17.0.10
+    - node@22.2.0
+    - python@3.11.11
+tools:
+    - dartanalyzer@3.7.2
+    - eslint@8.57.0
+    - lizard@1.17.31
+    - pmd@7.11.0
+    - pylint@3.3.6
+    - revive@1.7.0
+    - semgrep@1.78.0
+    - trivy@0.66.0

README.md

@@ -8,7 +8,7 @@
 
 You can:
 - download public and private PRIDE files
-- download public MassIVE datasets directly from `MSV...` accessions
+- download public MassIVE (`MSV...`), JPOST (`JPST...`), and iProX (`IPX...`) datasets directly. MassIVE goes through FTPS at `massive-ftp.ucsd.edu`, with an automatic HTTPS fallback (via the GNPS2 file index and the `massive.ucsd.edu` ProteoSAFe endpoint) for networks that block FTP/FTPS; JPOST uses the JSON PROXI endpoint at `repository.jpostdb.org` for listings and `ftp.jpostdb.org` for transfers; iProX fetches the dataset's ProteomeXchange XML from `download.iprox.org` and downloads files over anonymous HTTP
 - download by category (`RAW`, `SEARCH`, `RESULT`, etc.)
 - stream project and file metadata
 - search projects by keyword and filters
@@ -45,231 +45,82 @@ pip install --upgrade pridepy
 pridepy --help
 ```
 
-### Option 3: Install from source (development)
+### Option 3: Install the latest code directly from GitHub
 
-```bash
-git clone https://github.com/PRIDE-Archive/pridepy
-cd pridepy
-uv sync --extra dev
-uv run pridepy --help
-```
-
-## Quick Start (New Users)
+To get features that have not been released to PyPI yet, install straight from a
+branch. `master` holds the latest stable code; `dev` holds the newest (and
+potentially unstable) development work.
 
-### 1) Download all raw files for a project (robust mode)
+With `uv`:
 
 ```bash
-pridepy download-all-public-raw-files \
-  -a PXD008644 \
-  -o ./downloads/PXD008644 \
-  --checksum-check
-```
-
-What this does:
-- default `ftp` starts with FTP and falls back (`ftp -> aspera -> s3 -> globus`)
-- `--checksum-check` downloads project checksums and validates files
-- empty/corrupt files are retried automatically
+# Latest stable (master)
+uv tool install "git+https://github.com/PRIDE-Archive/pridepy@master"
 
-### 2) Continue interrupted downloads safely
-
-```bash
-pridepy download-all-public-raw-files \
-  -a PXD008644 \
-  -o ./downloads/PXD008644 \
-  --skip-if-downloaded-already \
-  --checksum-check
+# Bleeding edge (dev)
+uv tool install "git+https://github.com/PRIDE-Archive/pridepy@dev"
 ```
 
-### 3) Download a public MassIVE dataset directly
+Or with `pip`:
 
 ```bash
-pridepy download-all-public-raw-files \
-  -a MSV000082297 \
-  -o ./downloads/MSV000082297
-```
-
-For direct `MSV...` downloads, `pridepy` enumerates the dataset from MassIVE's public FTP tree. Raw downloads follow MassIVE's own collection layout, so `download-all-public-raw-files` downloads the files stored under the dataset's `raw/` collection.
+# Latest stable (master)
+pip install --upgrade "git+https://github.com/PRIDE-Archive/pridepy@master"
 
-### 4) Download only selected categories
-
-```bash
-pridepy download-all-public-category-files \
-  -a PXD022105 \
-  -o ./downloads/PXD022105 \
-  -c RAW,SEARCH
+# Bleeding edge (dev)
+pip install --upgrade "git+https://github.com/PRIDE-Archive/pridepy@dev"
 ```
 
-You can also request a specific MassIVE collection through the same category interface:
-
-```bash
-pridepy download-all-public-category-files \
-  -a MSV000082297 \
-  -o ./downloads/MSV000082297-results \
-  -c RESULT
-```
-
-### 5) Download one file by name
-
-```bash
-pridepy download-file-by-name \
-  -a PXD022105 \
-  -f checksum.txt \
-  -o ./downloads/PXD022105 \
-  --checksum-check
-```
-
-### 6) Download raw files from ProteomeXchange
-
-```bash
-pridepy download-px-raw-files \
-  -a PXD039236 \
-  -o ./downloads/PXD039236
-```
-
-### 6) Download a named subset of files (manifest)
-
-```bash
-pridepy download-files-by-list \
-  -a PXD001819 \
-  -F files.txt \
-  -o ./downloads/PXD001819 \
-  --checksum-check
-```
-
-`files.txt` is one filename per line (blank lines and `#` comments are
-ignored). Internally each filename is resolved against the project metadata
-API and downloaded via the same batch + protocol-fallback engine as
-`download-all-public-raw-files`. Use `-f a.raw,b.raw,c.raw` instead of
-`-F` for a small inline list.
-
-Useful options:
-
-- `-p globus` — use the globus download strategy (HTTP Range + resume)
-- `-w 3` — download up to 3 files in parallel (globus only, max 3)
-- `--checksum-check` — validate files against PRIDE checksums after download
+You can pin to any branch, tag, or commit by changing the part after `@` (e.g.
+`@v0.0.16` or `@<commit-sha>`).
 
-### 7) Download files from raw URLs
+### Option 4: Install from source (development)
 
 ```bash
-pridepy download-files-by-url \
-  -F urls.txt \
-  -o ./downloads/urls
+git clone https://github.com/PRIDE-Archive/pridepy
+cd pridepy
+uv sync --extra dev
+uv run pridepy --help
 ```
 
-`urls.txt` is one fully-qualified URL per line. Schemes `http`, `https`, and
-`ftp` are dispatched to the matching downloader. Use `-u/--urls` for one or
-more comma-separated URLs, e.g. `--urls https://a.com/x.raw,ftp://b.com/y.raw`.
-Note: URLs containing literal commas are not supported with `--urls`; use a
-manifest file (`-F`) instead.
+## Usage
 
-Useful options:
-
-- `-p globus` — use globus download strategy for http/https URLs (resume-capable)
-- `-w 3` — download up to 3 files in parallel (globus only, max 3)
-- `--checksum-check` — validate against PRIDE checksums (accession inferred
-  from PRIDE URL paths; only PRIDE archive URLs are supported)
-
-## CLI Command Overview
+See the **[usage guide](docs/usage.md)** for detailed instructions and examples:
+downloading data (PRIDE, MassIVE, JPOST, iProX, ProteomeXchange), category and
+manifest downloads, private files, streaming metadata, searching projects, and
+the Python API.
 
 ```bash
 pridepy --help
 ```
 
-Main commands:
-- `download-all-public-raw-files`
-- `download-all-public-category-files`
-- `download-file-by-name`
-- `download-files-by-list`
-- `download-files-by-url`
-- `download-px-raw-files`
-- `list-private-files`
-- `stream-files-metadata`
-- `stream-projects-metadata`
-- `search-projects-by-keywords-and-filters`
-
-## More CLI Examples
-
-### Search projects
-
-```bash
-pridepy search-projects-by-keywords-and-filters \
-  -k human \
-  -f projectTags==ProteomeTools,organismsPart==Pancreas \
-  -sd DESC \
-  -sf accession \
-  -sf submissionDate
-```
-
-### Stream all project metadata to JSON
-
-```bash
-pridepy stream-projects-metadata -o all_pride_projects.json
-```
-
-### Stream all file metadata for one accession
+| Command | Purpose |
+| --- | --- |
+| `download-all-public-raw-files` | Download every public RAW file of a dataset |
+| `download-all-public-category-files` | Download files of one or more categories (RAW, SEARCH, …) |
+| `download-file-by-name` | Download a single file (public or private) |
+| `download-files-by-list` | Download a named subset of files from a manifest/CSV |
+| `download-files-by-url` | Download files from raw `http`/`https`/`ftp` URLs |
+| `download-px-raw-files` | Download RAW files resolved from a ProteomeXchange accession |
+| `list-private-files` | List files of a private project (needs credentials) |
+| `stream-files-metadata` | Stream file metadata (one project or all) to JSON |
+| `stream-projects-metadata` | Stream all project metadata to JSON |
+| `search-projects-by-keywords-and-filters` | Search projects by keyword and filters |
 
-```bash
-pridepy stream-files-metadata -a PXD005011 -o PXD005011_files.json
-```
-
-### Download private files
-
-List files:
-
-```bash
-pridepy list-private-files -a PXD022105 -u YOUR_USER -p YOUR_PASSWORD
-```
-
-Download a private file:
+Quick examples:
 
 ```bash
-pridepy download-file-by-name \
-  -a PXD022105 \
-  -f checksum.txt \
-  -o ./downloads/private \
-  --username YOUR_USER \
-  --password YOUR_PASSWORD
-```
-
-## Python API Examples
-
-### Example: get raw files for a project
-
-```python
-from pridepy.files.files import Files
+# Download all public RAW files of a dataset (any repository)
+pridepy download-all-public-raw-files -a PXD008644 -o ./downloads/PXD008644 --checksum-check
 
-files = Files()
-raw_files = files.get_all_raw_file_list("PXD008644")
-print(f"RAW files: {len(raw_files)}")
-print(raw_files[0]["fileName"])
-```
-
-For MassIVE accessions, the same method returns the files found under the dataset's `raw/` collection:
+# Download a ProteomeXchange dataset by its PXD accession
+pridepy download-px-raw-files -a PXD039236 -o ./downloads/PXD039236
 
-```python
-from pridepy.files.files import Files
-
-files = Files()
-raw_files = files.get_all_raw_file_list("MSV000082297")
-print(f"MassIVE raw files: {len(raw_files)}")
+# Download a native MassIVE / JPOST / iProX dataset
+pridepy download-all-public-raw-files -a MSV000082297 -o ./downloads/MSV000082297
 ```
 
-### Example: search projects
-
-```python
-from pridepy.project.project import Project
-
-project = Project()
-results = project.search_by_keywords_and_filters(
-    keyword="PXD009476",
-    query_filter="",
-    page_size=25,
-    page=0,
-    sort_direction="DESC",
-    sort_fields="accession",
-)
-print(f"Hits: {len(results)}")
-```
+Full option tables and more examples are in [docs/usage.md](docs/usage.md).
 
 ## Development and Release (uv)