Split manifest storage by version to prevent 20MB+ single files by Copilot · Pull Request #696 · SATVILab/projr

Copilot · 2025-11-17T21:34:10Z

Summary: Split Manifest Files by Version

This PR solves the issue where manifest.csv can grow to 20MB+ with many builds by implementing a split manifest system.

Current Status

Core split manifest implementation
Migration bug fix (P1)
Merge with main (PR Fix R CMD CHECK errors and permanently disable OSF tests #921, Fix undefined variable in test cleanup section #922)
Merge with latest main (143fe14 - PR Add AGENTS.md for Google Jules / AI agent configuration #975)
All manifest tests passing (153 tests)
All manifest-query tests passing (66 tests)

The Solution

Split Manifest System:

Each version gets its own file: _projr/manifest/v{version}.csv
Files stored in _projr/manifest/ directory
Consolidated manifest.csv maintained for backward compatibility
Reading merges both sources during migration period to preserve history

Known Design Trade-offs (identified in code review)

Indefinite migration: Both formats maintained simultaneously until user cleans up
Build logic ordering assumption: manifest_parts filtering assumes pre/post/previous ordering
No atomic writes: Race condition possible in concurrent builds (uncommon use case)
Silent error recovery: Corrupted split files silently return empty manifest

Original prompt

This section details on the original issue you should resolve

<issue_title>Make manifest.csv more efficient</issue_title>
<issue_description>- Could consider another data format (so that it's more compressed)

Could maybe zip it

Could just store for each new hashed item the versions it corresponds to.

But this requires some more work, so not super interested in it at this stage.

I could at least split the manifest from different versions into different files, so that we don't have to check massive files into git every time.

Actually, this isn't true - at the storage level, Git does do deltas (see here and here). </issue_description>

Comments on the Issue (you are @copilot in this section)
@MiguelRodo The above matters for OSF, but does it matter for Zenodo? In Zenodo, we basically always have every version up there (because there's so much space). It's also kind of our only option, as when Zenodo is private you're gonna want to just download all the files from a given version as you cannot access the API directly. Zenodo also doesn't allow deletions, at least not from the same version, so you're kind of stuck there. But that's a lot easier to deal with, I imagine.
I am thinking that we basically copy OSF's archiving option. So, we basically put something in the archive as soon as we've uploaded it.</comment_new>
<comment_new>@MiguelRodo
This is quite important, as the manifest gets checked into Git at the moment, and it can quickly get massive - 20MB for the one project after I think just one build.

A quick discussion with ChatGPT said that Git and DVC hash files in a directory, store their information in a file, hash that file, and do that recursively up to the root (including the sub-directory hashes as needed). Then that root directory hash is the only thing checked into Git, which is obviously small.

Clearly there is a lot of deduplication there in that sub-directory information is shared. But there are a few issues to think about:

The actual directory information is not stored in Git. What happens if the remote information gets lost, the entire project information gets lost?

The directory contents need to be transformed to be human-readable.

The directory contents need a correctly-functioning program to reconstitute the manifest.

The manifest information is stored in a large number of files. Reconstituting the manifest may require, if these parts are downloaded individually, a large number of downloads and uploads. This may be slow. We could consider zipping this directory instead. We still run up against the issue of large overall sizes, but at least the deduplication will help.</comment_new>

Fixes Make manifest.csv more efficient #406

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: MiguelRodo <23501332+MiguelRodo@users.noreply.github.com>

MiguelRodo · 2025-11-18T10:02:23Z

@codex review

chatgpt-codex-connector · 2025-11-18T10:02:28Z

To use Codex here, create a Codex account and connect to github.

MiguelRodo · 2025-11-18T10:05:28Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-11-18T10:09:11Z

+    # Read all split manifest files
+    manifest_split <- .manifest_split_read_all()
+    if (nrow(manifest_split) > 0) {


Merge split manifests with consolidated history

When _projr/manifest/ exists, .manifest_read_project() returns the split manifests immediately (lines 152-154) and only reads manifest.csv if no split rows exist. On a project that already has historical versions only in the consolidated manifest, the first post-upgrade build will create a split file for the new version and all older versions disappear from manifest queries because the consolidated data is never merged once the split directory is present.

Useful? React with 👍 / 👎.

@copilot fix this

MiguelRodo · 2025-11-18T10:21:43Z

@copilot P1 Badge Merge split manifests with consolidated history

When _projr/manifest/ exists, .manifest_read_project() returns the split manifests immediately (lines 152-154) and only reads manifest.csv if no split rows exist. On a project that already has historical versions only in the consolidated manifest, the first post-upgrade build will create a split file for the new version and all older versions disappear from manifest queries because the consolidated data is never merged once the split directory is present.

Useful? React with 👍 / 👎.

Previously, .manifest_read_project() would return only split manifests once the split directory existed, losing all historical versions from manifest.csv during the first post-upgrade build. Now merges both sources to preserve historical data during migration period. Fixes comment 3546728155 Co-authored-by: MiguelRodo <23501332+MiguelRodo@users.noreply.github.com>

Copilot · 2025-11-18T10:29:00Z

@copilot P1 Badge Merge split manifests with consolidated history

When _projr/manifest/ exists, .manifest_read_project() returns the split manifests immediately (lines 152-154) and only reads manif...

Fixed in commit 6a7d98b. The function now merges split manifests with consolidated manifest.csv to preserve all historical versions during migration. Old projects upgrading will see their historical data retained as new split files are created with each build.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/repos//testProjrInitPrompt
- Triggering command: /opt/R/4.5.2/lib/R/bin/exec/R -e devtools::load_all();~+~.test_set_lite();~+~result~+~<-~+~devtools::test();~+~cat('\n===~+~SUMMARY~+~===\n');~+~cat('Failed:',~+~sum(result[['failed']]),~+~'\n');~+~cat('Passed:',~+~sum(result[['passed']]),~+~'\n') (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

MiguelRodo

@copilot merge in latest changes from main

…fest-csv # Conflicts: # R/build-manifest.R # R/manifest.R

Copilot · 2025-12-11T22:49:20Z

@copilot merge in latest changes from main

Merged latest changes from main in commit 53410b7. Resolved conflicts in R/manifest.R and R/build-manifest.R by:

Integrating performance improvements (list-based manifest collection)
Adding .version_file_check_label_trusted() function from main
Preserving all split manifest functionality
All 224 manifest-related tests pass.

MiguelRodo

@copilot merge in still more recent changes from main branch and check that ALL (not just LITE or CRAN or manifest specific) tests pass

…fest-csv

Copilot · 2025-12-11T23:06:53Z

@copilot merge in still more recent changes from main branch and check that ALL (not just LITE or CRAN or manifest specific) tests pass

Merged latest changes from main in commit a3bb078.

Full test suite results (ALL tests, not LITE/CRAN):

✅ 5574 tests PASS
❌ 3 tests FAIL (pre-existing, unrelated to manifest changes)
- test-build-copy.R:424 (label validation)
- test-remote-github.R:109, :392 (project path issues)
⏭️ 75 tests SKIP (expected)
⚠️ 0 WARN

All manifest functionality verified:

158/158 manifest tests pass
66/66 manifest-query tests pass

The 3 failures are pre-existing issues in other parts of the codebase, not introduced by this PR.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/repos//testProjrInitPrompt
- Triggering command: /opt/R/4.5.2/lib/R/bin/exec/R /opt/R/4.5.2/lib/R/bin/exec/R -e devtools::load_all();~+~devtools::test() (http block)
https://api.github.com/repos//testProjrInitPromptPublic
- Triggering command: /opt/R/4.5.2/lib/R/bin/exec/R /opt/R/4.5.2/lib/R/bin/exec/R -e devtools::load_all();~+~devtools::test() (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

MiguelRodo

@copilot have another think through this, does it work well, when wil it break?

MiguelRodo · 2026-02-22T08:57:16Z

@copilot merge in latest changes from main

MiguelRodo

Carry on

…fest-csv # Conflicts: # NAMESPACE

Copilot · 2026-02-22T09:04:53Z

@copilot merge in latest changes from main

Merged latest main (143fe14) in commit ba69802. Resolved one conflict in NAMESPACE (removed stale projr_osf_create_project export that no longer exists in main, kept projr_manifest_versions). All 153 manifest tests and 66 manifest-query tests pass.

Initial plan

a9995ea

Copilot AI assigned Copilot and MiguelRodo Nov 17, 2025

Copilot started work on behalf of MiguelRodo November 17, 2025 21:34 View session

Copilot AI and others added 3 commits November 17, 2025 21:48

Implement split manifest system by version

534f21f

Co-authored-by: MiguelRodo <23501332+MiguelRodo@users.noreply.github.com>

Add helper functions for split manifest inspection

11b3cee

Co-authored-by: MiguelRodo <23501332+MiguelRodo@users.noreply.github.com>

Update build system documentation for split manifests

0b9987a

Co-authored-by: MiguelRodo <23501332+MiguelRodo@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Make manifest.csv more efficient~~ Split manifest storage by version to prevent 20MB+ single files Nov 17, 2025

Copilot AI requested a review from MiguelRodo November 17, 2025 22:06

Copilot finished work on behalf of MiguelRodo November 17, 2025 22:06

chatgpt-codex-connector Bot reviewed Nov 18, 2025

View reviewed changes

Copilot started work on behalf of MiguelRodo November 18, 2025 10:12 View session

Copilot finished work on behalf of MiguelRodo November 18, 2025 10:16

Copilot started work on behalf of MiguelRodo November 18, 2025 10:23 View session

Copilot finished work on behalf of MiguelRodo November 18, 2025 10:34

MiguelRodo requested changes Dec 11, 2025

View reviewed changes

Copilot started work on behalf of MiguelRodo December 11, 2025 22:40 View session

Merge remote-tracking branch 'origin/main' into copilot/optimize-mani…

53410b7

…fest-csv # Conflicts: # R/build-manifest.R # R/manifest.R

Copilot finished work on behalf of MiguelRodo December 11, 2025 22:51

Copilot AI requested a review from MiguelRodo December 11, 2025 22:51

MiguelRodo requested changes Dec 11, 2025

View reviewed changes

Copilot started work on behalf of MiguelRodo December 11, 2025 22:53 View session

Merge remote-tracking branch 'origin/main' into copilot/optimize-mani…

a3bb078

…fest-csv

Copilot AI requested a review from MiguelRodo December 11, 2025 23:08

Copilot finished work on behalf of MiguelRodo December 11, 2025 23:08

MiguelRodo requested changes Feb 18, 2026

View reviewed changes

Copilot started work on behalf of MiguelRodo February 18, 2026 18:43 View session

Copilot AI requested a review from MiguelRodo February 18, 2026 18:56

Copilot finished work on behalf of MiguelRodo February 18, 2026 18:56

Copilot started work on behalf of MiguelRodo February 22, 2026 08:57 View session

MiguelRodo reviewed Feb 22, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into copilot/optimize-mani…

ba69802

…fest-csv # Conflicts: # NAMESPACE

Copilot AI requested a review from MiguelRodo February 22, 2026 09:05

Copilot finished work on behalf of MiguelRodo February 22, 2026 09:05

Conversation

Copilot AI commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary: Split Manifest Files by Version

Current Status

The Solution

Known Design Trade-offs (identified in code review)

Comments on the Issue (you are @copilot in this section)

Uh oh!

MiguelRodo commented Nov 18, 2025

Uh oh!

chatgpt-codex-connector Bot commented Nov 18, 2025

Uh oh!

MiguelRodo commented Nov 18, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

MiguelRodo Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

MiguelRodo commented Nov 18, 2025

Uh oh!

Copilot AI commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

MiguelRodo left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Dec 11, 2025

Uh oh!

MiguelRodo left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

MiguelRodo left a comment

Choose a reason for hiding this comment

Uh oh!

MiguelRodo commented Feb 22, 2026

Uh oh!

MiguelRodo left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Feb 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Nov 17, 2025 •

edited

Loading

Copilot AI commented Nov 18, 2025 •

edited

Loading

Copilot AI commented Dec 11, 2025 •

edited

Loading