A local issue for tracking development issues in the main queue that are crucial for the operations of HDV

[landreev]

I'm going to maintain a (check)list of the development efforts that are deemed super important for speeding things up/improving overall stability of our prod. instance.

• Speeding up exports of unexportable monster datasets: Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality dataverse#11799 [in progress; PR in draft]
• Jim's improvements addressing unacceptable performance in reindexing after making changes to role assignments in large collections (an exotic/edge case; but the performance degradation is really bad): QDR: Multiple transactions for permission reindexing dataverse#11822 [PR merged, will be in 6.9; already deployed in prod. as a patch]
• A performance issue with re-harvesting of harvested datasets; is a major blocker in refreshing some large harvested collections that are now stale: Unacceptable performance deleting datasets and/or datasetversions in the prod. database - needs to be improved.  dataverse#11828
• Add a dedicated File page rate-limiting command: A dedicated rate-limiting command for the File page dataverse#11366 [a trivial PR I'm already including in my prod. deployment patch; but needs to be merged into the main source tree at some point]
• Expanding Steven's internal rate limiting subsystem to make it more fine-tunable [discussing this w/ Steven; haven't opened an issue yet]
• Adding Payara-level caching for rendered pages [discussing w/ Steven; no issue yet]
• Harvesting bug: Harvesting: a bug in dataset update (UpdateHarvestedDatasetCommand) dataverse#11933
• Another Harvesting bug: Harvesting: DDI import fails when there's a xml:lang attribute present in the codeBook element dataverse#11932
• Local Globus patch; makes the workflow easier for the users: Globus patch - no need to lock the dataset while an upload is in progress, when tasks are monitored asynchronously. dataverse#11971
• Local Storage quotas patch PR: largely cosmetic, but improves the use experience - Storage quotas patch: Minor/cosmetic improvements dataverse#11988
• Storage quotas [needs to be prioritized!]: extending quotas to be configurable for specific users: Extend Storage Quotas to individual datasets and user accounts dataverse#11529
  • A low-hanging fruit/easily achievable part of the issue above isolated as a sub-issue, a quick incremental improvement for immediate HDV deployment: Add storage quotas on individual datasets dataverse#11987
  • Another low-hanging/easily achievable sub-part of the above: Display current collection size on Add Files page dataverse#11528
• A bug in storage quotas: when deleting a draft the total storage size cached for the parent collection(s) not updated: Storage Quota for a Collection Not Updated when Deleting a Dataset dataverse#11543
• [somewhat urgent at this point?] the old, JSF-based dataset page needs to be re-factored and optimized; we simply can't afford to wait until it's retired. Will open an issue shortly. (will coordinate with Eryk since he expressed interest in/may already be working on a related effort in parallel)
• There's a pressing need to have a supported API for destroying/permanently delete published files: Add admin-only destroy datafile command and the api that could be used on *published* files dataverse#12253
• Croissant-lite PR (is part of 6.10!) - will spare us the expense of (unnecessarily) embedding huge croissant records into xhtml for variable-rich datasets: Display current collection size on Add Files page dataverse#11528 (thanks to Phil!)

... work in progress - there is definitely more!

[landreev]

Updated the master list above to reflect the fact that IQSS/dataverse#11987 was completed (in Dec.)