Skip to content

HPCC-35694 Expose single field for physical File Size#151

Closed
asselitx wants to merge 211 commits intoGordonSmith:HPCC-35694-DFUSORTBY_COMPRESSEDSIZEfrom
asselitx:hpcc-35694-dfustortby-compressedsize
Closed

HPCC-35694 Expose single field for physical File Size#151
asselitx wants to merge 211 commits intoGordonSmith:HPCC-35694-DFUSORTBY_COMPRESSEDSIZEfrom
asselitx:hpcc-35694-dfustortby-compressedsize

Conversation

@asselitx
Copy link

@asselitx asselitx commented Mar 18, 2026 •
edited
Loading

Regardless of compression status this field has the actual physical size on disk.

  • Add FileSize field to DFUQuery response to hold DFUSFSize
  • Update ECL Watch to use FileSize when present

Type of change:

  • This change is a bug fix (non-breaking change which fixes an issue).
  • This change is a new feature (non-breaking change which adds functionality).
  • This change improves the code (refactor or other change that does not change the functionality)
  • This change fixes warnings (the fix does not alter the functionality or the generated code)
  • This change is a breaking change (fix or feature that will cause existing behavior to change).
  • This change alters the query API (existing queries will have to be recompiled)

Checklist:

  • My code follows the code style of this project.
    • My code does not create any new warnings from compiler, build system, or lint.
  • The commit message is properly formatted and free of typos.
    • The commit message title makes sense in a changelog, by itself.
    • The commit is signed.
  • My change requires a change to the documentation.
    • I have updated the documentation accordingly, or...
    • I have created a JIRA ticket to update the documentation.
    • Any new interfaces or exported functions are appropriately commented.
  • I have read the CONTRIBUTORS document.
  • The change has been fully tested:
    • I have added tests to cover my changes.
    • All new and existing tests passed.
    • I have checked that this change does not introduce memory leaks.
    • I have used Valgrind or similar tools to check for potential issues.
  • I have given due consideration to all of the following potential concerns:
    • Scalability
    • Performance
    • Security
    • Thread-safety
    • Cloud-compatibility
    • Premature optimization
    • Existing deployed queries will not be broken
    • This change fixes the problem, not just the symptom
    • The target branch of this pull request is appropriate for such a change.
  • There are no similar instances of the same problem that should be addressed
    • I have addressed them here
    • I have raised JIRA issues to address them separately
  • This is a user interface / front-end modification
    • I have tested my changes in multiple modern browsers
    • The component(s) render as expected

Smoketest:

  • Send notifications about my Pull Request position in Smoketest queue.
  • Test my draft Pull Request.

Testing:

Panagiotatos and others added 30 commits February 5, 2026 12:14
HPCC-34819 Document SSL/TLS setup for Containerized ESPs
22b1918 
Signed-off-by: Panagiotatos <greg.panagiotatos+copilot@lexisnexisrisk.com>
@kenrowland
HPCC-35791 ESP not copying vaults section to component.xml
de2906e 
Added XSL statement to copy vaults section to esp.xml
Added code in esp to insert vaults properties into component config.

Signed-Off-By: Kenneth Rowland kenneth.rowland@lexisnexisrisk.com
@kenrowland
HPCC-35803 Vaults section not copied to Dali component xml
7f63f87 
Added copy of vaults section to Dali XSL

Signed-Off-By: Kenneth Rowland kenneth.rowland@lexisnexisrisk.com
@timothyklemm
HPCC-35760: Omit EventStackTrace from event dump output
aff3be8 
Signed-off-by: Tim Klemm <Tim.Klemm@lexisnexisrisk.com>
HPCC-35714 Update Dali feature documentation
1e0a55a 
Signed-off-by: Panagiotatos <greg.panagiotatos+copilot@lexisnexisrisk.com>
@jeclrsg
HPCC-35776 ECL Watch v9 WU Processes container name
1a2ca37 
display ContainerName, Graphs and Sequence on the WU Details page's
Processes tab

Signed-off-by: Jeremy Clements <79224539+jeclrsg@users.noreply.github.com>
@kunalaswani
HPCC-35720 Move Notifications from Page View to Dialog
464bd21 
Notifications moved to dialog to prevent page changes while working.

Signed-off-by: Kunal Aswani <Kunal.Aswani@lexisnexisrisk.com>
@jackdelv
HPCC-35656 Fix Parquet plugin reading empty file
0acc6bb 
- Throw a more descriptive error if a table with 0 rows is found
- Throw an error before writing a record with 0 rows or trying to bind a row with no fields
@GordonSmith
Split off 9.12.78
601b22a 
Signed-off-by: Gordon Smith <GordonJSmith@gmail.com>
@mckellyln
HPCC-35833 Roxie cache incorrect offset math requires casting
68ba0c5 
Signed-off-by: M Kelly <mark.kelly@lexisnexisrisk.com>
@GordonSmith
HPCC-35835 DFU Workunits sorting issues
15ce2a2 
Realign the sort ids with the ESP API
Persist sort and page info from address

Signed-off-by: Gordon Smith <GordonJSmith@gmail.com>
@ghalliday
HPCC-35840 Add timing tests for 8K index compression
244af57 
Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
@ghalliday
Merge pull request hpcc-systems#20958 from timothyklemm/hpcc-35760-om…
882c24f 
…it-event-stack-trace

HPCC-35760: Omit EventStackTrace from event dump output

Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#20983 from mckellyln/hpcc-35833
4442304 
HPCC-35833 Roxie cache incorrect offset math requires casting

Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
HPCC-35845 Add roxieMode and transportProtocol to the values schema
175b12e 
Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
@shamser
HPCC-35312 Add a warning if thor is blocked on a soapcall
a8dd01a 
Implements analysis rule to detect SOAPCALL activities running on a
large Thor clusters where distributed execution provides no benefit.
Slow SOAPCALL operations waste resources when run on clusters with
many nodes.

The warning identifies workloads where moving SOAPCALL operations to
single-node clusters would reduce execution time and infrastructure costs
while maintaining identical functionality.

Signed-off-by: Shamser Ahmed <shamser.ahmed@lexisnexisrisk.com>

Apply review comments: add safety checks and logging

Co-authored-by: shamser <11410590+shamser@users.noreply.github.com>
Signed-off-by: Shamser Ahmed <shamser.ahmed@lexisnexisrisk.com>
@shamser
HPCC-35407 Report sg duration in nanoseconds
df4e978 
Signed-off-by: Shamser Ahmed <shamser.ahmed@lexisnexisrisk.com>
@GordonSmith
HPCC-35826 Add support for min/max/avg in ScopesTable
0cb55b4 
Signed-off-by: Gordon Smith <GordonJSmith@gmail.com>
@dcamper
HPCC-35821 Race condition in unit test AsyncSocketConnectionTests::te…
23253b7 
…stAsyncReconnect

Signed-off-by: Dan S. Camper <dan.camper@lexisnexisrisk.com>
@timothyklemm
HPCC-35763: Omit disabled optional attributes from event dump output
686f3dd 
- Make the source file properties available to an operation outside of its
  traverseEvents method.
- Update the CSV dump visitor to copy the source file properties on
  construction.
- Update the CSV dump visitor to skip the remaining optional attributes
  based on the source file properties.

Signed-off-by: Tim Klemm <Tim.Klemm@lexisnexisrisk.com>
@jakesmith
HPCC-35849 The coalescer must have Dali access.
ba3f32c 
Fix getSashaServiceAccess, which should have enumerated that
the coalescer required 'dali' access.
Exposed by changes in HPCC-27051.

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@ghalliday
HPCC-35848 Add a IndexOpen event to the event recording
5e19f69 
Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
@ghalliday
Merge pull request hpcc-systems#20991 from jakesmith/HPCC-35849-coale…
c272716 
…scer-dali-access

HPCC-35849 The coalescer must have Dali access.

Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@jakesmith
Merge remote-tracking branch 'origin/candidate-9.12.x' into candidate…
61a5df5 
…-9.14.x

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>

# Conflicts:
#	helm/hpcc/Chart.yaml
#	helm/hpcc/templates/_helpers.tpl
#	version.cmake
@jakesmith
Merge remote-tracking branch 'origin/candidate-9.14.x' into candidate…
e385111 
…-10.0.x

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@jakesmith
Merge remote-tracking branch 'origin/candidate-10.0.x' into candidate…
2e0168c 
…-10.2.x

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@jakesmith
Split off 9.12.80
1673ac4 
Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@jakesmith
Split off 9.14.62
1cb7305 
Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@jakesmith
Split off 10.0.36
e02d9d0 
Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@jakesmith
Split off 10.2.10
c30e314 
Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
GordonSmith and others added 21 commits March 12, 2026 08:29
@GordonSmith
Merge pull request hpcc-systems#21078 from GordonSmith/HPCC-35864-MET…
28245d3 
…RICS_VIEWS

HPCC-35864 Enable custom views for Metrics
@jakesmith
HPCC-35984 build: Add CMakeUserPresets.json to .gitignore
0762043 
Ignore CMakeUserPresets.json which users use for customized cmake
rules.
@GordonSmith
HPCC-35866 Improve lineage selection for URLs with only a scope id
3ec2bf8 
Refactored Metrics and MetricsGraph components to better differenciate between scope name and id

Signed-off-by: Gordon Smith <GordonJSmith@gmail.com>
@GordonSmith
Merge pull request hpcc-systems#21075 from GordonSmith/HPCC-35866-LIN…
553a585 
…KS2ACTIVITY_SCOPE

HPCC-35866 Improve lineage selection for URLs with only a scope id
@shamser
HPCC-34698 unittests for file cost and num reads updates
6d46ee8 
Added unit tests to verify proper tracking and updating of file cost
metrics and read count statistics of subfile and owning superfile statistics
following subfile reads.

Signed-off-by: Shamser Ahmed <shamser.ahmed@lexisnexisrisk.com>
@GordonSmith
HPCC-35857 Improve column sizing for metrics
800d1d4 
Also resolves HPCC-35854
Bump versions (security)

Signed-off-by: Gordon Smith <GordonJSmith@gmail.com>
@jakesmith
HPCC-35985 feat(dfs): Enable BM and alt. plane access to remote storage
ec27e8e 
Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@GordonSmith
Merge pull request hpcc-systems#21060 from GordonSmith/HPCC-35857_HPC…
78ac848 
…C-35854-DGRID_COLSWIDTHS

HPCC-35857 Improve column sizing for metrics
@ghalliday
Merge pull request hpcc-systems#21004 from jpmcmu/HPCC-35885
0c16fcf 
HPCC-35885 Provide upmergeability test workflow

Reviewed-by: Gordon Smith <gordon.smith@lexisnexisrisk.com>
Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#20982 from Michael-Gardner/HPCC-35787
fcfa6df 
HPCC-35787 Avoid rerunning tests on an empty force push

Reviewed-by: Gordon Smith <gordon.smith@lexisnexisrisk.com>
Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#21051 from jackdelv/issue35234
b308f09 
HPCC-35234 Ensure XRef handles trailing extensions correctly

Reviewed-by: Jake Smith <jake.smith@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#21094 from jakesmith/remotedfs-alt-ph…
1eb36c7 
…ysical-access

HPCC-35985 feat(dfs): Enable BM and alt. plane access to remote storage

Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#21093 from jakesmith/HPCC-35984-gitig…
6ed0722 
…nore-CMakeUserPresets.json

HPCC-35984 build: Add CMakeUserPresets.json to .gitignore

Reviewed-by: Gordon Smith <gordon.smith@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#21089 from jakesmith/HPCC-35981-deplo…
7af2b60 
…ymentName-envvar

HPCC-35981 feat(config): Allow deployment name config via env var

Reviewed-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#21088 from JamesDeFabia/HPCC-21197Doc…
ec82258 
…umentISOdatesfunctions

HPCC-21197 Document ISO date functions

Reviewed-By: Dan S. Camper <dan.camper@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#21040 from jackdelv/add-heartbeat-to-…
367344a 
…lost-and-logical-scans

HPCC-35935 Add progress logs to logical and lost file scans in XRef

Reviewed-by: Jake Smith <jake.smith@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@ghalliday
Merge pull request hpcc-systems#20250 from shamser/issue34698
aacd9b2 
HPCC-34698 Unittests for file cost and num reads updates

Reviewed-by: Jake Smith <jake.smith@lexisnexisrisk.com>
Merged-by: Gavin Halliday <gavin.halliday@lexisnexisrisk.com>
@jakesmith
Merge remote-tracking branch 'origin/candidate-9.12.x' into candidate…
c7fc528 
…-9.14.x

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>

# Conflicts:
#	helm/hpcc/Chart.yaml
#	helm/hpcc/templates/_helpers.tpl
#	version.cmake
@jakesmith
Merge remote-tracking branch 'origin/candidate-9.14.x' into candidate…
98ffaf9 
…-10.0.x

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>

# Conflicts:
#	helm/hpcc/Chart.yaml
#	helm/hpcc/templates/_helpers.tpl
#	version.cmake
@jakesmith
Merge remote-tracking branch 'origin/candidate-10.0.x' into candidate…
9ed8e16 
…-10.2.x
@jakesmith
Split off 10.2.22
331955a 
Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
@asselitx
Copy link
Author

asselitx commented Mar 18, 2026

@GordonSmith I thought this was the easiest way to maintain state for the change, but let me know if you'd prefer that I open a PR directly against upstream with all our changes.

GordonSmith and others added 5 commits March 19, 2026 10:48
@GordonSmith @asselitx
HPCC-35694 Refactor file size + compression ratio
63788f3 
Signed-off-by: Gordon Smith<GordonJSmith@gmail.com>
@GordonSmith @asselitx
HPCC-35694 Refactor file size + compression ratio
da2c1ef 
Signed-off-by: Gordon Smith<GordonJSmith@gmail.com>
@GordonSmith @asselitx
Address latest copilot comments.
a768dd3
@GordonSmith @asselitx
Refactor ESP service interface instructions and update DFU query sort…
0945eea 
… order mappings
@asselitx
HPCC-35694 Expose single field for physical File Size
4814af3 
Regardless of compression status this field has the actual physical size
on disk.

- Add FileSize field to DFUQuery response to hold DFUSFSize
- Update ECL Watch to use FileSize when present

Signed-off-by: Terrence Asselin <terrence.asselin+copilot@lexisnexisrisk.com>
@asselitx asselitx force-pushed the hpcc-35694-dfustortby-compressedsize branch from a963176 to 4814af3 Compare March 19, 2026 16:24
@asselitx asselitx closed this Mar 19, 2026
@asselitx
Copy link
Author

asselitx commented Mar 19, 2026

MOved to direct PR against HPCC-Platform

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.