Skip to content

Implement remote DFS support for LogicalFileListFiltered#170

Draft
Copilot wants to merge 4 commits intoHPCC-35827-LogicalFileListFilteredfrom
copilot/add-remote-dfs-support
Draft

Implement remote DFS support for LogicalFileListFiltered#170
Copilot wants to merge 4 commits intoHPCC-35827-LogicalFileListFilteredfrom
copilot/add-remote-dfs-support

Conversation

Copy link

Copilot AI commented Feb 19, 2026

Adds remote DFS capability to LogicalFileListFiltered, enabling file listing across remote storage planes. Previously, the remoteDfs parameter threw "not supported yet" exception.

Changes

ESDL Interface (ws_dfs.ecm)

  • Added DFSListFiltered method (version 1.02)
  • Request: mask, filters (human-readable), fields, limits
  • Response: compressed property tree with file metadata

Server Handler (ws_dfsservice.cpp)

  • CWsDfsEx::onDFSListFiltered(): parses user-friendly filters server-side, queries Dali via getDFAttributesFilteredIterator(), returns LZ-compressed results
  • parseUserFilterSyntax(): translates owner:jsmith, size>1000 to internal Dali filter format
  • validateFileField(): field name validation with alias support (superfile→numsubfiles, rowcount→recordcount)

Client (ws_dfsclient.cpp)

  • listFilteredDFSFiles(): discovers remote service via getRemoteStorage(), handles SSL/auth, sends human-readable parameters, decompresses response

Plugin (fileservices.cpp)

  • fsLogicalFileListFiltered(): routes to remote service when remoteDfs specified, otherwise local Dali query unchanged

Usage

IMPORT Std;

files := Std.File.LogicalFileListFiltered(
    mask := 'production::*',
    filters := 'owner:analytics, size>1000000, modified>=2024-01-01',
    fields := 'name,size,modified,cluster',
    remoteDfs := 'backup-cluster'  // Now functional
);

Implementation Notes

  • Filter parsing centralized on server to maintain single source of truth
  • Follows existing lookupDFSFile pattern for remote communication, lease management, and retry logic
  • Limits: 1M client-side, 10M server-side (validated at respective layers)
  • Requires ESDL build to generate interface stubs before compilation

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • downloads.apache.org
    • Triggering command: /usr/bin/curl curl --fail --retry 3 -L REDACTED --create-dirs --output apr-1.7.6.tar.bz2.7325.part -H User-Agent: vcpkg/2025-09-03-4580816534ed8fd9634ac83d46471440edd82dfe (curl) u/13/cc1 --64�� -o conftest.o /snap/bin/as h include/antlr3commontoken.h include/antlr3commontree.h include/antlr3commontreeadaptor.h incl --global ild/vcpkg_downlo. as (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

jakesmith and others added 3 commits February 13, 2026 17:51
Introduces new file listing API with user-friendly filter syntax and field selection.
LogicalFileList is marked DEPRECATED, users should use new function for
efficiency and capability.

Features:
- Filter syntax: wildcards (owner:jsmith), numeric comparisons (size>1000000),
  date ranges (modified>2024-01-01), property checks (has:description),
  file type filtering (is:superfile, is:normal)
- Field projection to select specific result fields
- Configurable file limits with breach detection
- Overflow/underflow protection for numeric filters

Implementation:
- FileListResultFieldSource for IFieldSource pattern
- parseUserFilterSyntax converts user syntax to internal DFUQFilter
- Field validation via dfuFieldAttrPathMap
- Default fields: name, modified, size, rowcount, cluster, superfile, owner

Testing:
- 12 regression test cases covering filtering, projection, limits, edge cases

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
Tighten LogicalFileListFiltered parsing; rename remoteDfs; update regress tests

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
…se 1

Co-authored-by: jakesmith <902700+jakesmith@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement missing remoteDfs side of LogicalFileListFiltered Implement remote DFS support for LogicalFileListFiltered Feb 19, 2026
Copilot AI requested a review from jakesmith February 19, 2026 11:38
@jakesmith jakesmith force-pushed the HPCC-35827-LogicalFileListFiltered branch from 90b89c5 to c7a19c7 Compare February 20, 2026 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants