Skip to content

feat: changed datacrumbs generation based logic to dynamic probe attach.#179

Open
hariharan-devarajan wants to merge 2 commits into
developfrom
checkpoint/04-13-26
Open

feat: changed datacrumbs generation based logic to dynamic probe attach.#179
hariharan-devarajan wants to merge 2 commits into
developfrom
checkpoint/04-13-26

Conversation

@hariharan-devarajan
Copy link
Copy Markdown
Member

@hariharan-devarajan hariharan-devarajan commented Apr 13, 2026

  1. Reduces the footprint of sudo review code.
  2. Removes unneccessary scripts only support systemd in sudo.
  3. Creates a system configuration and secrete to sign user probes.
  4. No predefined probes, everything is a probe submitted by user.
  5. Validation of probe is on runtime and we maintain invalid list.
  6. Updated documentation.
  7. No MPI dependency.
  8. All other features moved to utils.
  9. Composability replaced with user input of probes.
  10. Include utils part of build for ease of deployment no direct dependency.
  11. Updated the RuntimeConfigurationManager to utilize an SQLite database for managing runtime probe states, replacing the previous JSON-based approach.
  12. Introduced new methods for loading and persisting runtime probe states, including successful and invalid probes.
  13. Modified the attach_runtime_probes function to record successful and invalid probes in the new database.
  14. Enhanced error handling and logging for database operations.
  15. Removed JSON dependencies in favor of SQLite for system configuration management in SystemConfigurator.
  16. Added new utility functions for executing SQLite commands and writing key-value pairs to the database.
  17. Updated related header files to reflect changes in data structures and method signatures.

The flow for the new changes are as follows:

  • The system configuration is done by system admin in user-space and they create an install folder for datacrumbs.
  • The administrator will deploy datacrumbs
    • deploy signing service
    • deploy datacrumbs service as a part of prolog and epilog
    • add flux args plugin for allocation
  • User will clone datacrumbs-utils and create probe lists and sign them using the signing service
  • Then during allocation like flux alloc users can enable datacrumsbs and pass the probes file to the allocation.
  • Datacrumbs on load will verify the probes using the signing key and then dynamically attach every probe in the system using the bpf APIs directly.
  • The user use datacrumsb_client_so to allow datacrumbs to track which processes should be traced/profiled.
  • On epilog, it detaches the probes , creates the performance events, and exits.

@hariharan-devarajan hariharan-devarajan requested a review from a team as a code owner April 13, 2026 23:32
1. Reduces the footprint of sudo review code.
2. Removes unneccessary scripts only support systemd in sudo.
3. Creates a system configuration and secrete to sign user probes.
4. No predefined probes, everything is a probe submitted by user.
5. Validation of probe is on runtime and we maintain invalid list.
6. Updated documentation.
7. No MPI dependency.
8. All other features moved to utils.
9. Composability replaced with user input of probes.
10. Include utils part of build for ease of deployment no direct
    dependency.
11. Updated the RuntimeConfigurationManager to utilize an SQLite database for managing runtime probe states, replacing the previous JSON-based approach.
12. Introduced new methods for loading and persisting runtime probe states, including successful and invalid probes.
13. Modified the attach_runtime_probes function to record successful and invalid probes in the new database.
14. Enhanced error handling and logging for database operations.
15. Removed JSON dependencies in favor of SQLite for system configuration management in SystemConfigurator.
16. Added new utility functions for executing SQLite commands and writing key-value pairs to the database.
17. Updated related header files to reflect changes in data structures and method signatures.
@hariharan-devarajan hariharan-devarajan force-pushed the checkpoint/04-13-26 branch 10 times, most recently from 1fb6fea to 41c9430 Compare May 26, 2026 04:12
- Added ProbeManagerService to handle probe management functionality, including socket initialization, client handling, and response building.
- Created main entry point for ProbeManagerService.
- Removed SignProbesService and its associated files, as its functionality is now integrated into ProbeManagerService.
- Updated server main function to remove probe signing warnings and streamline the process.
- Deleted system configurator files as they are no longer needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant