Add Host Profiler Feature by mackjmr · Pull Request #2403 · DataDog/datadog-operator

mackjmr · 2025-12-12T15:51:30Z

What does this PR do?

Add support for full host profiler.

Motivation

OTAGENT-711

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

Agent: Feature has not been released yet. For now, only image available in dockerhub datadog/ddot-ebpf-dev repo.
Cluster Agent: vX.Y.Z

Describe your test plan

Prereqs:

Staging API Key
Staging APP key with continuous_profiler_read permission
Cluster in Real Linux environment (The host profiler relies on eBPF, which requires direct access to a real Linux kernel)

Test 1. Configmap

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  annotations:
    agent.datadoghq.com/host-profiler-enabled: "true"
    agent.datadoghq.com/host-profiler-configmap-name: "custom-config-map"
    experimental.agent.datadoghq.com/image-override-config: |
      {
        "host-profiler": {
          "name": "datadog/ddot-ebpf-dev:nightly-latest"
        }
      }
spec:
  override:
    nodeAgent:
      env:
        - name: DD_HOSTNAME
          value: "test1"
        - name: DD_API_KEY
          value: xxx
        - name: DD_APP_KEY
          value: xxx
        - name: DD_SITE
          value: datad0g.com

  global:
    site: datad0g.com
    kubelet:
      tlsVerify: false
    credentials:
      apiKey: xxx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-config-map
  namespace: system
data:
  host-profiler-config.yaml: |-
    extensions:
      ddprofiling/default:
      hpflare/default:
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      hostprofiler:
        symbol_uploader:
          enabled: true
          symbol_endpoints:
            - site: ${env:DD_SITE}
              api_key: ${env:DD_API_KEY}
              app_key: ${env:DD_APP_KEY}
    processors:
      infraattributes/default:
        cardinality: 2 # HighCardinality
        allow_hostname_override: true
      cumulativetodelta: {}

    exporters:
      debug: {}
      otlphttp:
        metrics_endpoint: https://otlp.datad0g.com/v1/metrics
        profiles_endpoint: https://intake.profile.datad0g.com/v1development/profiles
        headers:
          dd-api-key: ${env:DD_API_KEY}
          dd-otel-metric-config: '{"resource_attributes_as_tags": true}'
    service:
      extensions: [hpflare/default]
      pipelines:
        profiles:
          receivers: [hostprofiler]
          processors: [infraattributes/default]
          exporters: [otlphttp, debug]   
        metrics:
          receivers: [otlp]
          processors: [cumulativetodelta, infraattributes/default]
          exporters: [otlphttp]

Ensure:

host-profiler container is running in agent pod with no error
you see profiles under host:test1 in staging.

Test 2. Inline config

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
    agent.datadoghq.com/host-profiler-enabled: "true"
    experimental.agent.datadoghq.com/image-override-config: |
      {
        "host-profiler": {
          "name": "datadog/ddot-ebpf-dev:nightly-latest"
        }
      }
    agent.datadoghq.com/host-profiler-configdata: |-
          extensions:
            ddprofiling/default:
            hpflare/default:
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            hostprofiler:
              symbol_uploader:
                enabled: true
                symbol_endpoints:
                  - site: ${env:DD_SITE}
                    api_key: ${env:DD_API_KEY}
                    app_key: ${env:DD_APP_KEY}
          processors:
            infraattributes/default:
              cardinality: 2 # HighCardinality
              allow_hostname_override: true
            cumulativetodelta: {}
          exporters:
            debug: {}
            otlphttp:
              metrics_endpoint: https://otlp.datad0g.com/v1/metrics
              profiles_endpoint: https://intake.profile.datad0g.com/v1development/profiles
              headers:
                dd-api-key: ${env:DD_API_KEY}
                dd-otel-metric-config: '{"resource_attributes_as_tags": true}'
          service:
            extensions: [hpflare/default]
            pipelines:
              profiles:
                receivers: [hostprofiler]
                processors: [infraattributes/default]
                exporters: [otlphttp, debug]   
              metrics:
                receivers: [otlp]
                processors: [cumulativetodelta, infraattributes/default]
                exporters: [otlphttp]
spec:
  override:
    nodeAgent:
      env:
        - name: DD_HOSTNAME
          value: "test2"
        - name: DD_API_KEY
          value: xxx
        - name: DD_APP_KEY
          value: xxx
        - name: DD_SITE
          value: datad0g.com
  global:
    site: datad0g.com
    kubelet:
      tlsVerify: false
    credentials:
      apiKey: xxx

Test 3. HostPID manually disabled

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  annotations:
    experimental.agent.datadoghq.com/image-override-config: |
      {
        "host-profiler": {
          "name": "datadog/ddot-ebpf-dev:nightly-latest"
        }
      }
    agent.datadoghq.com/host-profiler-enabled: "true"
spec:
  override:
    nodeAgent:
      env:
        - name: DD_HOSTNAME
          value: "test3"
        - name: DD_API_KEY
          value: xxx
        - name: DD_APP_KEY
          value: xxx
        - name: DD_SITE
          value: datad0g.com
      hostPID: false

  global:
    site: datad0g.com
    kubelet:
      tlsVerify: false
    credentials:
      apiKey: xxx

Ensure:

host-profiler container is not running in agent pod
operator logs error: Host PID is required to run the host profiler. Please enable host PID or disable the host profiler

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label

maycmlee

Just a tiny nit, but approving

…nto mackjmr/host-profiler-feature

codecov-commenter · 2026-01-09T14:37:01Z

Codecov Report

❌ Patch coverage is 56.20915% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.09%. Comparing base (f2d7517) to head (5154dd9).

Files with missing lines	Patch %	Lines
...oller/datadogagent/feature/hostprofiler/feature.go	68.54%	29 Missing and 10 partials ⚠️
...controller/datadogagent/component/agent/default.go	0.00%	20 Missing ⚠️
...nal/controller/datadogagent/feature/utils/utils.go	0.00%	5 Missing ⚠️
pkg/images/images.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2403      +/-   ##
==========================================
+ Coverage   37.98%   38.09%   +0.11%     
==========================================
  Files         298      299       +1     
  Lines       25029    25182     +153     
==========================================
+ Hits         9508     9594      +86     
- Misses      14796    14853      +57     
- Partials      725      735      +10

Flag	Coverage Δ
unittests	`38.09% <56.20%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/datadogagent/controller.go	`53.57% <ø> (ø)`
...ontroller/datadogagent/override/podtemplatespec.go	`77.39% <100.00%> (+0.15%)`	⬆️
pkg/images/images.go	`95.27% <0.00%> (-2.31%)`	⬇️
...nal/controller/datadogagent/feature/utils/utils.go	`0.00% <0.00%> (ø)`
...controller/datadogagent/component/agent/default.go	`36.36% <0.00%> (-1.14%)`	⬇️
...oller/datadogagent/feature/hostprofiler/feature.go	`68.54% <68.54%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2d7517...5154dd9. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

levan-m · 2026-01-09T20:45:59Z

+	// If a user disabled HostPID manually, error out rather than enabling it for them.
+	if nodeAgent, ok := ddaSpec.Override[v2alpha1.NodeAgentComponentName]; ok {
+		if nodeAgent.HostPID != nil && apiutils.BoolValue(nodeAgent.HostPID) == false {
+			o.logger.Error(errHostPIDDisabledManually, "Host PID is required to run the host profiler. Please enable host PID or disable the host profiler")


nit - this won't be logged unless someone has node agent override configured. Probably log error only if hostProfilerEnabled == true and there is no override or no HostPID is set.

nit - this won't be logged unless someone has node agent override configured

This is the goal. My assumption is that if hostPID was disabled manually by the user, then this was voluntary and we shouldn't enable it for the purpose of the host profiler feature. Instead I want to error out. That said, based on #2403 (comment) I guess I can just force enable it always ?

sorry, I misunderstood intent here. What you say makes sense and could go either way so we can leave it as is.

Ill remove it to stay consistent with other paths that do this.

levan-m · 2026-01-09T20:53:13Z

+	if nodeAgent, ok := ddaSpec.Override[v2alpha1.NodeAgentComponentName]; ok {
+		if nodeAgent.HostPID != nil && apiutils.BoolValue(nodeAgent.HostPID) == false {
+			o.logger.Error(errHostPIDDisabledManually, "Host PID is required to run the host profiler. Please enable host PID or disable the host profiler")
+			o.hostPIDDisabledManually = true


nit - recently there was a change #2365 in npm, usm features to enabled host pid when system probe is enabled. I think same can be done here.

I do enable it

datadog-operator/internal/controller/datadogagent/feature/hostprofiler/feature.go

Line 172 in b4a6f43

managers.PodTemplateSpec().Spec.HostPID = *apiutils.NewBoolPointer(true)

. But based on the other PR, I guess I can skip the check that hostPID is disabled manually check althogether and just force enable ?

see above comment.

levan-m · 2026-01-10T00:47:09Z


+// GetLatestAgentImage returns the latest host profiler image
+func GetLatestHostProfilerImage() string {
+	image := newImage(DockerHubContainerRegistry, DefaultHostProfilerDevImageName, DefaultHostProfilerDevImageLatestTag, false, false, false)


This doesn't really work and isn't really needed at all. This is called when Daemonset is initialized and is overwritted by global defaults so it's reset to gcr here.

To override container level image you can include this annotation with the one which enabled profile (this mechanism was added by #1730)

agent.datadoghq.com/host-profiler-enabled: "true" experimental.agent.datadoghq.com/image-override-config: | { "host-profiler": { "name": "docker.io/datadog/ddot-ebpf-dev:nightly-latest" } }

This doesn't really work and isn't really needed at all.

Yeah, I noticed that and had to set registry: docker.io/datadog

I'm fine with using annotation until we get official image. SHould I just unset the image field:

datadog-operator/internal/controller/datadogagent/component/agent/default.go

Line 496 in d88efc7

Image: hostProfilerImage(),

in the container struct in that case ?

you can remove docker.io/GetLatestHostProfilerImage and use default image.

levan-m · 2026-01-12T16:15:10Z

+	// If a user disabled HostPID manually, error out rather than enabling it for them.
+	if nodeAgent, ok := ddaSpec.Override[v2alpha1.NodeAgentComponentName]; ok {
+		if nodeAgent.HostPID != nil && apiutils.BoolValue(nodeAgent.HostPID) == false {
+			o.logger.Error(errHostPIDDisabledManually, "Host PID is required to run the host profiler. Please enable host PID or disable the host profiler")


sorry, I misunderstood intent here. What you say makes sense and could go either way so we can leave it as is.

levan-m · 2026-01-12T16:15:34Z

+	if nodeAgent, ok := ddaSpec.Override[v2alpha1.NodeAgentComponentName]; ok {
+		if nodeAgent.HostPID != nil && apiutils.BoolValue(nodeAgent.HostPID) == false {
+			o.logger.Error(errHostPIDDisabledManually, "Host PID is required to run the host profiler. Please enable host PID or disable the host profiler")
+			o.hostPIDDisabledManually = true


see above comment.

levan-m · 2026-01-12T16:16:36Z


+// GetLatestAgentImage returns the latest host profiler image
+func GetLatestHostProfilerImage() string {
+	image := newImage(DockerHubContainerRegistry, DefaultHostProfilerDevImageName, DefaultHostProfilerDevImageLatestTag, false, false, false)


you can remove docker.io/GetLatestHostProfilerImage and use default image.

mackjmr added 4 commits December 10, 2025 14:00

Host Profiler Feature

22678ac

Add volume mounts + host Pid

c575953

error out on host PID disabled manually

cea4560

add tests

76def70

mackjmr added the enhancement New feature or request label Dec 15, 2025

mackjmr added 4 commits December 15, 2025 17:46

add test

ebd7661

update crd + image + add experimental warning

827edb6

update images

8b70422

fix example

884206f

mackjmr added this to the v1.23.0 milestone Dec 16, 2025

mackjmr added 2 commits December 16, 2025 17:58

fix lint

465e8a7

undo version change

a900e6f

mackjmr changed the title ~~Mackjmr/host profiler feature~~ Add Host Profiler Feature Dec 18, 2025

mackjmr marked this pull request as ready for review December 18, 2025 13:56

mackjmr requested review from a team as code owners December 18, 2025 13:56

maycmlee approved these changes Dec 18, 2025

View reviewed changes

Comment thread docs/configuration.v2alpha1.md Outdated

mackjmr and others added 7 commits January 5, 2026 12:58

Merge branch 'main' into mackjmr/host-profiler-feature

e743f76

switch to annotations

f5588f1

Merge remote-tracking branch 'origin/mackjmr/host-profiler-feature' i…

7e6870a

…nto mackjmr/host-profiler-feature

update tests to reflect annotations

da7898a

fix lint

a8c00d6

update version

ba58bd1

fix lint

a1ce5d2

make generate

b4a6f43

levan-m reviewed Jan 10, 2026

View reviewed changes

feeback

d88efc7

levan-m approved these changes Jan 12, 2026

View reviewed changes

mackjmr mentioned this pull request Jan 13, 2026

Cleanup Host Profiler #2485

Merged

3 tasks

Merge branch 'main' into mackjmr/host-profiler-feature

5154dd9

levan-m merged commit b4b6e01 into main Jan 13, 2026
33 checks passed

levan-m deleted the mackjmr/host-profiler-feature branch January 13, 2026 19:56

Conversation

mackjmr commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Minimum Agent Versions

Describe your test plan

Test 1. Configmap

Test 2. Inline config

Test 3. HostPID manually disabled

Checklist

Uh oh!

maycmlee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mackjmr Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mackjmr commented Dec 12, 2025 •

edited

Loading

codecov-commenter commented Jan 9, 2026 •

edited

Loading

mackjmr Jan 12, 2026 •

edited

Loading