Increasing log level for logs that are not necessary for debugging on prod by nakshah87 · Pull Request #211 · aws/amazon-network-policy-controller-k8s

nakshah87 · 2026-02-17T04:28:26Z

What type of PR is this?
With an increase in pod scale, and high pod churn the "Pod and policy namespace mismatch" log was causing an increase in disk utilization.

The code logic walks every policy for every pod CRUD operation and prints this during mismatch.

Optimizing the following logs to not print in a loop but consolidate the info in a list and print once at the end of the loop.

skipping headless service when populating EndpointInfo
skipping pod selector mismatched service when populating EndpointInfo

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add steps to reproduce and the controller logs:

Testing done on this change:

Automation added to e2e:

Will this PR introduce any new dependencies?:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:

Does this PR introduce any user-facing change?: No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

parlakisik

Could you provide more context in the PR details?
Why do you characterize this as noisy? How much did the log volume change after this improvement? Are there any additional logs being generated?

nakshah87 · 2026-02-21T20:46:13Z

Could you provide more context in the PR details? Why do you characterize this as noisy? How much did the log volume change after this improvement? Are there any additional logs being generated?

@parlakisik I updated the Description and provided more context.

oliviassss · 2026-02-23T17:33:31Z

 			Ports: portList,
 		})
 	}
+	r.logger.Info("headlessServiceList", headlessServiceList)


nit: with consolidating log, we can just remove L429. can add more clear log message here like r.logger.Info("skipping headless services: ", headlessServiceList)

oliviassss · 2026-02-23T17:34:34Z

 	}
+	r.logger.Info("headlessServiceList", headlessServiceList)
+	r.logger.Info("nonMatchingPodSelectorList", nonMatchingPodSelectorList)
+	r.logger.Info("Network peers appended ", networkPeers)


L467 is not needed, since it may add a very large chunk of log lines in every reconcile. we have the skipping list, so it should be fine.

oliviassss · 2026-02-23T17:34:38Z

 		})
 	}
+	r.logger.Info("headlessServiceList", headlessServiceList)
+	r.logger.Info("nonMatchingPodSelectorList", nonMatchingPodSelectorList)


same here. remove L435 and add more clear log message here.

Copilot

Pull request overview

This PR reduces production log volume in the resolvers by lowering verbosity for high-churn informational logs and consolidating per-item skip logging into aggregated output.

Changes:

Downgrade the “Pod and policy namespace mismatch” log to verbosity level V(1) in the policy reference resolver.
Accumulate “skipped service” reasons while resolving service ClusterIPs and emit them once after iterating services.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`pkg/resolvers/policies_for_pod.go`	Lowers verbosity for namespace-mismatch logging during pod/policy evaluation.
`pkg/resolvers/endpoints.go`	Aggregates skip details for headless and selector-mismatched services during service ClusterIP resolution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-26T00:46:51Z

+				headlessServiceList = append(headlessServiceList, fmt.Sprintf("%s/%s", svc.Namespace, svc.Name))
+			}
+			headlessServiceCount++
+			r.logger.V(1).Info("skipping headless service when populating EndpointInfo", "serviceName", svc.Name, "serviceNamespace", svc.Namespace)


The V(1) log inside the loop should be removed. The purpose of this PR is to reduce logging overhead by consolidating logs, but this line still logs for every headless service encountered (at V(1) verbosity). Since the consolidated summary is logged at lines 475-476, this per-iteration log is redundant and should be removed to achieve the intended optimization.

Suggested change

r.logger.V(1).Info("skipping headless service when populating EndpointInfo", "serviceName", svc.Name, "serviceNamespace", svc.Namespace)

Copilot · 2026-02-26T00:46:51Z

+				nonMatchingPodSelectorList = append(nonMatchingPodSelectorList, fmt.Sprintf("%s/%s (expectedPS: %s)", svc.Namespace, svc.Name, svcSelector.String()))
+			}
+			nonMatchingPodSelectorCount++
+			r.logger.V(1).Info("skipping pod selector mismatched service when populating EndpointInfo", "serviceName", svc.Name, "serviceNamespace", svc.Namespace, "expectedPS", svcSelector)


The V(1) log inside the loop should be removed. Similar to the headless service case, this line still logs for every mismatched service encountered (at V(1) verbosity), which defeats the purpose of consolidating logs. Since the consolidated summary is logged at lines 479-480, this per-iteration log is redundant and should be removed to achieve the intended optimization.

Suggested change

r.logger.V(1).Info("skipping pod selector mismatched service when populating EndpointInfo", "serviceName", svc.Name, "serviceNamespace", svc.Namespace, "expectedPS", svcSelector)

V(1) log is retained for testing will not affect production. should be fine.

Copilot · 2026-02-26T00:46:52Z

+	var headlessServiceList []string
+	var nonMatchingPodSelectorList []string
+	var headlessServiceCount int
+	var nonMatchingPodSelectorCount int
+	const maxSampleSize = 50


Consider initializing these slices with a capacity (e.g., make([]string, 0, maxSampleSize)) to avoid multiple allocations as items are appended. This is a minor optimization but aligns with Go best practices when the maximum size is known in advance.

Suggested change

var headlessServiceList []string

var nonMatchingPodSelectorList []string

var headlessServiceCount int

var nonMatchingPodSelectorCount int

const maxSampleSize = 50

const maxSampleSize = 50

headlessServiceList := make([]string, 0, maxSampleSize)

nonMatchingPodSelectorList := make([]string, 0, maxSampleSize)

var headlessServiceCount int

var nonMatchingPodSelectorCount int

… prod

nakshah87 requested a review from a team as a code owner February 17, 2026 04:28

nakshah87 requested review from haouc and oliviassss February 17, 2026 04:28

oliviassss reviewed Feb 17, 2026

View reviewed changes

Comment thread pkg/resolvers/endpoints.go

oliviassss reviewed Feb 17, 2026

View reviewed changes

Comment thread pkg/resolvers/endpoints.go

oliviassss reviewed Feb 17, 2026

View reviewed changes

Comment thread pkg/resolvers/endpoints.go Outdated

parlakisik reviewed Feb 17, 2026

View reviewed changes

nakshah87 force-pushed the log-verbosity branch from 92759a5 to 4752755 Compare February 21, 2026 19:40

oliviassss reviewed Feb 23, 2026

View reviewed changes

nakshah87 force-pushed the log-verbosity branch 2 times, most recently from 461463f to 26696f0 Compare February 24, 2026 03:40

oliviassss previously approved these changes Feb 24, 2026

View reviewed changes

jayanthvn requested a review from Copilot February 24, 2026 20:35

Copilot started reviewing on behalf of jayanthvn February 24, 2026 20:35 View session

Copilot AI reviewed Feb 24, 2026

View reviewed changes

Comment thread pkg/resolvers/endpoints.go Outdated

Comment thread pkg/resolvers/endpoints.go Outdated

nakshah87 dismissed oliviassss’s stale review via aae21b5 February 24, 2026 23:38

nakshah87 force-pushed the log-verbosity branch from 26696f0 to aae21b5 Compare February 24, 2026 23:38

oliviassss requested a review from Copilot February 26, 2026 00:44

Copilot started reviewing on behalf of oliviassss February 26, 2026 00:44 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

Increasing log level for logs that are not necessary for debugging on…

75d7aaf

… prod

nakshah87 force-pushed the log-verbosity branch from aae21b5 to 75d7aaf Compare February 26, 2026 22:17

oliviassss approved these changes Feb 26, 2026

View reviewed changes

nakshah87 merged commit d977c0e into aws:main Feb 26, 2026
4 checks passed

Conversation

nakshah87 commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parlakisik left a comment

Choose a reason for hiding this comment

Uh oh!

nakshah87 commented Feb 21, 2026

Uh oh!

oliviassss Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

nakshah87 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

oliviassss Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nakshah87 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

oliviassss Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

nakshah87 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

oliviassss Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nakshah87 commented Feb 17, 2026 •

edited

Loading

oliviassss Feb 23, 2026 •

edited

Loading