Skip to content

Add telemetry for ProxySettingsPolicy #4388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tataruty wants to merge 1 commit into
base: feat/proxy-settings-policy
Choose a base branch
from
Open

Add telemetry for ProxySettingsPolicy #4388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 43 additions & 20 deletions internal/controller/telemetry/collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,48 @@ type NGFResourceCounts struct {
UpstreamSettingsPolicyCount int64
// GatewayAttachedNpCount is the total number of NginxProxy resources that are attached to a Gateway.
GatewayAttachedNpCount int64
// GatewayAttachedProxySettingsPolicyCount is the number of relevant ProxySettingsPolicies
// attached at the Gateway level.
GatewayAttachedProxySettingsPolicyCount int64
// RouteAttachedProxySettingsPolicyCount is the number of relevant ProxySettingsPolicies
// attached at the Route level.
RouteAttachedProxySettingsPolicyCount int64
Comment on lines +113 to +118
Copy link
Contributor

@bjee19 bjee19 Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something to note that might make sense when you test later on, the order we send data over in data.avdl can't actually change. XCDF pipeline won't allow us to change the order, so all additions need to be appended, meaning you'll run into issues adding the count to this struct since its autogenerated. You'll need to add it outside of the struct like i do with inferencepoolCount.

Copy link
Contributor Author

@tataruty tataruty Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i've already noticed it here in unit tests.. was a bit strange to fix the order :D
So, you mean that order should be same as in the tests or as in the fields list here?

Copy link
Contributor

@bjee19 bjee19 Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The avdl file, needs to have its additional fields appended. Since that file is autogenerated, we'll need to add these new policy count fields to a different place so the autogenerated file is correct.

Copy link
Contributor Author

@tataruty tataruty Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so no changes needed? I just wanted to add 16, 17, but then noticed that 16 is used for inference and used 17, 18 instead.

Copy link
Contributor

@bjee19 bjee19 Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry its a little confusing, but not quite. In the data.avdl file that is changed by this PR, we've added these two fields:

long? GatewayAttachedProxySettingsPolicyCount = null;
long? RouteAttachedProxySettingsPolicyCount = null;

However, these were added inbetween an existing: long? GatewayAttachedNpCount = null; and long? NginxPodCount = null;.

This disrupts the order and does not follow the rule set by the xcdf folks that when we change the avdl file, the additional fields have to be appended.

We'll need to find a way to add these new fields at the bottom of the auto-generated avdl file.

Copy link
Contributor Author

@tataruty tataruty Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't see how it can be done, if ProxySettingsPolicy is inside NGFResourceCount, and it is in the middle of the generated structure

Copy link
Contributor Author

@tataruty tataruty Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image

}

func (rc *NGFResourceCounts) CountPolicies(g *graph.Graph) {
rc.BackendTLSPolicyCount = int64(len(g.BackendTLSPolicies))

for policyKey, policy := range g.NGFPolicies {
switch policyKey.GVK.Kind {
case kinds.ClientSettingsPolicy:
if len(policy.TargetRefs) == 0 {
continue
}

if policy.TargetRefs[0].Kind == kinds.Gateway {
rc.GatewayAttachedClientSettingsPolicyCount++
} else {
rc.RouteAttachedClientSettingsPolicyCount++
}
case kinds.ObservabilityPolicy:
rc.ObservabilityPolicyCount++
case kinds.UpstreamSettingsPolicy:
rc.UpstreamSettingsPolicyCount++
case kinds.ProxySettingsPolicy:
if len(policy.TargetRefs) == 0 {
continue
}

for _, tr := range policy.TargetRefs {
switch tr.Kind {
case kinds.Gateway:
rc.GatewayAttachedProxySettingsPolicyCount++
case kinds.HTTPRoute, kinds.GRPCRoute:
rc.RouteAttachedProxySettingsPolicyCount++
}
}
}
}
}

// DataCollectorConfig holds configuration parameters for DataCollectorImpl.
Expand Down Expand Up @@ -244,26 +286,7 @@ func collectGraphResourceCount(
}
}

ngfResourceCounts.BackendTLSPolicyCount = int64(len(g.BackendTLSPolicies))

for policyKey, policy := range g.NGFPolicies {
switch policyKey.GVK.Kind {
case kinds.ClientSettingsPolicy:
if len(policy.TargetRefs) == 0 {
continue
}

if policy.TargetRefs[0].Kind == kinds.Gateway {
ngfResourceCounts.GatewayAttachedClientSettingsPolicyCount++
} else {
ngfResourceCounts.RouteAttachedClientSettingsPolicyCount++
}
case kinds.ObservabilityPolicy:
ngfResourceCounts.ObservabilityPolicyCount++
case kinds.UpstreamSettingsPolicy:
ngfResourceCounts.UpstreamSettingsPolicyCount++
}
}
ngfResourceCounts.CountPolicies(g)

ngfResourceCounts.NginxProxyCount = int64(len(g.ReferencedNginxProxies))
ngfResourceCounts.SnippetsFilterCount = int64(len(g.SnippetsFilters))
Expand Down
36 changes: 36 additions & 0 deletions internal/controller/telemetry/collector_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,22 @@ var _ = Describe("Collector", Ordered, func() {
NsName: types.NamespacedName{Namespace: "test", Name: "UpstreamSettingsPolicy-1"},
GVK: schema.GroupVersionKind{Kind: kinds.UpstreamSettingsPolicy},
}: {},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-1"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.Gateway}}},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-2"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.HTTPRoute}}},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-3"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.GRPCRoute}}},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-4"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.Gateway}, {Kind: kinds.HTTPRoute}, {Kind: kinds.GRPCRoute}}},
},
ReferencedNginxProxies: map[types.NamespacedName]*graph.NginxProxy{
{Namespace: "test", Name: "NginxProxy-1"}: &gcNP,
Expand Down Expand Up @@ -494,6 +510,8 @@ var _ = Describe("Collector", Ordered, func() {
SnippetsFilterCount: 3,
UpstreamSettingsPolicyCount: 1,
GatewayAttachedNpCount: 2,
GatewayAttachedProxySettingsPolicyCount: 2,
RouteAttachedProxySettingsPolicyCount: 4,
}
expData.ClusterVersion = "1.29.2"
expData.ClusterPlatform = "kind"
Expand Down Expand Up @@ -700,6 +718,22 @@ var _ = Describe("Collector", Ordered, func() {
NsName: types.NamespacedName{Namespace: "test", Name: "UpstreamSettingsPolicy-1"},
GVK: schema.GroupVersionKind{Kind: kinds.UpstreamSettingsPolicy},
}: {},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-1"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.Gateway}}},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-2"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.HTTPRoute}}},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-plural"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {TargetRefs: []graph.PolicyTargetRef{{Kind: kinds.Gateway}, {Kind: kinds.HTTPRoute}, {Kind: kinds.GRPCRoute}}},
{
NsName: types.NamespacedName{Namespace: "test", Name: "ProxySettingsPolicy-empty"},
GVK: schema.GroupVersionKind{Kind: kinds.ProxySettingsPolicy},
}: {},
},
ReferencedNginxProxies: map[types.NamespacedName]*graph.NginxProxy{
{Namespace: "test", Name: "NginxProxy-1"}: {Valid: true},
Expand Down Expand Up @@ -794,6 +828,8 @@ var _ = Describe("Collector", Ordered, func() {
UpstreamSettingsPolicyCount: 1,
GatewayAttachedNpCount: 1,
BackendTLSPolicyCount: 1,
GatewayAttachedProxySettingsPolicyCount: 2,
RouteAttachedProxySettingsPolicyCount: 3,
}
expData.NginxPodCount = 1
expData.InferencePoolCount = 1
Expand Down
8 changes: 8 additions & 0 deletions internal/controller/telemetry/data.avdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,14 @@ attached at the Gateway level. */
/** GatewayAttachedNpCount is the total number of NginxProxy resources that are attached to a Gateway. */
long? GatewayAttachedNpCount = null;

/** GatewayAttachedProxySettingsPolicyCount is the number of relevant ProxySettingsPolicies
attached at the Gateway level. */
long? GatewayAttachedProxySettingsPolicyCount = null;

/** RouteAttachedProxySettingsPolicyCount is the number of relevant ProxySettingsPolicies
attached at the Route level. */
long? RouteAttachedProxySettingsPolicyCount = null;

/** NginxPodCount is the total number of Nginx data plane Pods. */
long? NginxPodCount = null;

Expand Down
6 changes: 6 additions & 0 deletions internal/controller/telemetry/data_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ func TestDataAttributes(t *testing.T) {
SnippetsFilterCount: 13,
UpstreamSettingsPolicyCount: 14,
GatewayAttachedNpCount: 15,
GatewayAttachedProxySettingsPolicyCount: 17,
RouteAttachedProxySettingsPolicyCount: 18,
},
SnippetsFiltersDirectives: []string{"main-three-count", "http-two-count", "server-one-count"},
SnippetsFiltersDirectivesCount: []int64{3, 2, 1},
Expand Down Expand Up @@ -84,6 +86,8 @@ func TestDataAttributes(t *testing.T) {
attribute.Int64("SnippetsFilterCount", 13),
attribute.Int64("UpstreamSettingsPolicyCount", 14),
attribute.Int64("GatewayAttachedNpCount", 15),
attribute.Int64("GatewayAttachedProxySettingsPolicyCount", 17),
attribute.Int64("RouteAttachedProxySettingsPolicyCount", 18),
attribute.Int64("NginxPodCount", 3),
attribute.Int64("ControlPlanePodCount", 3),
attribute.Bool("NginxOneConnectionEnabled", true),
Expand Down Expand Up @@ -132,6 +136,8 @@ func TestDataAttributesWithEmptyData(t *testing.T) {
attribute.Int64("SnippetsFilterCount", 0),
attribute.Int64("UpstreamSettingsPolicyCount", 0),
attribute.Int64("GatewayAttachedNpCount", 0),
attribute.Int64("GatewayAttachedProxySettingsPolicyCount", 0),
attribute.Int64("RouteAttachedProxySettingsPolicyCount", 0),
attribute.Int64("NginxPodCount", 0),
attribute.Int64("ControlPlanePodCount", 0),
attribute.Bool("NginxOneConnectionEnabled", false),
Expand Down
2 changes: 2 additions & 0 deletions internal/controller/telemetry/ngfresourcecounts_attributes_generated.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ func (d *NGFResourceCounts) Attributes() []attribute.KeyValue {
attrs = append(attrs, attribute.Int64("SnippetsFilterCount", d.SnippetsFilterCount))
attrs = append(attrs, attribute.Int64("UpstreamSettingsPolicyCount", d.UpstreamSettingsPolicyCount))
attrs = append(attrs, attribute.Int64("GatewayAttachedNpCount", d.GatewayAttachedNpCount))
attrs = append(attrs, attribute.Int64("GatewayAttachedProxySettingsPolicyCount", d.GatewayAttachedProxySettingsPolicyCount))
attrs = append(attrs, attribute.Int64("RouteAttachedProxySettingsPolicyCount", d.RouteAttachedProxySettingsPolicyCount))

return attrs
}
Expand Down
2 changes: 2 additions & 0 deletions tests/suite/telemetry_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ var _ = Describe("Telemetry test with OTel collector", Label("telemetry"), func(
"SnippetsFilterCount: Int(0)",
"UpstreamSettingsPolicyCount: Int(0)",
"GatewayAttachedNpCount: Int(0)",
"GatewayAttachedProxySettingsPolicyCount: Int(0)",
"RouteAttachedProxySettingsPolicyCount: Int(0)",
"NginxPodCount: Int(0)",
"ControlPlanePodCount: Int(1)",
"NginxOneConnectionEnabled: Bool(false)",
Expand Down
Loading