Skip to content

OSDOCS-16874-5: CQA for SCALE-4 Low-Latency/CNF Debugging and Related #107776

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dfitzmau wants to merge 1 commit into
base: main
Choose a base branch
from
Open

OSDOCS-16874-5: CQA for SCALE-4 Low-Latency/CNF Debugging and Related #107776

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions modules/accessing-an-example-cluster-node-tuning-operator-specification.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,23 @@
[id="accessing-an-example-node-tuning-operator-specification_{context}"]
= Accessing an example Node Tuning Operator specification

Use this process to access an example Node Tuning Operator specification.
[role="_abstract"]
To understand how to correctly format your tuning parameters, access an example Node Tuning Operator specification. By reviewing this template, you can properly configure node-level settings for your workloads.

For the example Node Tuning Operator specification provided in the procedure, the default CR is meant for delivering standard node-level tuning for the {product-title} platform and it can only be modified to set the Operator Management state. Any other custom changes to the default CR will be overwritten by the Operator. For custom tuning, create your own Tuned CRs. Newly created CRs will be combined with the default CR and custom tuning applied to {product-title} nodes based on node or pod labels and profile priorities.

[WARNING]
====
While in certain situations the support for pod labels can be a convenient way of automatically delivering required tuning, this practice is discouraged and strongly advised against, especially in large-scale clusters. The default Tuned CR ships without pod label matching. If a custom profile is created with pod label matching, then the functionality will be enabled at that time. The pod label functionality will be deprecated in future versions of the Node Tuning Operator.
====

.Procedure

* Run the following command to access an example Node Tuning Operator specification:
* Run the following command to access an example Node Tuning Operator specification:
+
[source,terminal]
----
oc get tuned.tuned.openshift.io/default -o yaml -n openshift-cluster-node-tuning-operator
----

The default CR is meant for delivering standard node-level tuning for the {product-title} platform and it can only be modified to set the Operator Management state. Any other custom changes to the default CR will be overwritten by the Operator. For custom tuning, create your own Tuned CRs. Newly created CRs will be combined with the default CR and custom tuning applied to {product-title} nodes based on node or pod labels and profile priorities.

[WARNING]
====
While in certain situations the support for pod labels can be a convenient way of automatically delivering required tuning, this practice is discouraged and strongly advised against, especially in large-scale clusters. The default Tuned CR ships without pod label matching. If a custom profile is created with pod label matching, then the functionality will be enabled at that time. The pod label functionality will be deprecated in future versions of the Node Tuning Operator.
====
9 changes: 6 additions & 3 deletions modules/cluster-node-tuning-operator-default-profiles-set.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@
[id="custom-tuning-default-profiles-set_{context}"]
= Default profiles set on a cluster

The following are the default profiles set on a cluster.
[role="_abstract"]
To understand the baseline configurations automatically applied to your environment, review the default profiles set on a cluster. By analyzing these built-in settings, you can determine if additional node tuning is necessary for your specific workloads.

The following configuration example shows default profiles set on a cluster:

[source,yaml]
----
Expand All @@ -32,10 +35,10 @@ spec:
- label: node-role.kubernetes.io/infra
- profile: openshift-node
priority: 40
# ...
----

Starting with {product-title} 4.9, all OpenShift TuneD profiles are shipped with
the TuneD package. You can use the `oc exec` command to view the contents of these profiles:
Starting with {product-title} 4.9, all {product-title} TuneD profiles are shipped with the TuneD package. You can use the following `oc exec` command to view the contents of these profiles:

[source,terminal]
----
Expand Down
35 changes: 19 additions & 16 deletions modules/cluster-node-tuning-operator-verify-profiles.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,20 @@

:_mod-docs-content-type: PROCEDURE
[id="verifying-tuned-profiles-are-applied_{context}"]
= Verifying that the TuneD profiles are applied
= Verifying that the TuneD profiles are applied

Verify the TuneD profiles that are applied to your cluster node.
[role="_abstract"]
To confirm that your node-level tuning configurations are active, verify the TuneD profiles applied to your cluster node. Checking these settings ensures that your system is correctly optimized for your specific workloads.

.Procedure

. Verify the TuneD profiles that are applied to your cluster node by entering the following command:
+
[source,terminal]
----
$ oc get profile.tuned.openshift.io -n openshift-cluster-node-tuning-operator
----

+
.Example output
[source,terminal]
----
Expand All @@ -23,27 +28,25 @@ master-2 openshift-control-plane True False 6h33m
worker-a openshift-node True False 6h28m
worker-b openshift-node True False 6h28m
----
+
** `NAME`: Specifies the name of the Profile object. There is one Profile object per node and their names match.
** `TUNED`: Specifies the name of the desired TuneD profile to apply.
** `APPLIED`: Set as `True` if the TuneD daemon applied the desired profile. Supported values include `True`, `False`, and `Unknown`.
** `DEGRADED`: Set as `True` if any errors were reported during application of the TuneD profile. Supported values include `True`, `False`, and `Unknown`.
** `AGE`: Specifies the time elapsed since the creation of Profile object.

* `NAME`: Name of the Profile object. There is one Profile object per node and their names match.
* `TUNED`: Name of the desired TuneD profile to apply.
* `APPLIED`: `True` if the TuneD daemon applied the desired profile. (`True/False/Unknown`).
* `DEGRADED`: `True` if any errors were reported during application of the TuneD profile (`True/False/Unknown`).
* `AGE`: Time elapsed since the creation of Profile object.

The `ClusterOperator/node-tuning` object also contains useful information about the Operator and its node agents' health. For example, Operator misconfiguration is reported by `ClusterOperator/node-tuning` status messages.

To get status information about the `ClusterOperator/node-tuning` object, run the following command:

. To get status information about the `ClusterOperator/node-tuning` object, run the following command. The `ClusterOperator/node-tuning` object also contains useful information about the Operator and the health status of node agents. For example, Operator misconfiguration is reported by `ClusterOperator/node-tuning` status messages.
+
[source,terminal]
----
$ oc get co/node-tuning -n openshift-cluster-node-tuning-operator
----

+
.Example output
[source,terminal,subs="attributes+"]
----
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
node-tuning {product-version}.1 True False True 60m 1/5 Profiles with bootcmdline conflict
----

If either the `ClusterOperator/node-tuning` or a profile object's status is `DEGRADED`, additional information is provided in the Operator or operand logs.
+
If either the `ClusterOperator/node-tuning` or the status of a profile object is `DEGRADED`, additional information is provided in the Operator or operand logs.
106 changes: 69 additions & 37 deletions modules/custom-tuning-specification.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@ endif::[]
[id="custom-tuning-specification_{context}"]
= Custom tuning specification

[role="_abstract"]
To define custom node-level configurations for your workloads, review the custom tuning specification. By understanding the structure of the custom resource (CR) for the Operator, you can correctly format your TuneD profiles and selection logic.

The custom resource (CR) for the Operator has two major sections. The first section, `profile:`, is a list of TuneD profiles and their names. The second, `recommend:`, defines the profile selection logic.

Multiple custom tuning specifications can co-exist as multiple CRs in the Operator's namespace. The existence of new CRs or the deletion of old CRs is detected by the Operator. All existing custom tuning specifications are merged and appropriate objects for the containerized TuneD daemons are updated.
Multiple custom tuning specifications can co-exist as multiple CRs in the namespace of the Operator. The existence of new CRs or the deletion of old CRs is detected by the Operator. All existing custom tuning specifications are merged and appropriate objects for the containerized TuneD daemons are updated.

*Management state*

Expand Down Expand Up @@ -102,26 +105,39 @@ The individual items of the list:
ifndef::rosa-hcp-tuning[]
[source,yaml]
----
- machineConfigLabels: <1>
<mcLabels> <2>
match: <3>
<match> <4>
priority: <priority> <5>
profile: <tuned_profile_name> <6>
operand: <7>
debug: <bool> <8>
- machineConfigLabels:
<mcLabels>
match:
<match>
priority: <priority>
profile: <tuned_profile_name>
operand:
debug: <bool>
tunedConfig:
reapply_sysctl: <bool> <9>
reapply_sysctl: <bool>
----
<1> Optional.
<2> A dictionary of key/value `MachineConfig` labels. The keys must be unique.
<3> If omitted, profile match is assumed unless a profile with a higher priority matches first or `machineConfigLabels` is set.
<4> An optional list.
<5> Profile ordering priority. Lower numbers mean higher priority (`0` is the highest priority).
<6> A TuneD profile to apply on a match. For example `tuned_profile_1`.
<7> Optional operand configuration.
<8> Turn debugging on or off for the TuneD daemon. Options are `true` for on or `false` for off. The default is `false`.
<9> Turn `reapply_sysctl` functionality on or off for the TuneD daemon. Options are `true` for on and `false` for off.
+
where:
+
--
`machineConfigLabels`:: Optional parameter.

`<mcLabels>`:: Specifies a dictionary of key/value `MachineConfig` labels. The keys must be unique.

`match`:: If omitted, profile match is assumed unless a profile with a higher priority matches first or `machineConfigLabels` is set.

`<match>`:: An optional list.

`priority`:: Specifies profile ordering priority. Lower numbers mean higher priority (`0` is the highest priority).

`<tuned_profile_name>`:: Specifies a TuneD profile to apply on a match. For example `tuned_profile_1`.

`operand`:: Optional operand configuration.

`debug`:: Turn debugging on or off for the TuneD daemon. Options are `true` for on or `false` for off. The default is `false`.

`tunedConfig.reapply_sysctl`:: Turn `reapply_sysctl` functionality on or off for the TuneD daemon. Options are `true` for on and `false` for off.
--
endif::rosa-hcp-tuning[]
ifdef::rosa-hcp-tuning[]
[source,json]
Expand All @@ -134,49 +150,65 @@ ifdef::rosa-hcp-tuning[]
],
"recommend": [
{
"profile": <tuned_profile_name>, <1>
"priority":{ <priority>, <2>
"profile": <tuned_profile_name>,
"priority":{ <priority>,
},
"match": [ <3>
"match": [
{
"label": <label_information> <4>
"label": <label_information>
},
]
},
]
}
----
<1> A TuneD profile to apply on a match. For example `tuned_profile_1`.
<2> Profile ordering priority. Lower numbers mean higher priority (`0` is the highest priority).
<3> If omitted, profile match is assumed unless a profile with a higher priority matches first.
<4> The label for the profile matched items.
+
where:
+
--
`profile`:: Specifies a TuneD profile to apply on a match. For example `tuned_profile_1`.

`priority`:: Specifies profile ordering priority. Lower numbers mean higher priority (`0` is the highest priority).

`match`:: If omitted, profile match is assumed unless a profile with a higher priority matches first.

`label`:: Specifies the label for the profile matched items.
--
endif::[]

`<match>` is an optional list recursively defined as follows:

ifndef::rosa-hcp-tuning[]
[source,yaml]
----
- label: <label_name> <1>
value: <label_value> <2>
type: <label_type> <3>
<match> <4>
- label: <label_name>
value: <label_value>
type: <label_type>
<match>
----
<1> Node or pod label name.
<2> Optional node or pod label value. If omitted, the presence of `<label_name>` is enough to match.
<3> Optional object type (`node` or `pod`). If omitted, `node` is assumed.
<4> An optional `<match>` list.
+
where:
+
--
`label`:: Specifies the node or pod label name.

`value`:: Specifies an optional node or pod label value. If omitted, the presence of `<label_name>` is enough to match.

`type`:: Specifies an optional object type, such as `node` or `pod`. If omitted, `node` is assumed.

`<match>` An optional `<match>` list.
--
endif::rosa-hcp-tuning[]
ifdef::rosa-hcp-tuning[]
[source,yaml]
----
"match": [
{
"label": <1>
"label":
},
]
----
<1> Node or pod label name.
** `label` Node or pod label name.
endif::[]

If `<match>` is not omitted, all nested `<match>` sections must also evaluate to `true`. Otherwise, `false` is assumed and the profile with the respective `<match>` section will not be applied or recommended. Therefore, the nesting (child `<match>` sections) works as logical AND operator. Conversely, if any item of the `<match>` list matches, the entire `<match>` list evaluates to `true`. Therefore, the list acts as logical OR operator.
Expand Down
32 changes: 16 additions & 16 deletions modules/node-observability-create-custom-resource.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@
[id="creating-node-observability-custom-resource_{context}"]
= Creating the Node Observability custom resource

You must create and run the `NodeObservability` custom resource (CR) before you run the profiling query. When you run the `NodeObservability` CR, it creates the necessary machine config and machine config pool CRs to enable the CRI-O profiling on the worker nodes matching the `nodeSelector`.
[role="_abstract"]
You must create and run the `NodeObservability` custom resource (CR) before you run the profiling query. When you run the `NodeObservability` CR, the CR creates the necessary machine config and machine config pool CRs to enable the CRI-O profiling on the compute nodes matching the `nodeSelector`.

[IMPORTANT]
====
If CRI-O profiling is not enabled on the worker nodes, the `NodeObservabilityMachineConfig` resource gets created. Worker nodes matching the `nodeSelector` specified in `NodeObservability` CR restarts. This might take 10 or more minutes to complete.
If CRI-O profiling is not enabled on the compute nodes, the `NodeObservabilityMachineConfig` resource gets created. Compute nodes matching the `nodeSelector` specified in `NodeObservability` CR restarts. This might take 10 or more minutes to complete.
====

[NOTE]
Expand All @@ -21,8 +22,9 @@ Kubelet profiling is enabled by default.
The CRI-O unix socket of the node is mounted on the agent pod, which allows the agent to communicate with CRI-O to run the pprof request. Similarly, the `kubelet-serving-ca` certificate chain is mounted on the agent pod, which allows secure communication between the agent and node's kubelet endpoint.

.Prerequisites

* You have installed the Node Observability Operator.
* You have installed the OpenShift CLI (oc).
* You have installed the {oc-first}.
* You have access to the cluster with `cluster-admin` privileges.

.Procedure
Expand All @@ -45,25 +47,25 @@ $ oc project node-observability-operator
+
[source,yaml]
----
apiVersion: nodeobservability.olm.openshift.io/v1alpha2
kind: NodeObservability
metadata:
name: cluster <1>
spec:
nodeSelector:
kubernetes.io/hostname: <node_hostname> <2>
type: crio-kubelet
apiVersion: nodeobservability.olm.openshift.io/v1alpha2
kind: NodeObservability
metadata:
name: cluster
spec:
nodeSelector:
kubernetes.io/hostname: <node_hostname>
type: crio-kubelet
----
<1> You must specify the name as `cluster` because there should be only one `NodeObservability` CR per cluster.
<2> Specify the nodes on which the Node Observability agent must be deployed.
+
** `metadata.name`: Specifies the name as `cluster` because there should be only one `NodeObservability` CR per cluster.
** `spec.nodeSelector.kubernetes.io/hostname`: Specifies the nodes on which the Node Observability agent must be deployed.

. Run the `NodeObservability` CR:
+
[source,terminal]
----
oc apply -f nodeobservability.yaml
----

+
.Example output
[source,terminal]
Expand All @@ -77,7 +79,6 @@ nodeobservability.olm.openshift.io/cluster created
----
$ oc get nob/cluster -o yaml | yq '.status.conditions'
----

+
.Example output
[source,terminal]
Expand All @@ -91,6 +92,5 @@ conditions:
status: "True"
type: Ready
----

+
`NodeObservability` CR run is completed when the reason is `Ready` and the status is `True`.
3 changes: 3 additions & 0 deletions modules/node-observability-high-level-workflow.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
[id="workflow-node-observability-operator_{context}"]
= Workflow of the Node Observability Operator

[role="_abstract"]
To systematically query and analyze profiling data, follow the workflow for the Node Observability Operator. By understanding this process, you can collect metrics and troubleshoot performance issues on your compute nodes.

The following workflow outlines on how to query the profiling data using the Node Observability Operator:

. Install the Node Observability Operator in the {product-title} cluster.
Expand Down
Loading