Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ A lightweight Kubernetes cloud controller for bare-metal and on-premise clusters
- **Configurable Targets**: Separate configuration for internal and external IP detection
- **Non-Destructive Updates**: Preserves existing addresses (Hostname, InternalIP from kubelet), updates only managed fields
- **Taint Removal**: Automatically removes `node.cloudprovider.kubernetes.io/uninitialized` taint
- **Azure Provider ID Support**: Automatic detection and setup of `spec.providerID` for Azure nodes via IMDS

**Node Lifecycle Controller:**
- **Autoscaler Integration**: Watches nodes tainted with `ToBeDeletedByClusterAutoscaler:NoSchedule` by default
Expand Down Expand Up @@ -179,6 +180,7 @@ The following command-line flags are available:
| `--remove-taint` | Remove node.cloudprovider.kubernetes.io/uninitialized taint | `true` | No |
| `--reconcile-interval` | Interval between reconciliation loops | `10s` | No |
| `--run-once` | Run once and exit instead of running in a loop | `false` | No |
| `--enable-azure-provider-id` | Enable Azure provider ID detection via IMDS | `false` | No |
| `--kubeconfig` | Path to kubeconfig file (for local testing only) | In-cluster config | No |
| `--v` | Log level (0-5) | `0` | No |

Expand Down Expand Up @@ -224,6 +226,18 @@ Result:
}
```

#### Azure Provider ID (for cluster-autoscaler)

```yaml
args:
- --node-name=$(NODE_NAME)
- --external-ip-target=8.8.8.8
- --enable-azure-provider-id=true
- --reconcile-interval=10s
```

This configuration enables automatic detection and setup of `spec.providerID` on Azure nodes. The controller queries Azure Instance Metadata Service (IMDS) to retrieve the resource ID and sets it as the node's provider ID. This is required for Azure cluster-autoscaler to properly track and manage nodes.

After updating the DaemonSet args, restart the pods:

```bash
Expand Down
13 changes: 13 additions & 0 deletions charts/local-ccm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Local Cloud Controller Manager for Kubernetes - automatically detects and manage
- Automatic node IP address detection using routing table
- Support for both internal and external IP detection
- Automatic removal of cloud provider initialization taint
- Azure provider ID detection for cluster-autoscaler integration
- Minimal resource footprint
- Runs as DaemonSet on all nodes

Expand Down Expand Up @@ -50,6 +51,17 @@ helm install local-ccm ./charts/local-ccm \
--set ipDetection.internalIPTarget=10.0.0.1
```

### Azure Configuration

For Azure environments with cluster-autoscaler:

```yaml
azure:
enableProviderID: true
```

This enables automatic detection and setting of `spec.providerID` on Azure nodes, which is required for cluster-autoscaler to properly track and manage nodes.

## Configuration

| Parameter | Description | Default |
Expand All @@ -64,6 +76,7 @@ helm install local-ccm ./charts/local-ccm \
| `controller.removeTaint` | Remove uninitialized taint | `true` |
| `controller.reconcileInterval` | Reconciliation interval | `10s` |
| `controller.verbosity` | Log verbosity level (0-5) | `2` |
| `azure.enableProviderID` | Enable Azure provider ID detection via IMDS | `false` |
| `resources.requests.cpu` | CPU resource requests | `10m` |
| `resources.requests.memory` | Memory resource requests | `32Mi` |
| `resources.limits.cpu` | CPU resource limits | `100m` |
Expand Down
3 changes: 3 additions & 0 deletions charts/local-ccm/templates/daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ spec:
- --remove-taint={{ .Values.controller.removeTaint }}
- --reconcile-interval={{ .Values.controller.reconcileInterval }}
- --v={{ .Values.controller.verbosity }}
{{- if .Values.azure.enableProviderID }}
- --enable-azure-provider-id=true
{{- end }}
env:
- name: NODE_NAME
valueFrom:
Expand Down
6 changes: 6 additions & 0 deletions charts/local-ccm/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ controller:
reconcileInterval: 10s
# Verbosity level (0-5)
verbosity: 2
# Azure-specific configuration
azure:
# Enable Azure provider ID detection via Instance Metadata Service (IMDS)
# When enabled, the controller will automatically detect and set spec.providerID
# on Azure nodes, which is required for cluster-autoscaler to properly track nodes
enableProviderID: false
# Pod resources
resources:
requests:
Expand Down
35 changes: 26 additions & 9 deletions cmd/local-ccm/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,14 @@ import (
)

var (
nodeName string
kubeconfig string
internalIPTarget string
externalIPTarget string
runOnce bool
removeTaint bool
reconcileInterval time.Duration
nodeName string
kubeconfig string
internalIPTarget string
externalIPTarget string
runOnce bool
removeTaint bool
reconcileInterval time.Duration
enableAzureProviderID bool
)

func init() {
Expand All @@ -51,6 +52,7 @@ func init() {
flag.BoolVar(&runOnce, "run-once", false, "Run once and exit instead of running in a loop")
flag.BoolVar(&removeTaint, "remove-taint", true, "Remove node.cloudprovider.kubernetes.io/uninitialized taint")
flag.DurationVar(&reconcileInterval, "reconcile-interval", 10*time.Second, "Interval between reconciliation loops")
flag.BoolVar(&enableAzureProviderID, "enable-azure-provider-id", false, "Enable Azure provider ID detection via IMDS")

klog.InitFlags(nil)
}
Expand All @@ -63,8 +65,8 @@ func main() {
}

klog.Infof("Starting local-ccm for node %s", nodeName)
klog.V(2).Infof("Configuration: internalIPTarget=%q externalIPTarget=%q",
internalIPTarget, externalIPTarget)
klog.V(2).Infof("Configuration: internalIPTarget=%q externalIPTarget=%q azureProviderID=%v",
internalIPTarget, externalIPTarget, enableAzureProviderID)

// Create Kubernetes client
k8sClient, err := createKubernetesClient(kubeconfig)
Expand Down Expand Up @@ -158,6 +160,21 @@ func reconcile(ctx context.Context, nodeUpdater *node.Updater) error {
}
}

// Set Azure provider ID if enabled
if enableAzureProviderID {
providerID, err := detector.DetectAzureProviderID(ctx)
if err != nil {
return fmt.Errorf("failed to detect Azure provider ID: %w", err)
}
if providerID != "" {
if err := nodeUpdater.UpdateProviderID(ctx, providerID); err != nil {
return fmt.Errorf("failed to update provider ID: %w", err)
}
} else {
klog.V(3).Info("Azure provider ID not detected (not running in Azure)")
}
}
Comment on lines +164 to +176
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The Azure provider ID is immutable for the lifetime of a node. Currently, the controller queries the Azure Instance Metadata Service (IMDS) and potentially the Kubernetes API server every reconciliation loop (default 10s), even after the provider ID has been successfully set.

Adding a check to see if currentNode.Spec.ProviderID is already populated will significantly reduce unnecessary network traffic and API pressure.

Suggested change
if enableAzureProviderID {
providerID, err := detector.DetectAzureProviderID(ctx)
if err != nil {
return fmt.Errorf("failed to detect Azure provider ID: %w", err)
}
if providerID != "" {
if err := nodeUpdater.UpdateProviderID(ctx, providerID); err != nil {
return fmt.Errorf("failed to update provider ID: %w", err)
}
} else {
klog.V(3).Info("Azure provider ID not detected (not running in Azure)")
}
}
// Set Azure provider ID if enabled and not already set
if enableAzureProviderID && currentNode.Spec.ProviderID == "" {
providerID, err := detector.DetectAzureProviderID(ctx)
if err != nil {
return fmt.Errorf("failed to detect Azure provider ID: %w", err)
}
if providerID != "" {
if err := nodeUpdater.UpdateProviderID(ctx, providerID); err != nil {
return fmt.Errorf("failed to update provider ID: %w", err)
}
} else {
klog.V(3).Info("Azure provider ID not detected (not running in Azure)")
}
}


// Remove taint if requested
if removeTaint {
if err := nodeUpdater.RemoveTaint(ctx); err != nil {
Expand Down
92 changes: 92 additions & 0 deletions pkg/detector/azure_provider.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/*
Copyright 2025 The local-ccm Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package detector

import (
"context"
"fmt"
"io"
"net/http"
"strings"
"time"

"k8s.io/klog/v2"
)

const (
// AzureIMDSURL is the Azure Instance Metadata Service endpoint
AzureIMDSURL = "http://169.254.169.254/metadata/instance/compute/resourceId?api-version=2021-02-01&format=text"

// AzureIMDSTimeout is the timeout for IMDS requests
AzureIMDSTimeout = 2 * time.Second
)

// DetectAzureProviderID detects Azure provider ID from Instance Metadata Service
// Returns empty string if not running in Azure environment
func DetectAzureProviderID(ctx context.Context) (string, error) {
klog.V(3).Info("Attempting to detect Azure provider ID from IMDS")

// Create HTTP client with timeout
client := &http.Client{
Timeout: AzureIMDSTimeout,
}
Comment on lines +44 to +46
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, locate and examine the azure_provider.go file
find . -name "azure_provider.go" -type f

Repository: cozystack/local-ccm

Length of output: 95


🏁 Script executed:

# Read the relevant section of the file
cat -n pkg/detector/azure_provider.go | head -80

Repository: cozystack/local-ccm

Length of output: 2891


🏁 Script executed:

# Check if there are other HTTP clients or similar patterns in the detector package
rg -n "http\.Client\{" pkg/detector/ -A 2

Repository: cozystack/local-ccm

Length of output: 221


🏁 Script executed:

# Verify if there are other cloud provider detectors that might have similar patterns
ls -la pkg/detector/

Repository: cozystack/local-ccm

Length of output: 304


IMDS calls must explicitly bypass HTTP proxies to ensure detection works in proxied environments

The default http.Client at lines 44-46 uses DefaultTransport, which respects HTTP_PROXY/HTTPS_PROXY environment variables. In proxied clusters or corporate networks, this routes the IMDS call (169.254.169.254) through the proxy, breaking Azure provider detection. IMDS is a link-local metadata service that must communicate directly, never through proxies.

Proposed fix
-	client := &http.Client{
-		Timeout: AzureIMDSTimeout,
-	}
+	transport := http.DefaultTransport.(*http.Transport).Clone()
+	transport.Proxy = nil
+	client := &http.Client{
+		Timeout:   AzureIMDSTimeout,
+		Transport: transport,
+	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
client := &http.Client{
Timeout: AzureIMDSTimeout,
}
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.Proxy = nil
client := &http.Client{
Timeout: AzureIMDSTimeout,
Transport: transport,
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/detector/azure_provider.go` around lines 44 - 46, The http.Client
currently created with Timeout: AzureIMDSTimeout uses the default transport
which respects HTTP_PROXY/HTTPS_PROXY; update the client to use an
http.Transport with Proxy set to nil so IMDS calls bypass proxies. Replace the
simple client creation (client := &http.Client{Timeout: AzureIMDSTimeout}) by
constructing a transport (e.g., clone or new http.Transport) with Proxy: nil and
any required defaults, assign it to client.Transport and keep Timeout:
AzureIMDSTimeout to ensure direct link-local access to 169.254.169.254 for the
Azure metadata detection.


// Create request with context
req, err := http.NewRequestWithContext(ctx, http.MethodGet, AzureIMDSURL, nil)
if err != nil {
return "", fmt.Errorf("failed to create IMDS request: %w", err)
}

// Azure IMDS requires Metadata header
req.Header.Set("Metadata", "true")

klog.V(4).Infof("Sending request to Azure IMDS: %s", AzureIMDSURL)

// Execute request
resp, err := client.Do(req)
if err != nil {
// This is expected when not running in Azure
klog.V(3).Infof("Azure IMDS not available (likely not running in Azure): %v", err)
return "", nil
}
Comment on lines +38 to +65
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Creating a new http.Client on every call to DetectAzureProviderID prevents connection reuse and increases overhead. It is better practice to reuse a single client instance across requests to benefit from connection pooling.

var azureIMDSClient = &http.Client{
	Timeout: AzureIMDSTimeout,
}

// DetectAzureProviderID detects Azure provider ID from Instance Metadata Service
// Returns empty string if not running in Azure environment
func DetectAzureProviderID(ctx context.Context) (string, error) {
	klog.V(3).Info("Attempting to detect Azure provider ID from IMDS")

	// Create request with context
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, AzureIMDSURL, nil)
	if err != nil {
		return "", fmt.Errorf("failed to create IMDS request: %w", err)
	}

	// Azure IMDS requires Metadata header
	req.Header.Set("Metadata", "true")

	klog.V(4).Infof("Sending request to Azure IMDS: %s", AzureIMDSURL)

	// Execute request
	resp, err := azureIMDSClient.Do(req)
	if err != nil {
		// This is expected when not running in Azure
		klog.V(3).Infof("Azure IMDS not available (likely not running in Azure): %v", err)
		return "", nil
	}

defer resp.Body.Close()

// Check response status
if resp.StatusCode != http.StatusOK {
klog.V(2).Infof("Azure IMDS returned non-OK status: %d", resp.StatusCode)
return "", nil
}

// Read response body
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("failed to read IMDS response: %w", err)
}

resourceID := strings.TrimSpace(string(body))
if resourceID == "" {
klog.V(2).Info("Azure IMDS returned empty resource ID")
return "", nil
}

// Format as Kubernetes provider ID
providerID := fmt.Sprintf("azure://%s", resourceID)

klog.V(2).Infof("Detected Azure provider ID: %s", providerID)

return providerID, nil
}
54 changes: 54 additions & 0 deletions pkg/node/updater.go
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,57 @@ func (u *Updater) RemoveTaint(ctx context.Context) error {
func (u *Updater) GetNode(ctx context.Context) (*v1.Node, error) {
return u.client.CoreV1().Nodes().Get(ctx, u.nodeName, metav1.GetOptions{})
}

// UpdateProviderID updates the node's spec.providerID
func (u *Updater) UpdateProviderID(ctx context.Context, providerID string) error {
klog.V(2).Infof("Updating providerID for node %s: %s", u.nodeName, providerID)

// Get current node to check if update is needed
node, err := u.GetNode(ctx)
if err != nil {
return fmt.Errorf("failed to get node: %w", err)
}

// Check if providerID is already set correctly
if node.Spec.ProviderID == providerID {
klog.V(3).Infof("ProviderID already set correctly on node %s, skipping update", u.nodeName)
return nil
}

// Log if we're overwriting an existing providerID
if node.Spec.ProviderID != "" {
klog.Warningf("Overwriting existing providerID on node %s: %s -> %s",
u.nodeName, node.Spec.ProviderID, providerID)
}

// Create JSON patch for providerID
patch := []map[string]interface{}{
{
"op": "replace",
"path": "/spec/providerID",
"value": providerID,
},
Comment on lines +174 to +179
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# First, locate and examine the file
find . -name "updater.go" -path "*/node/*" | head -5

Repository: cozystack/local-ccm

Length of output: 84


🏁 Script executed:

# Read the relevant section of the file
cat -n pkg/node/updater.go | sed -n '150,210p'

Repository: cozystack/local-ccm

Length of output: 1931


🏁 Script executed:

# Search for how this patch is applied
rg -n "JSONPatchType\|json\.Marshal\|patch\s*=" pkg/node/updater.go -A 3 -B 3

Repository: cozystack/local-ccm

Length of output: 45


🏁 Script executed:

# Check how the patch is used and if there are tests
rg -n "UpdateProviderID\|providerID" pkg/node/ -A 2 -B 2 | head -80

Repository: cozystack/local-ccm

Length of output: 45


🏁 Script executed:

# Check if there are tests for this method
find . -name "*test.go" -path "*/node/*" | xargs rg -l "UpdateProviderID" 2>/dev/null

Repository: cozystack/local-ccm

Length of output: 104


🏁 Script executed:

# Search for any related issue tracking or comments
rg -n "providerID.*patch\|patch.*providerID\|replace.*add" --type go -i

Repository: cozystack/local-ccm

Length of output: 45


🏁 Script executed:

# Let's check the test file that was found
cat -n pkg/node/updater.go

Repository: cozystack/local-ccm

Length of output: 6772


🏁 Script executed:

# Search for any Node object creation or initialization
rg -n "corev1.Node\|Node{" pkg/node/ -A 3

Repository: cozystack/local-ccm

Length of output: 162


🏁 Script executed:

# Check if there's a go.mod or similar to understand Kubernetes version
cat go.mod | grep kubernetes

Repository: cozystack/local-ccm

Length of output: 45


🏁 Script executed:

# Search for Kubernetes version in go.mod or go.sum
find . -name "go.mod" -o -name "go.sum" | xargs grep -h "k8s.io" | head -20

Repository: cozystack/local-ccm

Length of output: 1363


🏁 Script executed:

# Now let me verify the actual RFC 6902 JSON Patch behavior with an example
# Let's check the Kubernetes documentation approach
web_search
Does Kubernetes JSON Patch "replace" operation fail when field is missing/unset?

Repository: cozystack/local-ccm

Length of output: 156


🌐 Web query:

RFC 6902 JSON Patch "replace" operation behavior when target path does not exist

💡 Result:

According to RFC 6902, the "replace" operation in JSON Patch fails if the target path does not exist. The specification states: "The target location MUST exist for the operation to be successful." If the target location does not exist, the operation is not successful, and evaluation of the JSON Patch document SHOULD terminate, with the entire patch application deemed unsuccessful (Section 5: Error Handling).

Citations:


🌐 Web query:

Kubernetes JSON Patch replace operation optional fields missing null unset

💡 Result:

In Kubernetes JSON Patch (RFC 6902, used with kubectl patch --type=json), the replace operation requires the target path to exist in the resource. If the path (or any prefix) is missing, the operation fails with an error like "jsonpatch replace operation does not apply: doc is missing path". Key distinctions for optional/missing/unset/null fields: - replace: Replaces existing value at path. Path MUST exist (RFC 6902: "The target location MUST exist for the operation to be successful"). Fails if missing. Error example: "doc is missing key: /spec/template/spec/containers/0/image". - add: Adds value if path missing, or replaces if exists. Use for optional fields that might be unset. - remove: Removes value at path. Path must exist. - null values: Setting to null via replace keeps null (not removed). Use remove to delete fields. Kubernetes omits omitempty nulls in JSON, but literal null differs from absence. For missing optional fields, use add first to create parents (no auto-creation of nested objects), then replace/add child. Example from kustomize/CRDs: add "/spec/template/metadata/annotations" as {}, then add/replace sub-keys. JSON Patch differs from JSON Merge Patch (--type=merge): merge auto-creates missing paths/objects, null removes fields, but replaces lists entirely. Use JSON Patch for precise ops on existing structures; fails on missing paths to prevent unintended adds.

Citations:


replace cannot succeed when spec.providerID is initially missing

At line 176, using JSON Patch op replace on /spec/providerID fails on nodes where the field is unset, which is the exact first-write scenario this feature targets. RFC 6902 requires the target path to exist for replace to succeed. Use add instead to both create and update the field.

Proposed fix
  patch := []map[string]interface{}{
  	{
-		"op":    "replace",
+		"op":    "add",
  		"path":  "/spec/providerID",
  		"value": providerID,
  	},
  }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
patch := []map[string]interface{}{
{
"op": "replace",
"path": "/spec/providerID",
"value": providerID,
},
patch := []map[string]interface{}{
{
"op": "add",
"path": "/spec/providerID",
"value": providerID,
},
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/node/updater.go` around lines 174 - 179, The JSON Patch uses the
"replace" op on "/spec/providerID" which fails when spec.providerID is absent;
update the patch construction in pkg/node/updater.go (the variable patch that
includes the map with "path": "/spec/providerID" and "value": providerID) to use
the "add" operation instead of "replace" so the field will be created on
first-write and updated on subsequent writes.

}

patchBytes, err := json.Marshal(patch)
if err != nil {
return fmt.Errorf("failed to marshal patch: %w", err)
}

klog.V(4).Infof("Applying providerID patch to node %s: %s", u.nodeName, string(patchBytes))

// Apply patch
_, err = u.client.CoreV1().Nodes().Patch(
ctx,
u.nodeName,
types.JSONPatchType,
patchBytes,
metav1.PatchOptions{},
)
if err != nil {
return fmt.Errorf("failed to patch node providerID: %w", err)
}

klog.Infof("Successfully updated providerID for node %s to %s", u.nodeName, providerID)
return nil
Comment on lines +152 to +202
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The UpdateProviderID method currently performs a redundant GetNode call. Since the caller in main.go already retrieves the node object, this extra API request is unnecessary.

Additionally, consider using the add operation instead of replace in the JSON patch. If spec.providerID is currently empty and omitted from the node's JSON representation, the replace operation will fail because the path does not exist. The add operation is safer as it works whether the field exists or not.

// UpdateProviderID updates the node's spec.providerID
func (u *Updater) UpdateProviderID(ctx context.Context, providerID string) error {
	klog.V(2).Infof("Updating providerID for node %s: %s", u.nodeName, providerID)

	// Create JSON patch for providerID
	// Using "add" is safer than "replace" as the field might not exist in the JSON yet
	patch := []map[string]interface{}{
		{
			"op":    "add",
			"path":  "/spec/providerID",
			"value": providerID,
		},
	}

	patchBytes, err := json.Marshal(patch)
	if err != nil {
		return fmt.Errorf("failed to marshal patch: %w", err)
	}

	klog.V(4).Infof("Applying providerID patch to node %s: %s", u.nodeName, string(patchBytes))

	// Apply patch
	_, err = u.client.CoreV1().Nodes().Patch(
		ctx,
		u.nodeName,
		types.JSONPatchType,
		patchBytes,
		metav1.PatchOptions{},
	)
	if err != nil {
		return fmt.Errorf("failed to patch node providerID: %w", err)
	}

	klog.Infof("Successfully updated providerID for node %s to %s", u.nodeName, providerID)
	return nil
}

}