STAC-22297 Updated documents to match the deployment on sandbox and saas-tooling prod clusters.

Author: LouisParkin

.gitbook/assets/otel/aws_nodejs_otel_proxy_collector_configuration.svg

@@ -93,6 +93,7 @@
 ## 🔭 Open Telemetry
 * [Getting started](setup/otel/getting-started.md)
 * [Open telemetry collector](setup/otel/collector.md)
+  * [Collector as a proxy](setup/otel/proxy-collector.md)
 * [Languages](setup/otel/languages/README.md)
   * [Generic Exporter configuration](setup/otel/languages/sdk-exporter-config.md)
   * [Java](setup/otel/languages/java.md)

SUMMARY.md

@@ -89,15 +89,23 @@ service:
       processors: []
 ```
 
-Be aware this collector is used to send the data over to a next collector which then is used for tail sampling, metric aggregation, etc. before sending data over to SUSE Observability. This second collector also needs to run in the customers environment. See this page for [instructions](../../collector.md).
+Be aware this collector is used to send the data over to a next collector which then is used for tail sampling, metric aggregation, etc. before sending data over to SUSE Observability. This second collector also needs to run in the customer's environment. 
+
+Depending on the desired functionality, or based upon factors such as volumes of data being generated by lambdas instrumented in this way, collectors can be set up for batching, tail-sampling, and other pre-processing techniques to reduce the impact on SUSE Observability.
+
+See this page for [guidance and instruction](../../proxy-collector.md) on how to set up a batching collector that acts as a security proxy for SUSE Observability.
+See this page for [instructions](../../collector.md) on how to set up a collector that does tail-sampling as well.
+For more information about processor configuration on the opentelemetry collector, see the [official documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md).
 
 ![AWS Lambda Instrumentation With Opentelemetry](/.gitbook/assets/otel/aws_nodejs_otel_auto_instrumentation.svg)
 
 ## Package.json
 
 Make sure to add `"@opentelemetry/auto-instrumentations-node": "^0.55.2",` to `package.json` and execute `npm install` to add the auto-instrumentation client libraries to your NodeJS Lambda.
 
-## Troubleshooting Timeouts
+## Troubleshooting
+
+### Timeouts
 
 If the addition of the OTEL Lambda layers results in lambdas that time out (checking the logs might indicate that the collector was asked to shut down while still busy, e.g. seeing the following log entry):
 
@@ -141,12 +149,19 @@ Note the memory increment is 128MB
 
 Note Timeout is an integer value denoting seconds.
 
+### Authentication and Source IP Filtering
+
+If you encounter `error 403 Unauthorized` when submitting collector data to your cluster, or to any pre-processing or proxy collector, double-check the source IP address of the VPC NAT gateway matches what is whitelisted by the collector ingress, 
+also double check that the chosen authentication mechanism matches source and destination, and also that credentials (secrets, etc.) are set up correctly.
+
+For more information about configuring authentication for the opentelemetry collector, please refer to the [official documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configauth/README.md).
+
 ## References
 
 Auto-instrumentation docs → [https://opentelemetry.io/docs/faas/lambda-auto-instrument/](https://opentelemetry.io/docs/faas/lambda-auto-instrument/)
 
 Collector docs → [https://opentelemetry.io/docs/faas/lambda-collector/](https://opentelemetry.io/docs/faas/lambda-collector/)
 
-Github Releases Page for finding latest ARNs → [https://github.com/open-telemetry/opentelemetry-lambda/releases](https://github.com/open-telemetry/opentelemetry-lambda/releases)
+GitHub Releases Page for finding latest ARNs → [https://github.com/open-telemetry/opentelemetry-lambda/releases](https://github.com/open-telemetry/opentelemetry-lambda/releases)
 
 OTLP Exporter Configuration → [https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/)

setup/otel/languages/node.js/auto-instrumentation-of-lambdas.md

@@ -0,0 +1,95 @@
+---
+description: SUSE Observability
+---
+
+# Open Telemetry Collector as a proxy
+
+The normal configuration of the Opentelemetry Collector for tail-sampling traces can be found [here](collector.md)
+
+The below configuration describes a deployment that only does batching, and no further processing of traces, metrics, 
+or logs.  It is meant as a security proxy that exists outside the SUSE Observability cluster, but within trusted network
+infrastructure.  Security credentials for the proxy and SUSE Observability can be set up separately, adding a layer of
+authentication that does not reside with the caller, but with the host.
+
+![AWS Lambda Instrumentation With Opentelemetry via proxy collector](/.gitbook/assets/otel/aws_nodejs_otel_proxy_collector_configuration.svg)
+
+{% code title="otel-collector.yaml" lineNumbers="true" %}
+```yaml
+mode: deployment
+presets:
+   kubernetesAttributes:
+      enabled: true
+      # You can also configure the preset to add all the associated pod's labels and annotations to you telemetry.
+      # The label/annotation name will become the resource attribute's key.
+      extractAllPodLabels: true
+extraEnvsFrom:
+   - secretRef:
+        name: open-telemetry-collector
+image:
+   # Temporary override for image tag, the helm chart has not been released yet
+   tag: 0.97.0
+
+config:
+   receivers:
+      otlp:
+         protocols:
+            grpc:
+               endpoint: 0.0.0.0:4317
+            http:
+               endpoint: 0.0.0.0:4318
+
+   exporters:
+      # Exporter for traces to traffic mirror (used by the common config)
+      otlp:
+         endpoint: <url for opentelemetry ingestion by suse observability>
+         auth:
+            authenticator: bearertokenauth
+
+   extensions:
+      bearertokenauth:
+         scheme: SUSEObservability
+         token: "${env:API_KEY}"
+
+   service:
+      extensions: [health_check, bearertokenauth]
+      pipelines:
+         traces:
+            receivers: [otlp]
+            processors: [batch]
+            exporters: [otlp]
+         metrics:
+            receivers: [otlp]
+            processors: [batch]
+            exporters: [otlp]
+         logs:
+            receivers: [otlp]
+            processors: [batch]
+            exporters: [otlp]
+
+ingress:
+  enabled: true
+  annotations:
+    kubernetes.io/ingress.class: ingress-nginx-external
+    nginx.ingress.kubernetes.io/ingress.class: ingress-nginx-external
+    nginx.ingress.kubernetes.io/backend-protocol: GRPC
+    # "12.34.56.78/32" IP address of NatGateway in the VPC where the otel data is originating from
+    #  nginx.ingress.kubernetes.io/whitelist-source-range: "12.34.56.78/32"
+  hosts:
+    - host: "otlp-collector-proxy.${CLUSTER_NAME}"
+      paths:
+        - path: /
+          pathType: ImplementationSpecific
+          port: 4317
+  tls:
+    - secretName: ${CLUSTER_NODOT}-ecc-tls
+      hosts:
+        - "otlp-collector-proxy.${CLUSTER_NAME}"
+```
+{% endcode %}
+
+
+### Ingress Source Range Whitelisting
+
+To emphasize the role of the proxy collector as a security measure, it is recommended to use a source-range whitelist
+to filter out data from untrusted and/or unknown sources.  In contrast, the SUSE Observability ingestion collector may 
+have to accept data from multiple sources, maintaining a whitelist on that level does not scale well.