Add Transform Processor to Collector Layer

[wpessers]

Is your feature request related to a problem? Please describe.

The default collector layer build includes several processors for modifying telemetry data (attributes, resource, span), but there are transformation use cases that none of these included processors can currently address.

Describe the solution you'd like

Add the transform processor to the collector layer. This processor uses the OpenTelemetry Transformation Language (OTTL) to express arbitrary transformations on traces, metrics, and logs.

Capabilities not covered by current processors

The following are things the transform processor can do that no combination of the currently included processors supports:

1. Metric type conversions and manipulation

Convert between metric types, extract sub-metrics, and scale values:

transform:
  metric_statements:
    - convert_sum_to_gauge() where metric.name == "system.processes.count"
    - extract_count_metric(true) where metric.name == "http.server.duration"
    - scale_metric(0.001) where metric.name == "faas.duration"
    - convert_exponential_histogram_to_histogram()

2. Log body and field manipulation

Current processors can't modify log record bodies or do pattern-based redaction across arbitrary fields:

transform:
  log_statements:
    - replace_pattern(body, "password=\\S+", "password=***REDACTED***")
    - set(attributes["log.level"], severity_text)

3. Cross-field and cross-level operations

Move or derive values across the resource/scope/signal hierarchy (the attributes processor can only copy within the same level):

transform:
  trace_statements:
    - set(attributes["env"], resource.attributes["deployment.environment"])
    - set(name, Concat([attributes["http.method"], " ", attributes["http.route"]], ""))
      where attributes["http.route"] != nil

4. Span event manipulation

None of the current processors can target span events (not even the span processor):

transform:
  trace_statements:
    - context: spanevent
      statements:
        - set(attributes["handled"], true)
          where name == "exception" and attributes["exception.type"] == "TimeoutError"

5. Complex conditional logic

The where clause provides full OTTL expressions, significantly more powerful than the include/exclude matching in the attributes/span processors:

transform:
  trace_statements:
    - set(attributes["error.category"], "server_error")
      where span.kind == 2
        and Int(attributes["http.response.status_code"]) >= 500
        and resource.attributes["service.name"] == "my-lambda-function"

Overlap with existing processors

The transform processor could aalso fully replace the attributes, resource, and span processors:

Existing processor	Transform processor equivalent
attributes: insert/update/upsert/delete/hash/extract/convert actions	set, delete_key, delete_matching_keys, SHA256, replace_pattern + where clauses
resource: upsert/insert/delete on resource attributes	Same OTTL statements in the resource context
span: rename from attributes, extract attributes from name, set status	set(name, ...), replace_pattern(name, ...), set(status.code, ...)

This doesn't mean I'm advocating for those processors to be removed entirely. They remain simpler and maybe more intuitive for most operations, and removing them would break existing user configurations. It does mean that users who choose to adopt the transform processor can consolidate multiple processor steps into one. Part of the outcome of this issue should be a decision on whether to remove the other processors in favor of the transform processor in the future.

Describe alternatives you've considered

Default build vs. custom-build-only

The transform processor could either be provided in the default build or made available only via custom builds.

A reasonable middle ground could be to start with custom-build-only and promote to the default build once there's evidence of demand or if binary size impact turns out to be negligible.

Measuring binary size impact

The incremental cost should be measured before deciding. Since pkg/ottl is already pulled in transitively, the difference will mainly be in the transform processor itself and its OTTL function registrations. A comparison build can be done to check this.

Tasks

• Add the transform processor to the collector layer so that users can opt-in to use it in their custom layer builds.
• Investigate impact on binary size / cold start

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[wpessers]

Just checked the binary size impact for the collector artifact with & without transformprocessor:

Without:

> collector % wc -c < build/extensions/collector
 50155682

~48MB

With:

> collector % wc -c < build/extensions/collector
 51040418

~49MB

A difference of ~864KB. In my experience this shouldn't make any noticeable impact on cold start, especially comparing to the total size of the collector layer, it's peanuts (roughly +1.7%). Of course it's important to look at this stuff on a case-by-case basis. If we progressively allow more components to be included into the collector layer, each having a small impact on the size, in the end the cold start may become noticeable of course.

For the transform processor specifically, I'm leaning toward including it in the default build. Especially since this gives us the option to maybe eventually drop some of the specialized processors (attributes, resource, and span) entirely if we so desire in the future (should probably be a major release with big warning signs). But also the transform processor just allows some nice capabilities we don't yet have and may have direct impact on other ongoing efforts.

WDYT @open-telemetry/lambda-extension-approvers
And especially @RaphaelManke since you've done some thinking and were also a proponent of adding this processor. And also @maxday since you will have more knowledge about cold-start impact, do tell me if I'm wrong regarding this 😄

[RaphaelManke]

That resonates well with my thoughts as well.
The deprecation or breaking change will be a bigger issue as I can imagine a lot of people where using these transformers.

For the final decision we need to run some actual tests and measure the cold start impacts but I am confident that they are negligible.

[wpessers]

Some test results with a sample size of 200. So 200 lambda cold starts for each arch (x86_64 vs arm64) and memory config. Meaning a total of 1600 lambda cold starts were observed. Results:

Baseline

x86_64

	min	max	avg	median	P95
128 MB	377.04 ms	652.16 ms	501.10 ms	497.62 ms	550.23 ms
256 MB	413.67 ms	683.59 ms	509.51 ms	500.94 ms	583.31 ms
512 MB	421.92 ms	669.17 ms	508.09 ms	499.06 ms	591.55 ms
1024 MB	430.31 ms	666.78 ms	508.79 ms	502.45 ms	567.89 ms

arm64

	min	max	avg	median	P95
128 MB	339.52 ms	516.04 ms	416.74 ms	410.48 ms	483.61 ms
256 MB	332.13 ms	517.25 ms	417.14 ms	411.89 ms	494.14 ms
512 MB	343.20 ms	538.98 ms	422.10 ms	413.89 ms	495.64 ms
1024 MB	332.97 ms	558.50 ms	421.30 ms	413.17 ms	506.26 ms

Transform Processor

x86_64

	min	max	avg	median	P95
128 MB	416.21 ms	732.84 ms	530.16 ms	522.88 ms	619.45 ms
256 MB	416.22 ms	867.81 ms	534.11 ms	519.42 ms	667.59 ms
512 MB	376.66 ms	880.00 ms	550.05 ms	531.23 ms	694.90 ms
1024 MB	395.13 ms	873.61 ms	557.75 ms	538.71 ms	738.39 ms

arm64

	min	max	avg	median	P95
128 MB	344.89 ms	597.94 ms	426.86 ms	416.19 ms	507.76 ms
256 MB	338.61 ms	557.52 ms	426.57 ms	417.71 ms	499.80 ms
512 MB	352.14 ms	526.16 ms	431.52 ms	426.40 ms	507.31 ms
1024 MB	347.55 ms	583.65 ms	429.55 ms	420.12 ms	520.02 ms

Delta of medians

	x86_64	arm64
128 MB	+25.26 ms (+5.1%)	+5.71 ms (+1.4%)
256 MB	+18.48 ms (+3.7%)	+5.82 ms (+1.4%)
512 MB	+32.17 ms (+6.4%)	+12.51 ms (+3.0%)
1024 MB	+36.26 ms (+7.2%)	+6.95 ms (+1.7%)

So seems like a small, consistent increase across architectures & memory configs. The lambda function I deployed for this was a minimal lambda function without any dependencies. If the lambda function's deployment package itself would be of any significant size, I thus suspect the observed impact on cold start would become close to unnoticeable.

It's been a while since I've run a similar benchmark, but the cold start overhead of the collector layer never fails to disappoint. Will see if we can do some exploration into alternatives or improvements regardless of this issue.

[herin049]

A little off-topic, but on the topic of cold-start/layer sizes I looked into this a bit in #2100 and found that the attributes processor alone somehow contributed nearly 10 mb to the layer size due to some of its sub-dependencies. It seems like the transform processor covers a significant amount of the functionality in the attributes processor, so perhaps we can remove the attributes processor.

Regardless, I think the performance impact is negligible enough to add it, if some users are sensitive to cold-starts, I would encourage them to build their own layer anyways.

[wpessers]

Yeah the transform processor could replace the attributes processor entirely. But we would not immediately replace it as stated earlier. But good to know it contributes so much to the size / cold start! And indeed agree, when cold start is sensitive a custom build is of course advised.

[maxday]

Awesome work @wpessers, about the numbers, are you reporting "INIT DURATION" from the REPORT Log line?
If yes, would you mind re-running this from an end-to-end perspective and measure the time between the invoke request and the response? I would be curious to see if we can notice a delta in layer downloading time when measuring the full trip. Of course there will be a small unrelated network latency, but I believe with a high enough number of invocation, you should be able to see a trend (or hopefully no trend)

[wpessers]

@maxday this is indeed the "INIT DURATION". Sounds interesting, didn't think of that yet. Will do some more runs tomorrow!

[RaphaelManke]

How are you gonna measure that time? X-ray tracing? Because the pure request time will include network latency of the caller.

[wpessers]

I was thinking of using an EC2 instance to run my benchmark script, I assume this would drop the network roundtrip time so low that any variance on it will be insignificant. And then paired with high sample size I assumed all "external factors" should be accounted for. What do you think?

[RaphaelManke]

Just a thought.
X-ray enabled with transaction search emits all X-ray spans into a cloudwatch loggroup.
Then there are spans for the lambda environment started and a span for the init duration.
The time between
Lambda service start time -> init duration end time
Would be the cold start time I think.

Maybe that's to much effort to do but this would eliminate the caller latency.

[wpessers]

It sounds like a good idea, I don't have much time to spend on this the coming days though so will just kick off the benchmark script on ec2 for now and see if I can get the data analysis done by tonight. I do agree that your proposed test method would seem as precise as possible though. Parsing the spans from cloudwatch would be a bit more tedious I think, so indeed it may just be a bit too much effort. If there is any desire for this further testing I'm willing to explore it though!

[wpessers]

Collected a lot of data, 500 samples for each arch (arm, x86) + memory config (128MB, 256MB, 512MB, 1024MB) + build (baseline, including transform processor). So a total of 8000 invocations have been observed to take the measurements.

I've measured:

• init duration (from lambda reported INIT_DURATION)
• duration (from lambda reported DURATION)
• e2e = wall clock time of the entire function invocation, includes the network roundtrip from my benchmark runner (on EC2) to Lambda and back
• overhead = e2e - init - duration so basically "everything else". This is my attempt at approximating the layer download time @maxday was asking about. By subtracting the init duration and invoke duration from the total "wall clock" duration, what's left should be: network latency, some other scheduling/placement overhead (?) and layer / function code download

It runs on EC2 so I'm assuming the network latency and other overhead should be roughly constant across the baseline and the transform processor build. Function code is as small as it can be and also constant, meaning only the layer size has slightly grown as described earlier.

It's a lot of data so I tried to make some boxplots to display the results, but the raw data is available through the attached csv files as well. The build including the transform processor is what I have named the "candidate" build.
raw_data_baseline.csv
raw_data_candidate.csv

I've capped the y-axis in a way to keep it readable, meaning the box and whiskers should all be fully visible. There's some outliers that won't show but since they're outliers I accepted this trade-off since it makes the graphs more readable / easy to compare.

Overhead

Starting with what seems most interesting / important. Again, this was calculated by taking: e2e_ms - init_ms - duration_ms

I don't think there's anything shocking here. Impact is negligible in my opinion.

Init duration

Graphed this because why not. Again nothing shocking here in my opinion. Except for the baseline x86_64 1024MB subject's noticeably wider IQR. But I can't think of an explanation for that. The candidate seems quite consistent across memory configs.