Add Standard (cross-sdk) Benchmarks

[alextwoods]

Add Standard (cross-sdk) Benchmarks

Motivation and Context

This change as the sdk-standard-benchmarks module which contains JMH microbenchmarks for the AWS SDK for Java v2, coveringendpoint resolution and serialization/deserialization (serde) across all major
AWS protocols.

Modifications

Adds a new sdk-standard-benchmarks module with the standard (performance part-2) benchmarks for endpoint resolution and serde.

The SERDE tests are based on the standard performance test models and use our existing protocol test loader to load the c2j protocol tests defined on each of the protocol models.

Endpoint Resolution Benchmarks

Benchmarks for the standard endpoint resolution pipeline (ruleParams() →
resolveEndpoint()) for S3 and Lambda. These exercise the same code path that
runs during a real SDK API call.

Class	Service	Test Cases
S3EndpointResolverBenchmark	S3	5 (virtual host, path style, S3 Express, access point ARN, outposts ARN)
LambdaEndpointResolverBenchmark	Lambda	2 (standard region, GovCloud with FIPS + DualStack)

Serde Benchmarks

Benchmarks for serialization (marshaling) and deserialization (unmarshalling)
across five AWS protocol types. Each protocol has a pair of benchmark classes
parameterized by test case ID via JMH @Param.

| Protocol | Marshall Class | Unmarshall Class |
|---|---|---|---|---|
| JSON RPC 1.0 | JsonRpc10MarshallBenchmark | JsonRpc10UnmarshallBenchmark |
| AWS Query | QueryMarshallBenchmark | QueryUnmarshallBenchmark |
| REST JSON | RestJsonMarshallBenchmark | RestJsonUnmarshallBenchmark |
| REST XML | RestXmlMarshallBenchmark | RestXmlUnmarshallBenchmark |
| RPC v2 CBOR | RpcV2CborMarshallBenchmark | RpcV2CborUnmarshallBenchmark |

SERDE benchmarks use custom, cross-sdk defined models that are loosely based on existing services, but designed
to test SERDE performance across SDKs. The models are maintained internally and copied in the codegen resources of this module. In addition, the test cases are defined using the protocol test format - for the Java SDK V2 we use the c2j v1 format. Those test cases are copied into the test/sdk-standard-benchmarks/src/main/resources/serde-tests by protocol and by input/output. We mostly use the existing protocol test loading/runner utilities to setup the cases, but we do no assertions. The logic for loading test cases is in the BenchmarkTestCaseLoader which also defines patchMemberNames for handling fluentSetter names like SS => ss.

Testing

Run all benchmarks

Results:

id	protocol	n	mean	p50	p90	p95	p99	std_dev
GetItemInput_Baseline	JsonRpc10	3,582,753	949	891	960	994	1,116	15
GetMetricDataRequest_S	JsonRpc10	2,569,368	2,499	2,416	2,512	2,552	2,804	13
GetMetricDataRequest_M	JsonRpc10	2,974,765	8,433	8,304	8,816	8,960	10,048	14
GetMetricDataRequest_L	JsonRpc10	2,904,981	17,184	16,928	17,600	17,952	20,288	15
HealthcheckRequest_Example	JsonRpc10	2,081,862	445	385	438	478	605	13
PutItemRequest_Baseline	JsonRpc10	3,274,691	1,006	969	1,038	1,074	1,200	6
PutItemRequest_ShallowMap_S	JsonRpc10	1,889,743	3,389	3,276	3,404	3,452	3,980	17
PutItemRequest_ShallowMap_M	JsonRpc10	2,565,424	19,460	19,232	19,904	20,096	23,456	14
PutItemRequest_ShallowMap_L	JsonRpc10	1,664,483	59,985	59,392	61,504	62,272	68,864	24
PutItemRequest_Nested_M	JsonRpc10	2,609,591	2,455	2,384	2,480	2,516	2,656	12
PutItemRequest_Nested_L	JsonRpc10	3,162,163	7,932	7,784	8,064	8,160	9,456	13
PutItemRequest_MixedItem_S	JsonRpc10	3,371,145	3,748	3,676	3,832	3,884	4,224	9
PutItemRequest_MixedItem_M	JsonRpc10	2,671,651	18,688	18,592	19,328	19,520	22,464	14
PutItemRequest_MixedItem_L	JsonRpc10	1,798,543	55,505	54,848	56,640	57,408	63,744	24
PutItemRequest_BinaryData_S	JsonRpc10	2,334,472	1,417	1,348	1,412	1,440	1,590	12
PutItemRequest_BinaryData_M	JsonRpc10	2,921,177	17,114	16,800	17,440	17,824	20,352	21
PutItemRequest_BinaryData_L	JsonRpc10	774,213	129,017	126,848	131,328	138,496	148,480	106
PutMetricDataRequest_Baseline	JsonRpc10	3,069,204	566	535	569	599	702	6
PutMetricDataRequest_S	JsonRpc10	1,985,361	1,632	1,576	1,664	1,700	1,818	11
PutMetricDataRequest_M	JsonRpc10	3,394,488	7,391	7,264	7,520	7,608	8,800	12
PutMetricDataRequest_L	JsonRpc10	1,295,387	77,064	76,160	78,720	80,384	90,112	47
GetItemOutput_Baseline	JsonRpc10	2,949,964	253	230	250	256	306	6
GetItemOutput_S	JsonRpc10	3,018,691	2,150	2,076	2,152	2,180	2,308	11
GetItemOutput_M	JsonRpc10	2,921,541	8,593	8,432	8,736	8,816	10,448	15
GetItemOutput_L	JsonRpc10	1,800,530	55,438	54,848	56,960	58,496	66,688	36
GetItemOutputBinary_S	JsonRpc10	2,830,388	2,287	2,208	2,288	2,316	2,600	12
GetItemOutputBinary_M	JsonRpc10	2,735,492	9,180	8,992	9,296	9,392	11,216	17
GetItemOutputBinary_L	JsonRpc10	1,635,835	61,027	60,032	62,528	64,128	72,960	40
HealthcheckResponse_Example	JsonRpc10	2,058,022	252	222	241	248	315	9
GetItemInput_Baseline	Query	2,511,149	702	644	688	709	811	11
GetMetricDataRequest_S	Query	2,358,715	2,752	2,624	2,724	2,768	3,736	17
GetMetricDataRequest_M	Query	2,387,262	10,541	10,256	10,640	10,848	13,024	23
GetMetricDataRequest_L	Query	2,369,123	21,135	20,608	21,408	22,016	27,072	28
HealthcheckRequest_Example	Query	3,631,068	152	132	150	163	232	5
PutItemRequest_Baseline	Query	2,213,520	793	730	773	793	894	12
PutItemRequest_ShallowMap_S	Query	2,777,201	4,585	4,424	4,616	4,720	6,152	16
PutItemRequest_ShallowMap_M	Query	1,612,022	31,080	30,368	31,808	32,704	41,280	42
PutItemRequest_ShallowMap_L	Query	996,685	100,196	98,432	103,040	107,008	117,376	81
PutItemRequest_Nested_M	Query	1,948,392	3,348	3,152	3,356	3,440	4,840	22
PutItemRequest_Nested_L	Query	2,833,985	17,667	17,088	18,368	18,944	21,664	28
PutItemRequest_MixedItem_S	Query	2,708,315	4,706	4,528	4,720	4,808	6,480	17
PutItemRequest_MixedItem_M	Query	1,697,582	29,502	28,736	30,176	31,072	39,808	41
PutItemRequest_MixedItem_L	Query	995,411	100,321	98,048	105,344	109,184	119,168	83
PutItemRequest_BinaryData_S	Query	2,762,787	1,216	1,146	1,216	1,248	1,384	11
PutItemRequest_BinaryData_M	Query	1,890,323	6,714	6,448	7,184	7,456	8,960	26
PutItemRequest_BinaryData_L	Query	1,487,970	33,738	32,768	34,496	35,200	44,160	60
PutMetricDataRequest_Baseline	Query	2,217,675	258	223	248	267	372	9
PutMetricDataRequest_S	Query	2,432,611	1,377	1,294	1,364	1,396	1,580	14
PutMetricDataRequest_M	Query	2,827,006	8,905	8,640	8,992	9,200	11,472	19
PutMetricDataRequest_L	Query	1,276,698	78,190	76,416	81,664	87,296	94,976	63
GetMetricDataResponse_S	Query	1,895,585	26,474	25,760	26,784	27,520	36,416	41
GetMetricDataResponse_M	Query	1,222,726	81,648	80,000	82,816	84,608	98,048	72
GetMetricDataResponse_L	Query	102,045	979,057	966,656	997,376	1,011,712	1,642,496	1,033
HealthcheckResponse_Example	Query	2,810,741	111	99	115	120	166	4
CopyObjectRequest_Baseline	RestJson	2,496,576	1,331	1,262	1,320	1,352	1,532	12
CopyObjectRequest_M	RestJson	2,046,681	6,183	6,008	6,200	6,256	7,832	20
GetMetricDataRequest_S	RestJson	2,494,449	2,586	2,496	2,588	2,620	2,852	14
GetMetricDataRequest_M	RestJson	2,869,650	8,730	8,592	8,896	8,992	10,064	14
GetMetricDataRequest_L	RestJson	2,876,221	17,344	17,088	17,696	18,016	20,064	14
HealthcheckRequest_Example	RestJson	3,040,484	174	155	173	187	242	5
PutMetricDataRequest_Baseline	RestJson	3,064,009	568	535	573	593	714	7
PutMetricDataRequest_S	RestJson	1,951,341	1,673	1,606	1,678	1,710	1,854	13
PutMetricDataRequest_M	RestJson	3,341,181	7,509	7,408	7,672	7,776	8,912	12
PutMetricDataRequest_L	RestJson	1,299,669	76,812	76,288	78,976	80,384	89,856	45
PutObject_S	RestJson	2,265,625	2,894	2,712	2,860	2,940	4,312	21
PutObject_M	RestJson	2,254,160	2,939	2,720	2,880	2,956	4,376	23
PutObject_L	RestJson	2,249,064	2,922	2,732	2,884	2,960	4,352	22
CopyObjectOutput_Baseline	RestJson	2,979,878	588	556	582	592	648	7
CopyObjectOutput_M	RestJson	3,713,404	3,419	3,360	3,468	3,504	4,328	9
GetObject_S	RestJson	3,494,685	3,635	3,560	3,692	3,740	4,552	10
GetObject_M	RestJson	3,279,064	3,880	3,768	3,972	4,048	4,984	12
GetObject_L	RestJson	758,886	131,565	128,768	133,120	137,728	152,320	143
HealthcheckResponse_Example	RestJson	2,106,821	242	215	233	240	304	13
CopyObjectRequest_Baseline	RestXml	2,582,001	1,293	1,228	1,300	1,332	1,490	11
CopyObjectRequest_M	RestXml	2,321,648	5,457	5,368	5,568	5,640	6,784	18
GetMetricDataRequest_S	RestXml	2,409,949	5,260	5,136	5,392	5,488	6,576	17
GetMetricDataRequest_M	RestXml	2,296,193	21,768	21,632	22,496	22,848	25,856	23
GetMetricDataRequest_L	RestXml	1,148,816	43,560	43,328	44,864	45,504	54,784	50
HealthcheckRequest_Example	RestXml	2,724,766	189	162	209	231	316	6
PutMetricDataRequest_Baseline	RestXml	2,993,674	577	542	577	601	707	8
PutMetricDataRequest_S	RestXml	2,157,367	2,980	2,864	2,988	3,040	3,792	17
PutMetricDataRequest_M	RestXml	2,612,382	19,127	18,816	19,488	19,840	22,560	20
PutMetricDataRequest_L	RestXml	535,972	186,379	183,808	191,488	196,352	204,800	114
PutObject_S	RestXml	2,634,149	1,263	1,204	1,252	1,272	1,376	11
PutObject_M	RestXml	2,623,035	1,272	1,212	1,266	1,286	1,386	11
PutObject_L	RestXml	2,613,595	1,273	1,216	1,268	1,290	1,400	10
CopyObjectOutput_Baseline	RestXml	3,561,711	509	478	510	521	561	6
CopyObjectOutput_M	RestXml	1,886,030	13,368	13,008	13,552	13,936	15,947	33
GetMetricDataResponse_S	RestXml	2,295,753	21,867	21,504	22,336	22,880	30,400	32
GetMetricDataResponse_M	RestXml	1,704,179	58,574	57,792	60,544	61,632	72,448	48
GetMetricDataResponse_L	RestXml	171,311	583,001	576,512	602,112	610,304	666,624	576
GetObject_S	RestXml	3,690,432	3,434	3,376	3,520	3,568	4,376	9
GetObject_M	RestXml	3,376,338	3,745	3,656	3,904	3,988	4,704	10
GetObject_L	RestXml	771,930	129,355	126,592	131,072	135,424	150,016	140
HealthcheckResponse_Example	RestXml	3,021,617	180	163	181	190	268	5
GetItemInput_Baseline	RpcV2Cbor	3,326,942	1,002	951	1,048	1,086	1,212	8
GetMetricDataRequest_S	RpcV2Cbor	2,840,933	2,255	2,204	2,304	2,340	2,524	10
GetMetricDataRequest_M	RpcV2Cbor	3,321,241	7,552	7,464	7,808	7,936	8,928	12
GetMetricDataRequest_L	RpcV2Cbor	1,668,193	15,039	14,896	15,504	15,744	17,440	26
HealthcheckRequest_Example	RpcV2Cbor	3,519,357	500	461	506	536	653	7
PutItemRequest_Baseline	RpcV2Cbor	3,126,400	1,056	1,016	1,080	1,114	1,246	7
PutItemRequest_ShallowMap_S	RpcV2Cbor	1,967,287	3,259	3,180	3,312	3,360	3,976	16
PutItemRequest_ShallowMap_M	RpcV2Cbor	2,754,157	18,121	18,176	18,848	19,072	21,504	13
PutItemRequest_ShallowMap_L	RpcV2Cbor	1,805,462	55,295	55,616	57,600	58,304	64,192	22
PutItemRequest_Nested_M	RpcV2Cbor	2,622,445	2,432	2,388	2,496	2,532	2,964	9
PutItemRequest_Nested_L	RpcV2Cbor	3,200,386	7,846	7,688	8,136	8,272	9,648	13
PutItemRequest_MixedItem_S	RpcV2Cbor	3,530,075	3,577	3,532	3,692	3,744	4,296	8
PutItemRequest_MixedItem_M	RpcV2Cbor	2,798,987	17,829	17,760	18,368	18,560	20,512	12
PutItemRequest_MixedItem_L	RpcV2Cbor	1,945,556	51,309	50,816	52,480	53,056	58,880	21
PutItemRequest_BinaryData_S	RpcV2Cbor	2,396,766	1,383	1,318	1,394	1,430	1,576	11
PutItemRequest_BinaryData_M	RpcV2Cbor	2,427,877	5,277	5,024	5,352	5,576	6,584	22
PutItemRequest_BinaryData_L	RpcV2Cbor	1,232,822	40,713	39,424	41,664	42,624	51,392	72
PutMetricDataRequest_Baseline	RpcV2Cbor	2,712,211	636	592	642	683	810	8
PutMetricDataRequest_S	RpcV2Cbor	2,198,677	1,478	1,430	1,520	1,556	1,664	9
PutMetricDataRequest_M	RpcV2Cbor	2,172,778	5,803	5,696	6,000	6,112	6,872	15
PutMetricDataRequest_L	RpcV2Cbor	1,033,791	48,356	48,384	50,112	50,752	57,600	44
GetItemOutput_Baseline	RpcV2Cbor	3,947,439	247	223	243	254	331	5
GetItemOutput_S	RpcV2Cbor	2,913,445	2,218	2,172	2,248	2,272	2,440	10
GetItemOutput_M	RpcV2Cbor	3,021,875	8,310	8,240	8,528	8,640	10,384	14
GetItemOutput_L	RpcV2Cbor	1,865,354	53,510	53,632	55,424	56,192	65,280	34
GetItemOutputBinary_S	RpcV2Cbor	2,919,438	2,213	2,164	2,240	2,264	2,452	10
GetItemOutputBinary_M	RpcV2Cbor	3,055,915	8,218	8,184	8,448	8,528	9,840	14
GetItemOutputBinary_L	RpcV2Cbor	1,881,770	53,044	53,120	54,976	55,744	64,576	34
HealthcheckResponse_Example	RpcV2Cbor	2,115,632	238	212	235	246	318	7

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist

• I have read the CONTRIBUTING document
• Local run of mvn install succeeds
• My code follows the code style of this project
• My change requires a change to the Javadoc documentation
• I have updated the Javadoc documentation accordingly
• I have added tests to cover my changes
• All new and existing tests passed
• I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
• My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

• I confirm that this pull request can be released under the Apache 2 license

[alextwoods]

This is a limitation of JMH - we can't dynamically load the json set of test cases and define a JMH parameter for it. @param values must be constant expressions at compile-time because of how JMH handles the lifecycle of a benchmark. To get the best measurements JMH is a generator that during build generates raw classes/methods for all of these.

[RanVaknin]

This PR needs some better release notes about the various benchmarks introduced (high level) and also the patchMemberNames workaround for the reflection driven parameterized test approach.

It would be ideal if the PR notes would have the actual results from running these benchmarks before we merge.

[RanVaknin]

Not a blocker, but when profiling DDB and the JSON serailizer I found that there was quite a big variance even with 5 warmups, and 5 iterations. When I run the benchmarks on the linux host I usually pass in -wi 10 -i 10 -f 2 as arguments to the jmh harness JAR. That reduces noise almost completely - at the cost of benchmarks running for a long time.

I think its worth running these at least 2 times and see if there's any significant noise before committing to @Warmup(iterations = 2), @Measurement(iterations = 5)

[RanVaknin]

Some test case IDs appear multiple times in the results with different values (for ex awsQuery_GetMetricDataResponse_S). Can you separate the results by protocol so it's clear which number came from which marshaller?

[alextwoods]

Some test case IDs appear multiple times in the results with different values (for ex awsQuery_GetMetricDataResponse_S). Can you separate the results by protocol so it's clear which number came from which marshaller?

Good idea - Done!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud