[FLINK-24379][Formats] Avro Glue Schema Registry table format#122
[FLINK-24379][Formats] Avro Glue Schema Registry table format#122nicusX wants to merge 7 commits intoapache:mainfrom
Conversation
|
Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) |
|
@dannycranmer please review this PR when you have the chance |
Fixed param description to match actual default Fix typo in NOTICE
|
@z3d1k I addressed most of the comments. But I am not sure how to implement the IT, in particular for a format using an external registry. The only similar format, flink-avro-confluent-registry, doesn't have any IT. The GRS Serialization/DeserializationSchema IT relies on a AWS account that I assume is available during the CI/CD build. Any reference on how to leverage this? |
|
|
||
| /** Options for AWS Glue Schema Registry Avro format. */ | ||
| @PublicEvolving | ||
| public class AvroGlueFormatOptions { |
There was a problem hiding this comment.
WDYT about aligning the naming with AWSSchemaRegistryConstants (for all config options below)?
Example:
public static final ConfigOption<String> AWS_REGION =
ConfigOptions.key(AWSSchemaRegistryConstants.AWS_REGION)
.stringType()
.noDefaultValue()
.withDescription("AWS region");
| public static final ConfigOption<String> REGISTRY_NAME = | ||
| ConfigOptions.key("registry.name") | ||
| .stringType() | ||
| .noDefaultValue() |
There was a problem hiding this comment.
.defaultValue(AWSSchemaRegistryConstants.DEFAULT_REGISTRY_NAME)
| .withDescription( | ||
| "The Schema name under which to register the schema used by this format during serialization."); | ||
|
|
||
| public static final ConfigOption<AvroRecordType> SCHEMA_TYPE = |
There was a problem hiding this comment.
According to the serializer/deserializer shipped with this PR, you'll only ever operate with Generic Records. This option should not be part of the API.
Consequently, in the buildConfigMap of the factory, GENERIC_RECORD should be "forced" into the config map.
3 participants
Purpose of the change
Implement Table API support for Avro with AWS Glue Schema Registry
Verifying this change
Significant changes
@Public(Evolving))The implementation is mostly based on the original PR to the main Flink repo with minor changes. SQL uber-jar has been added and docs fixed and enriched.