diff --git a/codelabs/schema-registry-beta/codelab.json b/codelabs/schema-registry-beta/codelab.json deleted file mode 100644 index 851c7294..00000000 --- a/codelabs/schema-registry-beta/codelab.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "environment": "web", - "format": "html", - "prefix": "https://storage.googleapis.com", - "mainga": "UA-49880327-14", - "updated": "2025-03-21T21:50:10Z", - "id": "schema-registry-beta", - "duration": 38, - "title": "Introduction to Schema Registry using the Solace Messaging API for Java (JCSMP) and Apicurio Registry", - "summary": "This codelab will walk you through getting started with the Apicurio Registry and the Solace SERDEs components using the Solace Messaging API for Java (JCSMP).", - "source": "main/markdown/schema-registry-beta/schema-registry-beta.md", - "theme": "", - "status": [ - "published" - ], - "category": [ - "solace" - ], - "tags": [ - "web" - ], - "feedback": "https://github.com/SolaceDev/solace-dev-codelabs/blob/master/markdown/schema-registry-beta", - "url": "schema-registry-beta" -} diff --git a/codelabs/schema-registry-beta/img/19a409a4b68aefec.jpg b/codelabs/schema-registry-beta/img/19a409a4b68aefec.jpg deleted file mode 100644 index 9b2a98d5..00000000 Binary files a/codelabs/schema-registry-beta/img/19a409a4b68aefec.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/354f8ca343e7f6cb.jpg b/codelabs/schema-registry-beta/img/354f8ca343e7f6cb.jpg deleted file mode 100644 index 10a30545..00000000 Binary files a/codelabs/schema-registry-beta/img/354f8ca343e7f6cb.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/43e621476bd04394.jpg b/codelabs/schema-registry-beta/img/43e621476bd04394.jpg deleted file mode 100644 index 52dbc117..00000000 Binary files a/codelabs/schema-registry-beta/img/43e621476bd04394.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/44f356558033e250.gif b/codelabs/schema-registry-beta/img/44f356558033e250.gif deleted file mode 100644 index 5364f91e..00000000 Binary files a/codelabs/schema-registry-beta/img/44f356558033e250.gif and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/55f2ea66a1cb12f2.jpg b/codelabs/schema-registry-beta/img/55f2ea66a1cb12f2.jpg deleted file mode 100644 index 91e5fdfe..00000000 Binary files a/codelabs/schema-registry-beta/img/55f2ea66a1cb12f2.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/7fc4225c303820be.jpg b/codelabs/schema-registry-beta/img/7fc4225c303820be.jpg deleted file mode 100644 index 3b7af654..00000000 Binary files a/codelabs/schema-registry-beta/img/7fc4225c303820be.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/8fdfc83641421597.jpg b/codelabs/schema-registry-beta/img/8fdfc83641421597.jpg deleted file mode 100644 index e7392c38..00000000 Binary files a/codelabs/schema-registry-beta/img/8fdfc83641421597.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/a8e88af8691ee4ee.jpg b/codelabs/schema-registry-beta/img/a8e88af8691ee4ee.jpg deleted file mode 100644 index 2ae6a4a2..00000000 Binary files a/codelabs/schema-registry-beta/img/a8e88af8691ee4ee.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/ae2ff71ecc0caf62.jpg b/codelabs/schema-registry-beta/img/ae2ff71ecc0caf62.jpg deleted file mode 100644 index 38f1e295..00000000 Binary files a/codelabs/schema-registry-beta/img/ae2ff71ecc0caf62.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/afb336b29e671007.jpg b/codelabs/schema-registry-beta/img/afb336b29e671007.jpg deleted file mode 100644 index 3b93684c..00000000 Binary files a/codelabs/schema-registry-beta/img/afb336b29e671007.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/c2aa56b2a7de61ea.jpg b/codelabs/schema-registry-beta/img/c2aa56b2a7de61ea.jpg deleted file mode 100644 index 4594675d..00000000 Binary files a/codelabs/schema-registry-beta/img/c2aa56b2a7de61ea.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/img/fd43c225f39e0486.jpg b/codelabs/schema-registry-beta/img/fd43c225f39e0486.jpg deleted file mode 100644 index f44fb68c..00000000 Binary files a/codelabs/schema-registry-beta/img/fd43c225f39e0486.jpg and /dev/null differ diff --git a/codelabs/schema-registry-beta/index.html b/codelabs/schema-registry-beta/index.html deleted file mode 100644 index aa1a818e..00000000 --- a/codelabs/schema-registry-beta/index.html +++ /dev/null @@ -1,269 +0,0 @@ - - - - - - - - - Introduction to Schema Registry using the Solace Messaging API for Java (JCSMP) and Apicurio Registry - - - - - - - - - - -

In today's data-driven world, ensuring data consistency and interoperability across different systems is crucial. This is where a schema registry comes into play. In this walkthrough, we'll explore how to use a schema registry (Apicurio Registry) with the Solace JCSMP API, focusing on the Apache Avro format for schema definition.

-

You'll learn about:

-

✔️ What a schema registry is and why it's important
✔️ What a Serializer and Deserializer (SERDEs) is and the role they play
✔️ How to set up an instance of the Apicurio Registry using Docker Compose
✔️ How to create and register schemas
✔️ How to use schemas in your event-driven applications with Solace's JCSMP API
✔️ Best practices for schema evolution

-

You can also check out Aaron's office hours covering the schema registry here:

- - - - - - -
    -
  1. A general understanding of event-driven architecture (EDA) terms and concepts.
    2. -
  2. A locally running PubSub+ Broker or a free trial account of Solace PubSub+ Cloud. Don't have one? Sign up here.
      -
    • Along with the connection information for the broker
      • -
    -
    3. -
  3. Basic knowledge of Apache Avro schema format
    4. -
  4. Docker installed on your system
    5. -
  5. Java Development Kit (JDK) version 11+ installed on your system
    6. -
  6. An IDE of your choice (e.g., IntelliJ IDEA, Eclipse, Visual Studio Code)
    7. -
  7. Download the provided BETA zip package named Schema-Registry-Beta-Package_(latest-version).zip that contains all the necessary pieces you will need. This is available on the Solace Product Portal and unzip it to your preferred directory.
    8. -
-

NOTE: If you cannot access the Solace Product Portal, please click the Report a mistake at the bottom left of the codelab and open an issue asking for access.

- - - - - - -

A schema registry is a central repository for managing and storing schemas. It helps ensure data consistency, enables data governance, and supports schema evolution. Here's why it's important:

-
    -
  1. Data Consistency: Ensures that producers and consumers agree on the data format.
    2. -
  2. Interoperability: Allows different systems to communicate effectively.
    3. -
  3. Schema Evolution: Supports versioning and compatibility checks as schemas change over time.
    4. -
  4. Data Governance: Centralizes schema management for better control and auditing.
    5. -
- - - - - - -

A SERDEs (Serializer/Deserializer) in the context of a schema registry is a component that handles two key functions:

-
    -
  1. Serialization: Converting data objects from their native format (like Java objects) into a binary format suitable for transmission or storage.
    2. -
  2. Deserialization: Converting the binary data back into its original format for processing.
    3. -
-

In a schema registry system, SERDEs work closely with schemas to:

- -

For example:

- -

This is a fundamental concept in message-based systems where data needs to be:

- - - - - - - -

We'll use Docker Compose to set up the Apicurio Registry quickly and easily. We've prepared a Docker Compose file that will launch an instance of the Apicurio Registry and all the necessary components with a pre-defined configuration.

-
    -
  1. In these subsequent steps we will use the package that came from the downloaded zip from the prerequisites section. Navigate to the extracted folder called Schema-Registry-Beta-Package. You should see the following files and folders:
    2. -
-

-
    -
  1. Open a terminal or command prompt window and navigate to the extracted location of the folder called solace-schema-registry-dist.
    2. -
  2. Optionally you can make changes to the .env file to change things such as default login or ports. We will leave everything to defaults for this codelab.
    3. -
-

-
    -
  1. Run the following command: docker compose up -d and all the components will start up with the default values configured.
    2. -
  2. Once the script is done running, you should now be able to go to your browser and navigate to localhost:8888 which should re-direct you to the Apicurio Registry login screen.
    3. -
-

-

That's it, you have now installed an instance of the Apicurio Registry with the Postgres storage option!

- - - - - -

Let's create a simple schema for a User event:

-
    -
  1. Open the Apicurio Registry UI in your web browser by going to localhost:8888.
    2. -
  2. Login with the predefined credentials for a developer. In this case the username is sr-developer and password is devPassword.
    3. -
-

-
    -
  1. Click on Create artifact button. Once the dialogue opens enter the following as shown below:
      -
    • Group Id: Leave it empty (default)
      • -
    • Artifact Id: Set to solace/samples
      • -
    • Type: Set to Avro Schema
      • -
    -
    2. -
-

-

-
    -
  1. We will skip the Artifact Metadata step as it is optional and will click Next.
    2. -
  2. For the "Version Content" section, copy the Avro schema from below and either paste directly or save it into a file and upload it and click Next when done.
    3. -
-
{
-  "namespace": "com.solace.samples.serdes.avro.schema",
-  "type": "record",
-  "name": "User",
-  "fields": [
-    {"name": "id", "type": "string"},
-    {"name": "name", "type": "string"},
-    {"name": "email", "type": "string"}
-  ]
-}
-
-

-
    -
  1. We will skip the Version Metadata step as it is optional and will simply click the Create button.
    2. -
  2. Finally after you have successfully created the new schema you should see the following:
    3. -
-

-

You've now created and registered your first schema!

- - - - - -

Now, let's see how to use this schema in Java using the Solace Messaging API for Java (JCSMP):

-
    -
  1. Open a command window or terminal in the Schema-Registry-Beta-Package/jcsmp-samples directory.
    2. -
-

NOTE: For winows users, use the gradlew.bat file instead of gradlew in the below steps.

-
    -
  1. Run the following command to build the sample on Windows ./gradlew.bat build and . You should see a BUILD SUCCESSFUL message.
    2. -
  2. Run the sample application and provide the broker connection details like so ./gradlew.bat runHelloWorldJCSMPAvroSerde --args="localhost:55555 default default".
    3. -
-

This sample talks to the locally deployed Apicurio Schema Registry and retrieves the schema along with the schema ID. It will then do the following:

-

Configures the Serializer and Deserializer:

-
// Create and configure Avro serializer and deserializer
-        try (Serializer<GenericRecord> serializer = new AvroSerializer<>();
-             Deserializer<GenericRecord> deserializer = new AvroDeserializer<>()) {
-
-            serializer.configure(getConfig());
-            deserializer.configure(getConfig());
-
-
   /**
-     * Returns a configuration map for the Avro serializer and deserializer.
-     *
-     * @return A Map containing configuration properties
-     */
-    private static Map<String, Object> getConfig() {
-        Map<String, Object> config = new HashMap<>();
-        config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL);
-        config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME);
-        config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD);
-        return config;
-    }
-
-

Serializes the message payload and publishes the message to the connected broker on solace/samples destination with the serialized payload and schema ID:

-
   // Create and populate a GenericRecord with sample data
-            GenericRecord user = initEmptyUserRecord();
-            user.put("name", "John Doe");
-            user.put("id", "123");
-            user.put("email", "support@solace.com");
-
-            // Serialize and send the message
-            BytesMessage msg = JCSMPFactory.onlyInstance().createMessage(BytesMessage.class);
-            SerdeMessage.serialize(serializer, topic, msg, user);
-            System.out.printf("Sending Message:%n%s%n", msg.dump());
-            producer.send(msg, topic);
-
-

It will then receive the published message and deserialize it by looking up the schema ID and retrieving the schema and print it out to console:

-
  // Set up the message consumer with a deserialization callback
-            XMLMessageConsumer cons = session.getMessageConsumer(Consumed.with(deserializer, (msg, genericRecord) -> {
-                System.out.printf("Got record: %s%n", genericRecord);
-                latch.countDown(); // Signal the main thread that a message has been received
-            }, (msg, deserializationException) -> {
-                System.out.printf("Got exception: %s%n", deserializationException);
-                System.out.printf("But still have access to the message: %s%n", msg.dump());
-                latch.countDown();
-            }, jcsmpException -> {
-                System.out.printf("Got exception: %s%n", jcsmpException);
-                latch.countDown();
-            }));
-            cons.start();
-
-

The sourcecode can be further looked at by opening the HelloWorldJCSMPAvroSerde.java file.

- - - - - -

As your data model evolves, you'll need to update your schemas. Here are some best practices:

-
    -
  1. Backward Compatibility: Ensure new schema versions can read old data.
    2. -
  2. Forward Compatibility: Ensure old schema versions can read new data (ignoring new fields).
    3. -
  3. Full Compatibility: Aim for both backward and forward compatibility.
    4. -
  4. Versioning: Use semantic versioning for your schemas.
    5. -
  5. Default Values: Provide default values for new fields to maintain backward compatibility.
    6. -
  6. Avoid Renaming: Instead of renaming fields, add new fields and deprecate old ones.
    7. -
-

When updating a schema in Apicurio Registry:

-
    -
  1. Create a new version of the existing schema.
    2. -
  2. Make your changes, following the best practices above.
    3. -
  3. Use the compatibility rule in the registry to ensure your changes don't break existing consumers.
    4. -
- - - - - -

✔️ Schema registries are crucial for maintaining data consistency in event-driven architectures.
✔️ Apicurio Registry provides a powerful platform for managing schemas.
✔️ Apache Avro is a popular choice for schema definition due to its compact binary format and schema evolution capabilities.
✔️ Solace's JCSMP API can be used effectively with the Apicurio Registry for robust event-driven applications.
✔️ Proper schema evolution practices ensure smooth updates without breaking existing systems.
✔️ Using schemas in your applications helps catch data issues early and improves overall system reliability.
✔️ More schema SERDEs will be made available in the future including JSON.

-

Soly Image Caption

-

Thanks for participating in this codelab! Let us know what you thought in the Solace Community Forum! If you found any issues along the way we'd appreciate it if you'd raise them by clicking the Report a mistake button at the bottom left of this codelab.

- - - - - - - - - - - - - - diff --git a/codelabs/schema-registry/codelab.json b/codelabs/schema-registry/codelab.json deleted file mode 100644 index 2a713391..00000000 --- a/codelabs/schema-registry/codelab.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "environment": "web", - "format": "html", - "prefix": "https://storage.googleapis.com", - "mainga": "UA-49880327-14", - "updated": "2025-08-26T19:09:02Z", - "id": "schema-registry", - "duration": 46, - "title": "Introduction to Solace Schema Registry using the Solace Messaging API for Java (JCSMP) and REST Messaging", - "summary": "This codelab will walk you through getting started with the Solace Schema Registry and the Solace SERDES Collection using the Solace Messaging API for Java (JCSMP) or with REST messaging.", - "source": "main/markdown/schema-registry-beta/schema-registry.md", - "theme": "", - "status": [ - "published" - ], - "category": [ - "solace" - ], - "tags": [ - "web" - ], - "feedback": "https://github.com/SolaceDev/solace-dev-codelabs/blob/master/markdown/schema-registry", - "url": "schema-registry" -} diff --git a/codelabs/schema-registry/img/134b2dcf881816d7.png b/codelabs/schema-registry/img/134b2dcf881816d7.png deleted file mode 100644 index 781a593c..00000000 Binary files a/codelabs/schema-registry/img/134b2dcf881816d7.png and /dev/null differ diff --git a/codelabs/schema-registry/img/2102ac06805143bf.png b/codelabs/schema-registry/img/2102ac06805143bf.png deleted file mode 100644 index b0a09c62..00000000 Binary files a/codelabs/schema-registry/img/2102ac06805143bf.png and /dev/null differ diff --git a/codelabs/schema-registry/img/3b37e2a12d4f7660.png b/codelabs/schema-registry/img/3b37e2a12d4f7660.png deleted file mode 100644 index 6cc60987..00000000 Binary files a/codelabs/schema-registry/img/3b37e2a12d4f7660.png and /dev/null differ diff --git a/codelabs/schema-registry/img/4429ba3a9a93ef03.png b/codelabs/schema-registry/img/4429ba3a9a93ef03.png deleted file mode 100644 index 6df9c035..00000000 Binary files a/codelabs/schema-registry/img/4429ba3a9a93ef03.png and /dev/null differ diff --git a/codelabs/schema-registry/img/445fbe96f4560f60.png b/codelabs/schema-registry/img/445fbe96f4560f60.png deleted file mode 100644 index 0d29a185..00000000 Binary files a/codelabs/schema-registry/img/445fbe96f4560f60.png and /dev/null differ diff --git a/codelabs/schema-registry/img/44f356558033e250.gif b/codelabs/schema-registry/img/44f356558033e250.gif deleted file mode 100644 index 5364f91e..00000000 Binary files a/codelabs/schema-registry/img/44f356558033e250.gif and /dev/null differ diff --git a/codelabs/schema-registry/img/57321296ef7ab5fe.png b/codelabs/schema-registry/img/57321296ef7ab5fe.png deleted file mode 100644 index 21e9163b..00000000 Binary files a/codelabs/schema-registry/img/57321296ef7ab5fe.png and /dev/null differ diff --git a/codelabs/schema-registry/img/7fd0a0a88c387e48.png b/codelabs/schema-registry/img/7fd0a0a88c387e48.png deleted file mode 100644 index d699c1dd..00000000 Binary files a/codelabs/schema-registry/img/7fd0a0a88c387e48.png and /dev/null differ diff --git a/codelabs/schema-registry/img/9e1429bc511f4660.png b/codelabs/schema-registry/img/9e1429bc511f4660.png deleted file mode 100644 index 008cdbb0..00000000 Binary files a/codelabs/schema-registry/img/9e1429bc511f4660.png and /dev/null differ diff --git a/codelabs/schema-registry/img/a04532457850c591.png b/codelabs/schema-registry/img/a04532457850c591.png deleted file mode 100644 index 0a7e7911..00000000 Binary files a/codelabs/schema-registry/img/a04532457850c591.png and /dev/null differ diff --git a/codelabs/schema-registry/img/b490b0bb89d37a5f.png b/codelabs/schema-registry/img/b490b0bb89d37a5f.png deleted file mode 100644 index 0e18c414..00000000 Binary files a/codelabs/schema-registry/img/b490b0bb89d37a5f.png and /dev/null differ diff --git a/codelabs/schema-registry/img/b92182bdb2aa8a90.png b/codelabs/schema-registry/img/b92182bdb2aa8a90.png deleted file mode 100644 index a719f208..00000000 Binary files a/codelabs/schema-registry/img/b92182bdb2aa8a90.png and /dev/null differ diff --git a/codelabs/schema-registry/img/d97c4089aefc1da.png b/codelabs/schema-registry/img/d97c4089aefc1da.png deleted file mode 100644 index c1d14c79..00000000 Binary files a/codelabs/schema-registry/img/d97c4089aefc1da.png and /dev/null differ diff --git a/codelabs/schema-registry/img/f0a21d1cd9778ae6.png b/codelabs/schema-registry/img/f0a21d1cd9778ae6.png deleted file mode 100644 index 2247fa54..00000000 Binary files a/codelabs/schema-registry/img/f0a21d1cd9778ae6.png and /dev/null differ diff --git a/codelabs/schema-registry/img/f4f9351c0828add0.png b/codelabs/schema-registry/img/f4f9351c0828add0.png deleted file mode 100644 index 20f761e7..00000000 Binary files a/codelabs/schema-registry/img/f4f9351c0828add0.png and /dev/null differ diff --git a/codelabs/schema-registry/index.html b/codelabs/schema-registry/index.html deleted file mode 100644 index 121b083a..00000000 --- a/codelabs/schema-registry/index.html +++ /dev/null @@ -1,465 +0,0 @@ - - - - - - - - - Introduction to Solace Schema Registry using the Solace Messaging API for Java (JCSMP) and REST Messaging - - - - - - - - - - -

In today's data-driven world, ensuring data consistency and interoperability across different systems is crucial. This is where a schema registry comes into play. In this walkthrough, we'll explore how to use Solace Schema Registry with the Solace JCSMP API or REST applications, focusing on the JSON format for schema definition.

-

You'll learn about:

-

✔️ What a schema registry is and why it's important
✔️ What a Serializer and Deserializer (SERDES) is and the role they play
✔️ How to deploy and configure Solace Schema Registry using either Docker Compose or Kubernetes
✔️ How to create and manage schema artifacts in Solace Schema Registry web console
✔️ How to use schemas in your event-driven applications with Solace's JCSMP API or with REST messaging
✔️ Best practices for schema evolution

- - - - - -
    -
  1. A general understanding of event-driven architecture (EDA) terms and concepts.
    2. -
  2. A locally running Solace Broker or a free trial account of Solace Cloud. Don't have one? Sign up here.
      -
    • Along with the connection information for the broker
      • -
    -
    3. -
  3. Basic knowledge of JSON schema format
    4. -
  4. Docker installed on your system (for standalone deployment)
    5. -
  5. Kubernetes with Helm (for high availability deployments)
    6. -
  6. Java Development Kit (JDK) version 11+ installed on your system
    7. -
  7. An IDE of your choice (e.g., IntelliJ IDEA, Eclipse, Visual Studio Code)
    8. -
  8. Download the provided GA tarball package named schema-registry-v1.0.0.tar.gz that contains all the necessary pieces you will need. This is available on the Solace Product Portal.
    9. -
-

NOTE: If you cannot access the Solace Product Portal, please click the Report a mistake at the bottom left of the codelab and open an issue asking for access.

- - - - - -

A schema registry is a central repository for managing and storing schemas. It helps ensure data consistency, enables data governance, and supports schema evolution. Here's why it's important:

-
    -
  1. Data Consistency: Ensures that producers and consumers agree on the data format.
    2. -
  2. Interoperability: Allows different systems to communicate effectively.
    3. -
  3. Schema Evolution: Supports versioning and compatibility checks as schemas change over time.
    4. -
  4. Data Governance: Centralizes schema management for better control and auditing.
    5. -
- - - - - -

A SERDES (Serializer/Deserializer) in the context of a schema registry is a component that handles two key functions:

-
    -
  1. Serialization: Converting data objects from their native format (like Java objects) into a binary format suitable for transmission or storage.
    2. -
  2. Deserialization: Converting the binary data back into its original format for processing.
    3. -
-

In a schema registry system, SERDES works closely with schemas to:

- -

For example:

- -

This is a fundamental concept in message-based systems where data needs to be:

- - - - - - - -

We'll use Docker Compose to set up the Solace Schema Registry, for local development or standalone use cases with minimal resource and networking requirements. We've prepared a Docker Compose file that will launch an instance of the Solace Schema Registry and all the necessary components with a pre-defined configuration.

-
    -
  1. In these subsequent steps we will use the package that came from the downloaded tarball package from the prerequisites section. Navigate to the extracted folder called Schema-Registry-V1.0. You should see the following files and folders:
    2. -
-

-
    -
  1. Open a terminal or command prompt window and navigate to the extracted location of the folder called solace-schema-registry-dist.
    2. -
  2. Run the command for img in {docker-images}/*.tar.gz; do docker load -i "$img"; done to load docker images. For Window use this instead for %i in (docker-images\*.tar.gz \*.tar.gz) do docker load -i "%i".
    3. -
  3. You can make changes to the .env file to change things such as default login or ports. While using the built-in identity provider, we can leave everything to defaults for this codelab.
    4. -
-

-
    -
  1. Run the following command and all the components will start up with the specified values configured: docker compose -f compose.yaml -f compose.nginx.yaml -f compose.nginx.for.embedded.yaml -f compose.embedded.yaml up -d.
    2. -
  2. Once the script is done running, you should now be able to go to your browser and navigate to localhost:8888 which should re-direct you to the Solace Schema Registry login screen.
    3. -
-

-

That's it, you have now installed an instance of the Solace Schema Registry!

-

Alternatively, for enterprise-grade security features, Solace Schema Registry supports external identity providers. For that, make the necessary configurations in your IdP and set the environment variables in your .env file:

-

-

Run the following command docker compose -f compose.yaml -f compose.nginx.yaml -f compose.nginx.for.external.yaml -f compose.external.multiple.issuers.yaml up -d.

- - - - - -

Alternatively, we can set up Solace Schema Registry on a Kubernetes cluster using Helm.

-

NOTE: You must have an existing Kubernetes cluster available (for example, Amazon EKS, Google GKE, Microsoft AKS, or an on-prem/self-managed cluster).

-
    -
  1. Configure kubectl to communicate with your cluster.
    2. -
  2. In this method of deployment also, we will use the package that came from the downloaded tarball package from the prerequisites section. Navigate to the extracted folder called Schema-Registry-V1.0.
    3. -
  3. Open the commonad and run this command to load docker images at your preferred location:
    4. -
-
export REGISTRY="your-registry.com/project"
-for img in {docker-images}/*.tar.gz; do
-  LOADED=$(docker load -i "$img" | grep "Loaded image:" | cut -d' ' -f3)
-  NEW_NAME="$REGISTRY/$(basename "$img" .tar.gz)"
-  docker tag "$LOADED" "$NEW_NAME"
-  docker push "$NEW_NAME"
-done
-
-
    -
  1. Install the CloudNative PostgreSQL Operator for database management: kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.26/releases/cnpg-1.26.0.yaml.
    2. -
  2. Update a values.yaml file with your environment-specific configuration, database, authentication, ingress and TLS configuration: Insert image
    3. -
  3. Install Solace Schema Registry using Helm: helm upgrade --install schema-registry ./solace-schema-registry
    4. -
  4. Verify the deployment: kubectl get pods -n solace
    5. -
  5. To access the deployed services, replace <ingress.hostNameSuffix> with the actual hostname or IP address you configured for your ingress: https://ui.
    6. -
- - - - - -

Let's create a simple schema for a clock-in-out event:

-
    -
  1. Open the Solace Schema Registry UI in your web browser by going to localhost:8888.
    2. -
  2. Login with the predefined credentials for a developer. In this case the username is sr-developer and password is devPassword.
    3. -
-

-
    -
  1. Click on Create artifact button. Once the dialogue opens enter the following as shown below:
      -
    • Group Id: Leave it empty (default)
      • -
    • Artifact Id: Set to solace/samples/clock-in-out/json
      • -
    • Type: Set to JSON Schema
      • -
    -
    2. -
-

-

-
    -
  1. We will skip the Artifact Metadata step as it is optional and will click Next.
    2. -
  2. For the "Version Content" section, copy the Json schema from below and either paste directly or save it into a file and upload it and click Next when done.
    3. -
-
{
-  "$schema": "http://json-schema.org/draft-07/schema#",
-  "title": "clock-in-out",
-  "type": "object",
-  "additionalProperties": false,
-  "properties": {
-    "region_code": {
-      "description": "region code for clock in or out",
-      "type": "string"
-    },
-    "store_id": {
-      "description": "Store identifier",
-      "type": "string"
-    },
-    "employee_id": {
-      "description": "Employee ID for who clocked in or out",
-      "type": "string"
-    },
-    "datetime": {
-      "description": "Clock time",
-      "type": "string",
-      "format": "date-time"
-    }
-  },
-  "required": [
-    "region_code",
-    "store_id",
-    "employee_id",
-    "datetime"
-  ]
-}
-
-

-
    -
  1. We will skip the Version Metadata step as it is optional and will simply click the Create button.
    2. -
  2. Finally after you have successfully created the new schema you should see the following:
    3. -
-

-

You've now created and registered your first schema!

- - - - - -

Now, let's see how to use this schema in Java using the Solace Messaging API for Java (JCSMP):

-
    -
  1. Open a command window or terminal and clone this GitHub repository, and open the solace-samples-java-jcsmp directory. With git clone https://github.com/SolaceSamples/solace-samples-java-jcsmp and cd solace-samples-java-jcsmp
    2. -
  2. Run the command ./gradlew assemble to build the sample.
    3. -
  3. Open the build/staged directory by running cd build/staged and run the sample application with the broker connection details: bin/HelloWorldJCSMPJsonSchemaSerde localhost:55555 default default
    4. -
-

This sample talks to the locally deployed Solace Schema Registry and retrieves the schema along with the schema ID. It will then do the following:

-

Configures the Serializer and Deserializer:

-
// Create and configure JSON Schema serializer and deserializer
-        try (Serializer<JsonNode> serializer = new JsonSchemaSerializer<>();
-             Deserializer<JsonNode> deserializer = new JsonSchemaDeserializer<>()) {
-
-            serializer.configure(getConfig());
-            deserializer.configure(getConfig());
-
-
  /**
-     * Returns a configuration map for the JSON Schema serializer and deserializer.
-     *
-     * @return A Map containing configuration properties
-     */
-    private static Map<String, Object> getConfig() {
-        Map<String, Object> config = new HashMap<>();
-        config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL);
-        config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME);
-        config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD);
-        return config;
-    }
-
-
-

Serializes the message payload and publishes the message to the connected broker on solace/samples/clock-in-out/json destination with the serialized payload and schema ID:

-
   // Create and populate a ClockInOut JsonNode with sample data
-            ObjectMapper mapper = new ObjectMapper();
-            ObjectNode clockInOut = mapper.createObjectNode();
-            clockInOut.put("region_code", "NA-WEST");
-            clockInOut.put("store_id", "STORE-001");
-            clockInOut.put("employee_id", "EMP-12345");
-            clockInOut.put("datetime", "2025-01-20T15:30:00Z");
-
-            // Serialize and send the message
-            BytesMessage msg = JCSMPFactory.onlyInstance().createMessage(BytesMessage.class);
-            SerdeMessage.serialize(serializer, topic, msg, clockInOut);
-            System.out.printf("Sending ClockInOut Message:%n%s%n", msg.dump());
-             producer.send(msg, topic);
-
-

It will then receive the published message and deserialize it by looking up the schema ID and retrieving the schema and print it out to console:

-
   // Set up the message consumer with a deserialization callback
-            XMLMessageConsumer cons = session.getMessageConsumer(Consumed.with(deserializer, (msg, clockInOutData) -> {
-                System.out.printf("Got a ClockInOut JsonNode: %s%n", clockInOutData);
-                System.out.printf("Employee %s clocked in/out at store %s in region %s at %s%n",
-                        clockInOutData.get("employee_id").asText(),
-                        clockInOutData.get("store_id").asText(),
-                        clockInOutData.get("region_code").asText(),
-                        clockInOutData.get("datetime").asText());
-                latch.countDown(); // Signal the main thread that a message has been received
-            }, (msg, deserializationException) -> {
-                System.out.printf("Got exception: %s%n", deserializationException);
-                System.out.printf("But still have access to the message: %s%n", msg.dump());
-                latch.countDown();
-            }, jcsmpException -> {
-                System.out.printf("Got exception: %s%n", jcsmpException);
-                latch.countDown();
-            }));
-            cons.start();
-
-

The sourcecode can be further looked at by opening the HelloWorldJCSMPJsonSchemaSerde.java file.

- - - - - -

We can also use this schema with Solace Schema Registry in REST-based messaging environments.

- -

With this REST messaging, we will use the following Schema example. You can upload it on Solace Schema Registry as a new artifact with an Artifact Id of solace/samples/json - Refer to Step 7 Creating and Registering Schemas if you need to review the steps.

-
{
-  "$schema": "http://json-schema.org/draft-07/schema#",
-  "title": "User",
-  "type": "object",
-  "customJavaType": "com.solace.samples.serdes.jsonschema.User",
-  "additionalProperties": false,
-  "properties": {
-    "name": {
-      "description": "Name of the user",
-      "type": "string"
-    },
-    "id": {
-      "description": "Id of the user",
-      "type": "string"
-    },
-    "email": {
-      "description": "Email of the user",
-      "type": "string",
-      "format": "email"
-    }
-  },
-  "required": 
-  [  "name",
-    "id",
-    "email"
-  ]}
-
-

Let's build a Publisher and Consumer sample.

-
    -
  1. Open a command window or terminal and clone this GitHub repository: git clone https://github.com/SolaceSamples/solace-samples-rest-messaging
    2. -
  2. Run the command to build the sample ./gradlew build. You should see a BUILD SUCCESSFUL message. Note: For windows users, use the gradlew.bat file instead of gradlew
    3. -
  3. Run the sample application and provide the broker connection details like: ./gradlew runJsonSchemaRestPublisherHttpClient --args="brokerUrl 38080"
    4. -
-

This sample serializes and publishes by doing the following: Configures the Serializer:

-
// Create and configure Json serializer
- try (Serializer<JsonNode> serializer = new JsonSchemaSerializer<>()) {
-                serializer.configure(getConfig());
-
-
 /**
-     * Gets the configuration for the serializer.
-     * @return A map of configuration properties.
-     */
-    private static Map<String, Object> getConfig() {
-        Map<String, Object> config = new HashMap<>();
-        config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL);
-        config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME);
-        config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD);
-        // This configuration property will populate the SERDES header with a schema ID that is of type String
-        config.put(SerdeProperties.SCHEMA_HEADER_IDENTIFIERS, SchemaHeaderId.SCHEMA_ID_STRING);
-        return config;
-
-

Serializes the HTTP message with payload and headers:

-
 * Serializes the HTTP message with the given payload and headers.
- Map<String, Object> headers = new HashMap<>();
-        byte[] payloadBytes = serializer.serialize(topic, payload, headers);
-
-        for (String key : headers.keySet()) {
-            Object value = headers.get(key);
-            // No Integer/Long to String mapping is needed, as the SERDES schema ID header is already configured as a String.
-            // The type parameter is optional. By default all Solace User Properties in a REST message are assumed to have a type "string"
-            httpBuilder.header(String.format("%s%s", "Solace-User-Property-", key), value.toString());
-        }
-
-        if ("JSON".equals(CONTENT_TYPE)) {
-            httpBuilder.header("Content-Type", "application/json");
-        } else if ("BINARY".equals(CONTENT_TYPE)) {
-            httpBuilder.header("Content-Type", "application/octet-stream");
-        } else {
-            httpBuilder.header("Content-Type", CONTENT_TYPE);
-        }
-        return httpBuilder.POST(HttpRequest.BodyPublishers.ofByteArray(payloadBytes));
-
-
-

Publishes the message to specific topic:

-
//Publishes a message to a specific topic
-   ObjectNode user = new ObjectMapper().createObjectNode();
-        try {
-            String url = String.format("http://%s:%d/TOPIC/%s", brokerHost, port, topic);
-            HttpRequest.Builder httpBuilder = HttpRequest.newBuilder().uri(URI.create(url));
-
-            user.put("name", "John Doe");
-            user.put("id", id);
-            user.put("email", "support@solace.com");
-
-            HttpRequest request = serializeHttpMessage(httpBuilder, serializer, topic, user).build();
-
-            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
-
-            System.out.println("Published message with status: " + response.statusCode());
-            System.out.println("Published message with record: " + user);
-
-
    -
  1. For a Consumer Application, create a queue subscribed to solace/samples/json and configure RDP client as mentioned above. REST consumer samples start a local HTTP server that listens for incoming POST requests from a Solace broker's REST Delivery Point (RDP).
    2. -
-

Configure Solace Rest Delivery Point 1. Create Queue

-

Create queue 2. Create Rest Delivery Point Rest Delivery Point 3. Create REST Consumer REST Consumer 4. Create Queue Binding Queue Binding 5. Specify Queue Binding Endpoint Queue Binding Endpoint Make sure all of the RDP components are enabled.

-
    -
  1. Run the consumer sample like: ./gradlew runJsonSchemaRestConsumer --args="/my-rest-endpoint 8080 X-Solace-Topic"
    2. -
-

Configures the Deserializer:

-
// Create and configure Json deserializer
-Deserializer<User> deserializer = new JsonSchemaDeserializer<>()) {
-            deserializer.configure(getDeserializerConfig());
-
-
Returns a map of configuration properties for the deserializer.
-     *
-     * @return a map of configuration properties for the deserializer
-     */
-    private static Map<String, Object> getDeserializerConfig() {
-        HashMap<String,Object> config = new HashMap<>();
-        config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL);
-        config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME);
-        config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD);
-        config.put(JsonSchemaProperties.TYPE_PROPERTY, "customJavaType");
-        return config;
-
-

It will receive the published message from http headers and deserialize it and retrieving the schema and print it out:

-
   // Handles a SERDES message.
-            Headers httpHeaders = exchange.getRequestHeaders();
-
-            // Read the message body
-            InputStream is = exchange.getRequestBody();
-            byte[] messagePayload = is.readAllBytes();
-
-            // extract SERDES headers from http headers
-            Map<String, Object> serdesHeaders = getSerdesHeaders(httpHeaders);
-            // deserialize with topic, payload and SERDES headers
-            T object = deserializer.deserialize(optionalTopic.orElse(""), messagePayload, serdesHeaders);
-
-            System.out.printf("%n- - - - - - - - - - RECEIVED SERDES MESSAGE - - - - - - - - - -%n");
-            printAllHeaders(httpHeaders);
-            System.out.println("BodyBytesLength: " + messagePayload.length);
-            System.out.println("Body: " + objectToStringFunc.apply(object));
-            System.out.printf("- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -%n");
-
- - - - - -

As your data model evolves, you'll need to update your schemas. Here are some best practices:

-
    -
  1. Backward Compatibility: Ensure new schema versions can read old data.
    2. -
  2. Forward Compatibility: Ensure old schema versions can read new data (ignoring new fields).
    3. -
  3. Full Compatibility: Aim for both backward and forward compatibility.
    4. -
  4. Versioning: Use semantic versioning for your schemas.
    5. -
  5. Default Values: Provide default values for new fields to maintain backward compatibility.
    6. -
  6. Avoid Renaming: Instead of renaming fields, add new fields and deprecate old ones.
    7. -
-

When updating a schema in Solace Schema Registry:

-
    -
  1. Create a new version of the existing schema.
    2. -
  2. Make your changes, following the best practices above.
    3. -
  3. Use the compatibility rule in the registry to ensure your changes don't break existing consumers.
    4. -
- - - - - -

✔️ Schema registries are crucial for maintaining data consistency in event-driven architectures.
✔️ Solace Schema Registry provides a powerful platform for managing schemas.
✔️ Solace's JCSMP API can be used effectively with the Solace Schema Registry for robust event-driven applications.
✔️ Proper schema evolution practices ensure smooth updates without breaking existing systems.
✔️ Using schemas in your applications helps catch data issues early and improves overall system reliability.
✔️ Solace Schema Registry supports built-in authentication as well as external identity providers for enterprise environments. ✔️ Solace Schema Registry currently supports Avro and JSON Schema. More schema SERDES will be made available in the future.

-

Soly Image Caption

-

Thanks for participating in this codelab! Let us know what you thought in the Solace Community Forum! If you found any issues along the way we'd appreciate it if you'd raise them by clicking the Report a mistake button at the bottom left of this codelab.

- - - - - - - - - - - - - - diff --git a/markdown/schema-registry-beta/img/EnvFile.png b/markdown/schema-registry-beta/img/EnvFile.png deleted file mode 100644 index b0a09c62..00000000 Binary files a/markdown/schema-registry-beta/img/EnvFile.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/ExternalAuthEnv.png b/markdown/schema-registry-beta/img/ExternalAuthEnv.png deleted file mode 100644 index 2247fa54..00000000 Binary files a/markdown/schema-registry-beta/img/ExternalAuthEnv.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrCreateArtifact.png b/markdown/schema-registry-beta/img/SrCreateArtifact.png deleted file mode 100644 index 0d29a185..00000000 Binary files a/markdown/schema-registry-beta/img/SrCreateArtifact.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrCreateArtifact1.png b/markdown/schema-registry-beta/img/SrCreateArtifact1.png deleted file mode 100644 index 781a593c..00000000 Binary files a/markdown/schema-registry-beta/img/SrCreateArtifact1.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrCreateArtifact2.png b/markdown/schema-registry-beta/img/SrCreateArtifact2.png deleted file mode 100644 index c1d14c79..00000000 Binary files a/markdown/schema-registry-beta/img/SrCreateArtifact2.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrCreateArtifact3.png b/markdown/schema-registry-beta/img/SrCreateArtifact3.png deleted file mode 100644 index 008cdbb0..00000000 Binary files a/markdown/schema-registry-beta/img/SrCreateArtifact3.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrCreateArtifactUser1.png b/markdown/schema-registry-beta/img/SrCreateArtifactUser1.png deleted file mode 100644 index f3d7df90..00000000 Binary files a/markdown/schema-registry-beta/img/SrCreateArtifactUser1.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrCreateArtifactUser2.png b/markdown/schema-registry-beta/img/SrCreateArtifactUser2.png deleted file mode 100644 index 2d933ea1..00000000 Binary files a/markdown/schema-registry-beta/img/SrCreateArtifactUser2.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrLogin.png b/markdown/schema-registry-beta/img/SrLogin.png deleted file mode 100644 index a719f208..00000000 Binary files a/markdown/schema-registry-beta/img/SrLogin.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrLoginEmpty.png b/markdown/schema-registry-beta/img/SrLoginEmpty.png deleted file mode 100644 index 6df9c035..00000000 Binary files a/markdown/schema-registry-beta/img/SrLoginEmpty.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/SrPackageFolderView.png b/markdown/schema-registry-beta/img/SrPackageFolderView.png deleted file mode 100644 index 0f6e4875..00000000 Binary files a/markdown/schema-registry-beta/img/SrPackageFolderView.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/schemaRegistryZip.png b/markdown/schema-registry-beta/img/schemaRegistryZip.png deleted file mode 100644 index 21e9163b..00000000 Binary files a/markdown/schema-registry-beta/img/schemaRegistryZip.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/serdesQueue.png b/markdown/schema-registry-beta/img/serdesQueue.png deleted file mode 100644 index d699c1dd..00000000 Binary files a/markdown/schema-registry-beta/img/serdesQueue.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/serdesQueueBinding.png b/markdown/schema-registry-beta/img/serdesQueueBinding.png deleted file mode 100644 index 20f761e7..00000000 Binary files a/markdown/schema-registry-beta/img/serdesQueueBinding.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/serdesQueueBindingEndpoint.png b/markdown/schema-registry-beta/img/serdesQueueBindingEndpoint.png deleted file mode 100644 index 0a7e7911..00000000 Binary files a/markdown/schema-registry-beta/img/serdesQueueBindingEndpoint.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/serdesRDP.png b/markdown/schema-registry-beta/img/serdesRDP.png deleted file mode 100644 index 6cc60987..00000000 Binary files a/markdown/schema-registry-beta/img/serdesRDP.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/serdesRESTConsumer.png b/markdown/schema-registry-beta/img/serdesRESTConsumer.png deleted file mode 100644 index 0e18c414..00000000 Binary files a/markdown/schema-registry-beta/img/serdesRESTConsumer.png and /dev/null differ diff --git a/markdown/schema-registry-beta/img/solly_wave.webp b/markdown/schema-registry-beta/img/solly_wave.webp deleted file mode 100644 index 33949cbc..00000000 Binary files a/markdown/schema-registry-beta/img/solly_wave.webp and /dev/null differ diff --git a/markdown/schema-registry-beta/img/soly.gif b/markdown/schema-registry-beta/img/soly.gif deleted file mode 100644 index 5364f91e..00000000 Binary files a/markdown/schema-registry-beta/img/soly.gif and /dev/null differ diff --git a/markdown/schema-registry-beta/package.json b/markdown/schema-registry-beta/package.json deleted file mode 100644 index dc301e54..00000000 --- a/markdown/schema-registry-beta/package.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "name": "codelab-bootstrap", - "version": "1.0.0", - "description": "use nodemon to watch changes on md file, rebuild codelab on saves", - "main": "index.js", - "scripts": { - "watch": "nodemon --watch schema-registry-beta.md --exec \"claat export -o temp/ schema-registry-beta.md ; ./node_modules/kill-port/cli.js --port 9090 ; cd temp/schema-registry-beta ; claat serve\"" - }, - "repository": { - "type": "git", - "url": "https://github.com/SolaceDev/solace-dev-codelabs/tree/master/markdown/schema-registry-beta" - }, - "keywords": [ - "codelab" - ], - "dependencies": { - "kill-port": "^1.6.1", - "nodemon": "^2.0.3" - } -} diff --git a/markdown/schema-registry-beta/schema-registry.md b/markdown/schema-registry-beta/schema-registry.md deleted file mode 100644 index 307786b4..00000000 --- a/markdown/schema-registry-beta/schema-registry.md +++ /dev/null @@ -1,521 +0,0 @@ -author: Supreet Mann -summary: This codelab will walk you through getting started with the Solace Schema Registry and the Solace SERDES Collection using the Solace Messaging API for Java (JCSMP) or with REST messaging. -id: schema-registry -tags: -categories: Solace -environments: Web -status: Published -feedback link: https://github.com/SolaceDev/solace-dev-codelabs/blob/master/markdown/schema-registry - -# Introduction to Solace Schema Registry using the Solace Messaging API for Java (JCSMP) and REST Messaging - -## What you'll learn: Overview - -Duration: 0:01:00 - -In today's data-driven world, ensuring data consistency and interoperability across different systems is crucial. This is where a schema registry comes into play. In this walkthrough, we'll explore how to use Solace Schema Registry with the Solace JCSMP API or REST applications, focusing on the JSON format for schema definition. - -You'll learn about: - -✔️ What a schema registry is and why it's important -✔️ What a Serializer and Deserializer (SERDES) is and the role they play -✔️ How to deploy and configure Solace Schema Registry using either Docker Compose or Kubernetes -✔️ How to create and manage schema artifacts in Solace Schema Registry web console -✔️ How to use schemas in your event-driven applications with Solace's JCSMP API or with REST messaging -✔️ Best practices for schema evolution - - -## What you need: Prerequisites - -Duration: 0:01:00 - -1. A general understanding of [event-driven architecture (EDA) terms and concepts](https://docs.solace.com/#Messaging). -2. A locally running Solace Broker or a free trial account of Solace Cloud. Don't have one? [Sign up here.](https://console.solace.cloud/login/new-account) - * Along with the connection information for the broker -3. Basic knowledge of [JSON schema format](https://json-schema.org/draft-07) -4. [Docker](https://docs.docker.com/get-started/get-docker/) installed on your system (for standalone deployment) -5. [Kubernetes](https://kubernetes.io/docs/tasks/tools/) with [Helm](https://helm.sh/docs/intro/install/) (for high availability deployments) -6. [Java Development Kit (JDK) version 11+](https://openjdk.org/) installed on your system -7. An IDE of your choice (e.g., IntelliJ IDEA, Eclipse, Visual Studio Code) -8. Download the provided GA tarball package named ```schema-registry-v1.0.0.tar.gz``` that contains all the necessary pieces you will need. This is available on the Solace Product Portal. - -NOTE: If you cannot access the [Solace Product Portal](https://products.solace.com/prods/Schema_Registry/solace-schema-registry), please click the ```Report a mistake``` at the bottom left of the codelab and open an issue asking for access. - - -## What Is A Schema Registry - -Duration: 0:02:00 - -A schema registry is a central repository for managing and storing schemas. It helps ensure data consistency, enables data governance, and supports schema evolution. Here's why it's important: - -1. **Data Consistency**: Ensures that producers and consumers agree on the data format. -2. **Interoperability**: Allows different systems to communicate effectively. -3. **Schema Evolution**: Supports versioning and compatibility checks as schemas change over time. -4. **Data Governance**: Centralizes schema management for better control and auditing. - -## What Is A SERDES - -Duration: 0:02:00 - -A SERDES (Serializer/Deserializer) in the context of a schema registry is a component that handles two key functions: -1. **Serialization**: Converting data objects from their native format (like Java objects) into a binary format suitable for transmission or storage. -2. **Deserialization**: Converting the binary data back into its original format for processing. - -In a schema registry system, SERDES works closely with schemas to: -- Ensure data consistency during serialization/deserialization -- Validate that messages conform to the registered schema - -For example: -- A serializer might take a Java object and convert it to binary format using a schema from the registry -- A deserializer would then use the same schema to correctly reconstruct the object from the binary data - -This is a fundamental concept in message-based systems where data needs to be: -- Efficiently transmitted between different services -- Properly validated against defined schemas -- Correctly interpreted by different applications that might be written in different programming languages - -> aside positive -> In this walkthrough, we'll use the Solace JSON Schema SERDES for Java along with the Solace Messaging API for Java (JCSMP) to serialize and deserialize messages in the JSON format. We will also how how to leverage REST messaging to serialize and deserialize messages. - -## Setting up Solace Schema Registry with Docker - -Duration: 0:06:00 - -We'll use Docker Compose to set up the Solace Schema Registry, for local development or standalone use cases with minimal resource and networking requirements. We've prepared a Docker Compose file that will launch an instance of the Solace Schema Registry and all the necessary components with a pre-defined configuration. - -1. In these subsequent steps we will use the package that came from the downloaded tarball package from the prerequisites section. Navigate to the extracted folder called ```Schema-Registry-V1.0```. You should see the following files and folders: -

- -

- - -2. Open a terminal or command prompt window and navigate to the extracted location of the folder called ```solace-schema-registry-dist```. -3. Run the command ```for img in {docker-images}/*.tar.gz; do docker load -i "$img"; done``` to load docker images. For Window use this instead ```for %i in (docker-images\*.tar.gz \*.tar.gz) do docker load -i "%i"```. -4. You can make changes to the ```.env``` file to change things such as default login or ports. While using the built-in identity provider, we can leave everything to defaults for this codelab. - -

- -

- -5. Run the following command and all the components will start up with the specified values configured: ```docker compose -f compose.yaml -f compose.nginx.yaml -f compose.nginx.for.embedded.yaml -f compose.embedded.yaml up -d```. - -6. Once the script is done running, you should now be able to go to your browser and navigate to ```localhost:8888``` which should re-direct you to the Solace Schema Registry login screen. - -

- -

- -That's it, you have now installed an instance of the Solace Schema Registry! - -Alternatively, for enterprise-grade security features, Solace Schema Registry supports external identity providers. For that, make the necessary configurations in your IdP and set the environment variables in your ```.env``` file: - -

- -

- -Run the following command ```docker compose -f compose.yaml -f compose.nginx.yaml -f compose.nginx.for.external.yaml -f compose.external.multiple.issuers.yaml up -d```. - -## Setting up Solace Schema Registry with Kubernetes - -Duration: 0:06:00 - -Alternatively, we can set up Solace Schema Registry on a Kubernetes cluster using Helm. - -NOTE: You must have an existing Kubernetes cluster available (for example, Amazon EKS, Google GKE, Microsoft AKS, or an on-prem/self-managed cluster). - -1. Configure ```kubectl``` to communicate with your cluster. -2. In this method of deployment also, we will use the package that came from the downloaded tarball package from the prerequisites section. Navigate to the extracted folder called ```Schema-Registry-V1.0```. -3. Open the commonad and run this command to load docker images at your preferred location: - -```bash -export REGISTRY="your-registry.com/project" -for img in {docker-images}/*.tar.gz; do - LOADED=$(docker load -i "$img" | grep "Loaded image:" | cut -d' ' -f3) - NEW_NAME="$REGISTRY/$(basename "$img" .tar.gz)" - docker tag "$LOADED" "$NEW_NAME" - docker push "$NEW_NAME" -done -``` - -4. Install the CloudNative PostgreSQL Operator for database management: -```kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.26/releases/cnpg-1.26.0.yaml```. -5. Update a values.yaml file with your environment-specific configuration, database, authentication, ingress and TLS configuration: -Insert image -6. Install Solace Schema Registry using Helm: ```helm upgrade --install schema-registry ./solace-schema-registry``` -7. Verify the deployment: ```kubectl get pods -n solace``` - -8. To access the deployed services, replace with the actual hostname or IP address you configured for your ingress: ```https://ui.``` - -## Creating and Registering Schemas - -Duration: 0:04:00 - -Let's create a simple schema for a ```clock-in-out``` event: - -1. Open the Solace Schema Registry UI in your web browser by going to ```localhost:8888```. -2. Login with the predefined credentials for a developer. In this case the username is ```sr-developer``` and password is ```devPassword```. - -

- -

- -3. Click on ```Create artifact``` button. Once the dialogue opens enter the following as shown below: - * Group Id: Leave it empty (default) - * Artifact Id: Set to ```solace/samples/clock-in-out/json``` - * Type: Set to ```JSON Schema``` - -

- -

-

- -

- -4. We will skip the ```Artifact Metadata``` step as it is optional and will click ```Next```. - -5. For the "Version Content" section, copy the Json schema from below and either paste directly or save it into a file and upload it and click ```Next``` when done. -```json -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "clock-in-out", - "type": "object", - "additionalProperties": false, - "properties": { - "region_code": { - "description": "region code for clock in or out", - "type": "string" - }, - "store_id": { - "description": "Store identifier", - "type": "string" - }, - "employee_id": { - "description": "Employee ID for who clocked in or out", - "type": "string" - }, - "datetime": { - "description": "Clock time", - "type": "string", - "format": "date-time" - } - }, - "required": [ - "region_code", - "store_id", - "employee_id", - "datetime" - ] -} -``` - -

- -

- -6. We will skip the ```Version Metadata``` step as it is optional and will simply click the ```Create``` button. - -7. Finally after you have successfully created the new schema you should see the following: - -

- -

- -You've now created and registered your first schema! - -## Using Schemas with the Solace Messaging API for Java (JCSMP) - -Duration: 0:10:00 - -Now, let's see how to use this schema in Java using the Solace Messaging API for Java (JCSMP): - -1. Open a command window or terminal and clone this GitHub repository, and open the ```solace-samples-java-jcsmp``` directory. With `git clone https://github.com/SolaceSamples/solace-samples-java-jcsmp` and `cd solace-samples-java-jcsmp` - -2. Run the command ```./gradlew assemble``` to build the sample. - -3. Open the ```build/staged``` directory by running ```cd build/staged``` and run the sample application with the broker connection details: -```bin/HelloWorldJCSMPJsonSchemaSerde localhost:55555 default default``` - -This sample talks to the locally deployed Solace Schema Registry and retrieves the schema along with the schema ID. It will then do the following: - -Configures the Serializer and Deserializer: -```java -// Create and configure JSON Schema serializer and deserializer - try (Serializer serializer = new JsonSchemaSerializer<>(); - Deserializer deserializer = new JsonSchemaDeserializer<>()) { - - serializer.configure(getConfig()); - deserializer.configure(getConfig()); -``` -```java - /** - * Returns a configuration map for the JSON Schema serializer and deserializer. - * - * @return A Map containing configuration properties - */ - private static Map getConfig() { - Map config = new HashMap<>(); - config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL); - config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME); - config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD); - return config; - } - -``` - - -Serializes the message payload and publishes the message to the connected broker on ```solace/samples/clock-in-out/json``` destination with the serialized payload and schema ID: -```java - // Create and populate a ClockInOut JsonNode with sample data - ObjectMapper mapper = new ObjectMapper(); - ObjectNode clockInOut = mapper.createObjectNode(); - clockInOut.put("region_code", "NA-WEST"); - clockInOut.put("store_id", "STORE-001"); - clockInOut.put("employee_id", "EMP-12345"); - clockInOut.put("datetime", "2025-01-20T15:30:00Z"); - - // Serialize and send the message - BytesMessage msg = JCSMPFactory.onlyInstance().createMessage(BytesMessage.class); - SerdeMessage.serialize(serializer, topic, msg, clockInOut); - System.out.printf("Sending ClockInOut Message:%n%s%n", msg.dump()); - producer.send(msg, topic); -``` - -It will then receive the published message and deserialize it by looking up the schema ID and retrieving the schema and print it out to console: -```java - // Set up the message consumer with a deserialization callback - XMLMessageConsumer cons = session.getMessageConsumer(Consumed.with(deserializer, (msg, clockInOutData) -> { - System.out.printf("Got a ClockInOut JsonNode: %s%n", clockInOutData); - System.out.printf("Employee %s clocked in/out at store %s in region %s at %s%n", - clockInOutData.get("employee_id").asText(), - clockInOutData.get("store_id").asText(), - clockInOutData.get("region_code").asText(), - clockInOutData.get("datetime").asText()); - latch.countDown(); // Signal the main thread that a message has been received - }, (msg, deserializationException) -> { - System.out.printf("Got exception: %s%n", deserializationException); - System.out.printf("But still have access to the message: %s%n", msg.dump()); - latch.countDown(); - }, jcsmpException -> { - System.out.printf("Got exception: %s%n", jcsmpException); - latch.countDown(); - })); - cons.start(); -``` - -The sourcecode can be further looked at by opening the ```HelloWorldJCSMPJsonSchemaSerde.java``` file. - -## Using schemas with REST messaging - -Duration: 0:10:00 - -We can also use this schema with Solace Schema Registry in REST-based messaging environments. - -> aside positive -> Before running the REST samples, you need to configure the Solace broker with the appropriate queues and REST Delivery Points (RDPs). -For more detailed documentation, refer to the [Solace Documentation on REST Delivery Points](https://docs.solace.com/Services/Managing-RDPs.htm?Highlight=rest#configuring-REST-delivery-points). - -With this REST messaging, we will use the following Schema example. You can upload it on Solace Schema Registry as a new artifact with an `Artifact Id` of `solace/samples/json` - Refer to Step 7 Creating and Registering Schemas if you need to review the steps. - -```json -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "User", - "type": "object", - "customJavaType": "com.solace.samples.serdes.jsonschema.User", - "additionalProperties": false, - "properties": { - "name": { - "description": "Name of the user", - "type": "string" - }, - "id": { - "description": "Id of the user", - "type": "string" - }, - "email": { - "description": "Email of the user", - "type": "string", - "format": "email" - } - }, - "required": - [ "name", - "id", - "email" - ]} -``` - -Let's build a Publisher and Consumer sample. - -1. Open a command window or terminal and clone this GitHub repository: -```git clone https://github.com/SolaceSamples/solace-samples-rest-messaging``` - -2. Run the command to build the sample ```./gradlew build```. You should see a ```BUILD SUCCESSFUL``` message. -Note: For windows users, use the `gradlew.bat` file instead of `gradlew` - -3. Run the sample application and provide the broker connection details like: -```./gradlew runJsonSchemaRestPublisherHttpClient --args="brokerUrl 38080"``` - -This sample serializes and publishes by doing the following: -Configures the Serializer: -```java -// Create and configure Json serializer - try (Serializer serializer = new JsonSchemaSerializer<>()) { - serializer.configure(getConfig()); -``` -```java - /** - * Gets the configuration for the serializer. - * @return A map of configuration properties. - */ - private static Map getConfig() { - Map config = new HashMap<>(); - config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL); - config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME); - config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD); - // This configuration property will populate the SERDES header with a schema ID that is of type String - config.put(SerdeProperties.SCHEMA_HEADER_IDENTIFIERS, SchemaHeaderId.SCHEMA_ID_STRING); - return config; -``` - -Serializes the HTTP message with payload and headers: -```java - * Serializes the HTTP message with the given payload and headers. - Map headers = new HashMap<>(); - byte[] payloadBytes = serializer.serialize(topic, payload, headers); - - for (String key : headers.keySet()) { - Object value = headers.get(key); - // No Integer/Long to String mapping is needed, as the SERDES schema ID header is already configured as a String. - // The type parameter is optional. By default all Solace User Properties in a REST message are assumed to have a type "string" - httpBuilder.header(String.format("%s%s", "Solace-User-Property-", key), value.toString()); - } - - if ("JSON".equals(CONTENT_TYPE)) { - httpBuilder.header("Content-Type", "application/json"); - } else if ("BINARY".equals(CONTENT_TYPE)) { - httpBuilder.header("Content-Type", "application/octet-stream"); - } else { - httpBuilder.header("Content-Type", CONTENT_TYPE); - } - return httpBuilder.POST(HttpRequest.BodyPublishers.ofByteArray(payloadBytes)); - -``` - -Publishes the message to specific topic: -```java -//Publishes a message to a specific topic - ObjectNode user = new ObjectMapper().createObjectNode(); - try { - String url = String.format("http://%s:%d/TOPIC/%s", brokerHost, port, topic); - HttpRequest.Builder httpBuilder = HttpRequest.newBuilder().uri(URI.create(url)); - - user.put("name", "John Doe"); - user.put("id", id); - user.put("email", "support@solace.com"); - - HttpRequest request = serializeHttpMessage(httpBuilder, serializer, topic, user).build(); - - HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); - - System.out.println("Published message with status: " + response.statusCode()); - System.out.println("Published message with record: " + user); -``` - -4. For a Consumer Application, create a queue subscribed to `solace/samples/json` and configure RDP client as mentioned above. REST consumer samples start a local HTTP server that listens for incoming POST requests from a Solace broker's REST Delivery Point (RDP). -
- Configure Solace Rest Delivery Point - 1. Create Queue - - ![Create queue](img/serdesQueue.png) - 2. Create Rest Delivery Point - ![Rest Delivery Point](img/serdesRDP.png) - 3. Create REST Consumer - ![REST Consumer](img/serdesRESTConsumer.png) - 4. Create Queue Binding - ![Queue Binding](img/serdesQueueBinding.png) - 5. Specify Queue Binding Endpoint - ![Queue Binding Endpoint](img/serdesQueueBindingEndpoint.png) - Make sure all of the RDP components are enabled. -
- - -5. Run the consumer sample like: ```./gradlew runJsonSchemaRestConsumer --args="/my-rest-endpoint 8080 X-Solace-Topic"``` - -Configures the Deserializer: -```java -// Create and configure Json deserializer -Deserializer deserializer = new JsonSchemaDeserializer<>()) { - deserializer.configure(getDeserializerConfig()); -``` -```java -Returns a map of configuration properties for the deserializer. - * - * @return a map of configuration properties for the deserializer - */ - private static Map getDeserializerConfig() { - HashMap config = new HashMap<>(); - config.put(SchemaResolverProperties.REGISTRY_URL, REGISTRY_URL); - config.put(SchemaResolverProperties.AUTH_USERNAME, REGISTRY_USERNAME); - config.put(SchemaResolverProperties.AUTH_PASSWORD, REGISTRY_PASSWORD); - config.put(JsonSchemaProperties.TYPE_PROPERTY, "customJavaType"); - return config; -``` - -It will receive the published message from http headers and deserialize it and retrieving the schema and print it out: -```java - // Handles a SERDES message. - Headers httpHeaders = exchange.getRequestHeaders(); - - // Read the message body - InputStream is = exchange.getRequestBody(); - byte[] messagePayload = is.readAllBytes(); - - // extract SERDES headers from http headers - Map serdesHeaders = getSerdesHeaders(httpHeaders); - // deserialize with topic, payload and SERDES headers - T object = deserializer.deserialize(optionalTopic.orElse(""), messagePayload, serdesHeaders); - - System.out.printf("%n- - - - - - - - - - RECEIVED SERDES MESSAGE - - - - - - - - - -%n"); - printAllHeaders(httpHeaders); - System.out.println("BodyBytesLength: " + messagePayload.length); - System.out.println("Body: " + objectToStringFunc.apply(object)); - System.out.printf("- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -%n"); -``` - - - -## Best Practices for Schema Evolution - -Duration: 0:02:00 - -As your data model evolves, you'll need to update your schemas. Here are some best practices: - -1. **Backward Compatibility**: Ensure new schema versions can read old data. -2. **Forward Compatibility**: Ensure old schema versions can read new data (ignoring new fields). -3. **Full Compatibility**: Aim for both backward and forward compatibility. -4. **Versioning**: Use semantic versioning for your schemas. -5. **Default Values**: Provide default values for new fields to maintain backward compatibility. -6. **Avoid Renaming**: Instead of renaming fields, add new fields and deprecate old ones. - -When updating a schema in Solace Schema Registry: - -1. Create a new version of the existing schema. -2. Make your changes, following the best practices above. -3. Use the compatibility rule in the registry to ensure your changes don't break existing consumers. - -## Takeaways - -Duration: 0:02:00 - -✔️ Schema registries are crucial for maintaining data consistency in event-driven architectures. -✔️ Solace Schema Registry provides a powerful platform for managing schemas. -✔️ Solace's JCSMP API can be used effectively with the Solace Schema Registry for robust event-driven applications. -✔️ Proper schema evolution practices ensure smooth updates without breaking existing systems. -✔️ Using schemas in your applications helps catch data issues early and improves overall system reliability. -✔️ Solace Schema Registry supports built-in authentication as well as external identity providers for enterprise environments. -✔️ Solace Schema Registry currently supports Avro and JSON Schema. More schema SERDES will be made available in the future. - -![Soly Image Caption](img/soly.gif) - -Thanks for participating in this codelab! Let us know what you thought in the [Solace Community Forum](https://solace.community/)! If you found any issues along the way we'd appreciate it if you'd raise them by clicking the Report a mistake button at the bottom left of this codelab.