Overview of Problem
Helios currently supports data ingestion only through Amazon Kinesis. This limitation restricts potential users who rely on other popular streaming platforms like Apache Kafka.
Solution
Expand Helios' data ingestion capabilities to include Apache Kafka. This can be done in two ways:
- Amazon MSK Integration:
Integrate with Amazon Managed Service for Apache Kafka. This will allow the team to leverage existing Lambda function logic for a quicker implementation.
- Direct Kafka Support:
The other approach would be to re-architect the Lambda Connector function to support direct Kafka integration. This would enable broader Kafka support beyond Amazon's managed service.
Supporting Links
https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink
Overview of Problem
Helios currently supports data ingestion only through Amazon Kinesis. This limitation restricts potential users who rely on other popular streaming platforms like Apache Kafka.
Solution
Expand Helios' data ingestion capabilities to include Apache Kafka. This can be done in two ways:
Integrate with Amazon Managed Service for Apache Kafka. This will allow the team to leverage existing Lambda function logic for a quicker implementation.
The other approach would be to re-architect the Lambda Connector function to support direct Kafka integration. This would enable broader Kafka support beyond Amazon's managed service.
Supporting Links
https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink