CDC from DynamoDB to ClickHouse

This page covers how set up CDC from DynamoDB to ClickHouse using ClickPipes. There are 2 components to this integration:

1. The initial snapshot via S3 ClickPipes
2. Real-time updates via Kinesis ClickPipes

Data will be ingested into a ReplacingMergeTree. This table engine is commonly used for CDC scenarios to allow update operations to be applied. More on this pattern can be found in the following blog articles:

• Change Data Capture (CDC) with PostgreSQL and ClickHouse - Part 1
• Change Data Capture (CDC) with PostgreSQL and ClickHouse - Part 2

1. Set up Kinesis stream​

First, you will want to enable a Kinesis stream on your DynamoDB table to capture changes in real-time. We want to do this before we create the snapshot to avoid missing any data. Find the AWS guide located here.