Real-time website activity tracking
Overview
I built this demo to show the whole Kafka path in one place. A small Java web shop sends activity events into Kafka. From there they go into an S3 bucket, and a second connector can read them back out of the same bucket.
Most tutorials stop after the first step. I wanted the part that comes after it, where the data is stored and replayed.
The whole stack starts from one Docker Compose file. It is a single node lab setup with no authentication and simple JSON messages. That is on purpose, because the point is to learn how the pieces fit.
Problem
Most Kafka tutorials show how to send events into a topic and stop there. I wanted a small demo that follows the whole path: produce events, look at them, save them to object storage and play them back from it. I also wanted it to start on any machine with one command.
Approach
- I wrote a small Java web shop that sends website activity events into Kafka.
- I put the whole pipeline into one Docker Compose file, so one command brings it all up.
- I added two connectors for the same S3 bucket, one that writes events out and one that reads them back in.
- I put the connector plugin into the repository, so the demo starts without a manual download.
- I added a schema registry and a topic viewer, so I can see what is happening inside.
- I kept the scope small on purpose, because the demo is for learning.
Architecture and design decisions
- A Maven and JDK 17 container builds and runs the shop on port 8080.
- Zookeeper and Kafka form the broker layer. Kafka Connect runs next to them with a mounted plugin folder.
- Apicurio Registry 3.0.6 and its web interface handle schemas. Kafdrop lets me look at the topics.
- The Connect worker is set up for a single node. The config, offset and status topics each have one replica.
- Keys use the string converter. Values use the JSON converter with schema support switched off, so the messages stay easy to read.
- Two connector files do the real work. The first is an S3 sink and the second is an S3 source. Together they show both halves of the same bucket.
- The connector is the Confluent S3 plugin, version 10.5.19.
Reliability and performance
- The plugin is kept in the repository, so the demo does not depend on a download at start.
- The replication factors are set on purpose for a single node, so Connect starts cleanly in a lab.
- The setup is the same on every machine, so the demo is easy to repeat.
Outcomes
- The demo covers the full path: produce events, inspect them, save them to S3 and replay them from S3.
- Both the sink and the source connector run on the same bucket, which most tutorials skip.
- Anyone can start the whole stack with one Docker Compose file.
- Schemas and topics are visible through the registry and the topic viewer.
Known gaps
- It is a lab demo. There is one broker, no authentication and JSON messages instead of Avro.
- It is not built for real traffic. I have no numbers for throughput or latency.
- I do not describe how I ran it against a real S3 service, or which one.
- I do not say what happens when a connector fails or a broker goes down.
- I do not say which licence the repository uses.
- The repository was first called website-activity-tracking-kafka, so older links may use that name.