j-broker

A log-structured distributed message broker written from scratch in Java 21: a Raft-replicated metadata plane, partitioned replicated logs with high-watermark semantics and a min.insync.replicas durability floor, consumer groups, idempotent producers, transactions with exactly-once consume-transform-produce, log compaction, mTLS with per-principal ACLs, online cluster membership changes, a cluster-aware failover-transparent client, and a web admin UI. No Kafka libraries — the wire is gRPC, and the clients in this repository are the interface. Current release: v2.0.0-rc.1, Apache-2.0.

A producer and a consumer group, live.

Documentation

Install & deployDocker Compose, Kubernetes with Helm, published images/chart/jars, building from source, and the full broker configuration reference. ProducersThe batching idempotent producer, acks semantics, transactions, compression, and the delivery contract under failover. ConsumersConsumer groups, offset commits, rebalancing, read_committed isolation, and the CLI consumers. OperationsThe admin UI and REST API, cluster lifecycle (add/decommission/reassign), monitoring, alerts, and runbooks. SecuritymTLS on every gRPC hop, certificate-CN principals, default-deny ACLs, and operator login on the admin plane. ArchitectureDiagrams of the system: components, the Kubernetes topology, the produce path, transaction two-phase commit, leader failover, release artifacts.

Quickstart

Bring up a 3-broker cluster with the admin UI on http://localhost:15672:

docker compose up

Produce and consume from the CLI:

./gradlew :broker-app:installDist   # builds the CLI used below

./broker-app/build/install/broker-app/bin/broker-app topics create \
  --broker localhost:9092 --topic orders --partitions 3 --replication-factor 3

echo -e "o-1\no-2\no-3" | ./broker-app/build/install/broker-app/bin/broker-app \
  produce --broker localhost:9092 --topic orders --partition 0

./broker-app/build/install/broker-app/bin/broker-app consume \
  --broker localhost:9092 --group order-processor --topic orders

Topic creation must reach the controller and produce must reach the partition's leader; when you aim at the wrong broker, the CLI reads the redirect from the refusal and retries once against the right one on its own. Group consume works against any broker.

The full demo

One command boots the cluster, runs plain and transactional pipelines, kills a broker mid-transaction, audits exactly-once delivery, and leaves everything running for exploration:

scripts/demo/full-demo.sh

What each act shows and what to explore afterwards: scripts/demo/README.md. Add DEMO_MONITORING=1 for Prometheus and Grafana.