Keyed state on disk
Per-key state lives in Pebble, so it is bounded by disk instead of memory.
WithStateBackend(state.Pebble(dir))Weibo is an embeddable engine for stateful pipelines, event-time windows and exactly-once Kafka delivery. It runs in your process. No cluster, no JVM.
go get github.com/ASHUTOSH-SWAIN-GIT/weiboenv := weibo.NewEnv().
WithCheckpointing(5*time.Second, checkpoint.NewFileStorage("./checkpoints"))
src := source.NewKafkaSource(
source.KafkaBrokers("localhost:9092"),
source.KafkaTopic("orders"),
source.KafkaGroupID("order-totals"),
source.KafkaExactlyOnce(),
)
out := sink.NewTxnKafkaSink(
sink.TxnKafkaBrokers("localhost:9092"),
sink.TxnKafkaTopic("order-totals"),
sink.TxnKafkaTransactionalID("order-totals"),
)
env.FromSource(src).
KeyBy(byCustomer).WithPartitions(4).
Reduce(sumAmounts).
ToSink(out)
env.Execute(ctx)
A Kafka to keyed window to S3 pipeline ran at 10,000 events per second on one two-core machine. We stopped the job with SIGKILL, then compared each window total with the value computed independently from the input.
One machine, one Kafka broker, one kill per run. The benchmarks page lists what was not measured.
Per-key state lives in Pebble, so it is bounded by disk instead of memory.
WithStateBackend(state.Pebble(dir))Tumbling, sliding and session windows, with watermarks and allowed lateness.
Window(window.NewTumbling(d))Barrier-based snapshots restore operator state and source offsets after a crash.
WithCheckpointing(interval, storage)Kafka to Kafka delivery through a transactional sink that commits with each checkpoint.
sink.NewTxnKafkaSink(...)Bounded edges between stages slow a fast source down instead of dropping records.
WithBufferSize(n)Describe common pipelines in YAML and watch every job in the built-in web dashboard.
weibo dashboard