Weibo v1.0.3 is out

Stream processing that runs inside your Go service

Weibo is an embeddable engine for stateful pipelines, event-time windows and exactly-once Kafka delivery. It runs in your process. No cluster, no JVM.

Get started
go get github.com/ASHUTOSH-SWAIN-GIT/weibo
main.go
env := weibo.NewEnv().
    WithCheckpointing(5*time.Second, checkpoint.NewFileStorage("./checkpoints"))

src := source.NewKafkaSource(
    source.KafkaBrokers("localhost:9092"),
    source.KafkaTopic("orders"),
    source.KafkaGroupID("order-totals"),
    source.KafkaExactlyOnce(),
)

out := sink.NewTxnKafkaSink(
    sink.TxnKafkaBrokers("localhost:9092"),
    sink.TxnKafkaTopic("order-totals"),
    sink.TxnKafkaTransactionalID("order-totals"),
)

env.FromSource(src).
    KeyBy(byCustomer).WithPartitions(4).
    Reduce(sumAmounts).
    ToSink(out)

env.Execute(ctx)
Pipeline

We killed it mid-run and checked every answer

A Kafka to keyed window to S3 pipeline ran at 10,000 events per second on one two-core machine. We stopped the job with SIGKILL, then compared each window total with the value computed independently from the input.

orders processed
4,414,010
window totals exact
7,000 of 7,000
from kill to running again
6.4 s

One machine, one Kafka broker, one kill per run. The benchmarks page lists what was not measured.

Everything a stateful pipeline needs, in one process

Keyed state on disk

Per-key state lives in Pebble, so it is bounded by disk instead of memory.

WithStateBackend(state.Pebble(dir))

Event-time windows

Tumbling, sliding and session windows, with watermarks and allowed lateness.

Window(window.NewTumbling(d))

Checkpoints and recovery

Barrier-based snapshots restore operator state and source offsets after a crash.

WithCheckpointing(interval, storage)

Exactly-once Kafka

Kafka to Kafka delivery through a transactional sink that commits with each checkpoint.

sink.NewTxnKafkaSink(...)

Backpressure

Bounded edges between stages slow a fast source down instead of dropping records.

WithBufferSize(n)

YAML and a dashboard

Describe common pipelines in YAML and watch every job in the built-in web dashboard.

weibo dashboard

Start where it fits