Skip to main content
Beta access required. Beam is in beta. Contact [email protected] to request access before you begin.
A Beam pipeline that streams Polygon logs through a set filter on address into a Kafka sink named usdc-logs

Create your first pipeline

1

Create a pipeline

In Allium App, open Build > Beam and choose New. A new pipeline streams Polygon logs through an empty set filter into a Kafka sink. Start with Template builds one from a template instead, and can deploy it in the same step.
2

Pick the source

Choose a chain and entity to stream, for example polygon and log for every Polygon log. Zero-lag streams from the chain tip before finality, for lower latency at the cost of possible reorged data.
Beam source set to the polygon chain and log entity
3

Filter the records

Edit the set filter, pick the field to match, and add the values to keep. To keep only USDC on Polygon, filter on address, the contract that emitted the log, and add 0x3c499c542cef5e3811e1192ce70d8cc03d5c3359. Values can be pasted or uploaded as a .txt or .csv file.
Set filter matching address against the allowed set
4

Name the sink

Name the Kafka sink, for example usdc-logs. The name becomes part of the topic you consume from.
Kafka sink named usdc-logs
5

Deploy

Save keeps the pipeline without running it. Deploy provisions your Kafka topic and credentials and starts the pipeline, usually in under 30 seconds.
6

Consume your data

Once deployed, Get cooking has the Kafka credentials, the topic, and consumer code with the credentials filled in.
Get cooking panel with Kafka credentials and the pipeline's topic
Replace the placeholders with the values from Get cooking. Start the consumer group ID with your group ID prefix.

Verify your deployment

A deployed pipeline previews the latest messages from each sink, and reports worker health, events processed, latency, and errors over the last hour, day, or week. Every worker should be healthy. If one isn’t, see Troubleshooting.
Deployment messages preview for a healthy pipeline's Kafka sink
Deployment health with one healthy worker, events processed and latency charts

Updating a pipeline

Edit the source, transforms, or sinks and deploy again. Deploys are idempotent and zero-downtime; changes apply in place.
Don’t tear down before redeploying. Tearing down causes unnecessary downtime.

Next steps

Configuration reference

Sources, transforms, and sinks

Examples

Ready-to-use pipeline configs for common use cases