Configuration reference

Data flows through a Beam pipeline in three stages:

source → transforms[] → sinks[]

For field types, defaults, and every transform and sink variant, see the API configuration reference.

Source

Select a blockchain and entity to stream. Enable zerolag to stream from the chain tip before finality, for lower latency at the cost of possible reorged data. Every chain and entity Beam supports can be picked in the source. If what you need isn’t there, check the Datastreams catalog and contact [email protected].

Transforms

Two transform types are available. Chain several together; records pass through them in order.

Set filter

Set filters (redis_set_filter) are deprecated. New pipelines start with a JavaScript transform. Move your filter values to a dataset and check them with beam.contains() in a JavaScript transform instead.

Passes only records whose selected field matches a value in your set. Sets hold 10M+ values. In the UI, pick the Filter field from the source entity’s schema. In the API, set filter_expr instead:

root = this.address        # Contract address
root = this.topic0         # Event signature
root = this.from_address   # Sender address
root = this.to_address     # Recipient address

Use lowercase values when filtering by addresses, labels, or symbols. Allium data stores these values in lowercase, and matching is exact.

Managing filter values

Filter values live outside the pipeline config. Add, remove, replace, and browse them in the UI or with the filter values API. This keeps config changes small and lets sets scale to millions of entries. In Allium App, edit a set filter to pick its field, add or remove values, and check whether a value is in the set.

Set filter editor with address as the filter field and the USDC contract as its one allowed value

Upload a .txt or .csv file to add values to the set, remove them, or replace the set.

Bulk upload with add to set, remove from set and replace the set modes

Filter values persist across deploys and teardowns. They are deleted only when you delete the transform or the pipeline.

JavaScript

Modify, enrich, or reshape each record with a JavaScript function. Return null to drop the record. Example: tag each transfer by size.

function transform(record) {
  const amount = parseFloat(record.amount || "0");
  if (amount > 1000000) {
    record.size_tag = "whale";
  } else if (amount > 10000) {
    record.size_tag = "medium";
  } else {
    record.size_tag = "small";
  }
  return record;
}

Datasets

A dataset is a named list of values that JavaScript transforms check with beam.contains(), for a list that is large, changes often, or is shared across pipelines. Create one under Build > Beam > Datasets, then add or remove values by pasting them or uploading a .txt or .csv file. Changes apply to deployed pipelines without a redeploy.

A dataset with two contract addresses and a membership check

For beam.contains() and the datasets API, see Datasets.

Sinks

A sink is where the pipeline delivers records. Five types are available: Kafka, SNS, External Kafka, Webhook, and NATS JetStream. For every sink field, see the API configuration reference.

External Kafka sinks

Deliver records to your own Kafka cluster instead of an Allium-managed topic.

External Kafka sink with bootstrap servers, username and password secrets, and auth method

Store your credentials as organization secrets, select the username and password secrets by name, and choose the Auth Method (PLAIN or SCRAM-SHA-512). Connections use SASL_SSL.

If a referenced secret changes after the pipeline was last deployed, Beam flags the sink as stale. Redeploy to apply the new value.

Webhook sinks

Deliver each record as a POST request to your HTTPS endpoint.

The webhook URL must use HTTPS. Plain HTTP is not supported.

To sign requests and guard against replays, see Webhook authentication. Set enable_http_encoding to true to compress each request body with zstd. Beam adds a Content-Encoding: zstd header, and your endpoint must decompress the body before parsing it. Tune delivery to what your endpoint can handle. In the Beam editor, these settings are under Optional fields on the webhook sink. Leave a field blank to use its default.

Field	Default	What it controls
`max_in_flight`	`64`	Requests sent to your endpoint at once
`retries`	`3`	Retries for a failed request
`retry_period`	`1s`	Wait between retries
`timeout`	`5s`	Time allowed for each request

retry_period and timeout take durations such as 500ms, 30s, or 1m30s. Lower max_in_flight if your endpoint rate-limits or slows under load. Raise timeout if it takes longer than 5 seconds to respond.

NATS JetStream sinks

Publish records to a JetStream subject on a NATS broker you operate. Set the broker URL and the subject. The subject can include field interpolation, such as ${! json("chain") }. Authentication is token-based. Store the token as an organization secret and reference it with token_secret_id. If you omit it, Beam connects without authentication.

Don’t put the token in the url. When token_secret_id is set, Beam adds the token to the connection at deploy time: with a token of your-token, nats://your-broker.example.com:4222 becomes nats://[email protected]:4222.

If the token secret is rotated after deployment, the sink is flagged as stale. Redeploy to apply the new value.

See the NATS JetStream sink reference for every field.

Deployment settings

Each pipeline has a pod size and a delivery guarantee, applied the next time it deploys. Pod size runs from XXS to XL, and Beam recommends a size for the source.

Delivery guarantee	Behavior
At-least-once (default)	No message loss. The source is backpressured when the pipeline errors.
At-most-once	Messages can be lost. The source isn’t backpressured on pipeline errors.

Deployment settings with pod size and delivery guarantee

Best practices

Filter early. Return null early in your JavaScript transform to drop records you do not need.
Start simple. Begin with a filter and add transforms one at a time.
Check health after deploying. Confirm all workers are healthy.
Handle edge cases in JavaScript. Guard against missing fields and unexpected types.
Name things clearly. Descriptive pipeline and sink names make many pipelines easier to manage.
Use lowercase. Allium data stores addresses, labels, and symbols in lowercase, and matching is exact.
Redeploy instead of tearing down. Redeploys are idempotent and zero-downtime.
Manage filter values separately. Use the UI or filter values API to change values without touching the config.

Troubleshooting

Unhealthy workers

Check that the config is valid: the source chain and entity exist and filter expressions are correct.
Confirm data is available for the selected chain and entity.
Fix the config and redeploy. You don’t need to tear down.

Workers running out of memory

Simplify JavaScript transforms to use less memory.
Filter more aggressively, earlier in the pipeline.
Increase the pod size in deployment settings, or contact [email protected].

Crashing workers

Check JavaScript transforms for syntax errors.
Check that filter expressions are valid.
Look for runtime errors in transform logic, such as reading properties of undefined.

Queries

Dashboards

Your Data

Beam

Issuer Intelligence

Settings

Source

Transforms

Set filter

Managing filter values

JavaScript

Datasets

Sinks

External Kafka sinks

Webhook sinks

NATS JetStream sinks

Deployment settings

Best practices

Troubleshooting

Unhealthy workers

Workers running out of memory

Crashing workers

Queries

Dashboards

Your Data

Beam

Issuer Intelligence

Settings

​Source

​Transforms

​Set filter

​Managing filter values

​JavaScript

​Datasets

​Sinks

​External Kafka sinks

​Webhook sinks

​NATS JetStream sinks

​Deployment settings

​Best practices

​Troubleshooting

​Unhealthy workers

​Workers running out of memory

​Crashing workers

Source

Transforms

Set filter

Managing filter values

JavaScript

Datasets

Sinks

External Kafka sinks

Webhook sinks

NATS JetStream sinks

Deployment settings

Best practices

Troubleshooting

Unhealthy workers

Workers running out of memory

Crashing workers