Skip to main content
Data flows through a Beam pipeline in three stages:
For field types, defaults, and every transform and sink variant, see the API configuration reference.

​
Source

Select a blockchain and entity to stream. Enable zerolag to stream from the chain tip before finality, for lower latency at the cost of possible reorged data. To see what’s supported, open the Beam page, click New, select the Source node, and expand the chain and entity menus. If what you need isn’t listed, check the Datastreams catalog and contact [email protected].

​
Transforms

Two transform types are available. Chain several together; records pass through them in order.

​
Set filter

Passes only records whose selected field matches a value in your set. Sets hold 10M+ values. In the UI, pick the Filter field from the source entity’s schema. In the API, set filter_expr instead:
Use lowercase values when filtering by addresses, labels, or symbols. Allium data stores these values in lowercase, and matching is exact.

​
Managing filter values

Filter values live outside the pipeline config. Add, remove, replace, and browse them in the UI or with the filter values API. This keeps config changes small and lets sets scale to millions of entries. In the UI, click Edit on a set filter node to open the filter set editor.
The Beam filter set editor includes a filter field picker and paginated values.
From the editor you can:
  • Browse values, 100 per page
  • Add a value with the text input
  • Remove a value with the trash icon
  • Check membership of any value instantly
  • Bulk upload a .txt or .csv file to add, remove, or replace values
The filter value membership check shows In the set.
Bulk upload supports three modes: add to set, remove from set, and replace the set.
Filter values persist across deploys and teardowns. They are deleted only when you delete the transform or the pipeline.

​
JavaScript

Modify, enrich, or reshape each record with a JavaScript function. Return null to drop the record. Example: tag each transfer by size.

​
Sinks

A sink is where the pipeline delivers records. Five types are available: Kafka, SNS, External Kafka, Webhook, and NATS JetStream. For every sink field, see the API configuration reference.

​
External Kafka sinks

Deliver records to your own Kafka cluster instead of an Allium-managed topic.
Beam editor with external Kafka sink
External Kafka configuration fields
Store your credentials as organization secrets, select the username and password secrets by name, and choose the Auth Method (PLAIN or SCRAM-SHA-512). Connections use SASL_SSL.
If a referenced secret changes after the pipeline was last deployed, Beam flags the sink as stale. Redeploy to apply the new value.

​
Webhook sinks

Deliver each record as a POST request to your HTTPS endpoint.
The webhook URL must use HTTPS. Plain HTTP is not supported.
To sign requests and guard against replays, see Webhook authentication. Set enable_http_encoding to true to compress each request body with zstd. Beam adds a Content-Encoding: zstd header, and your endpoint must decompress the body before parsing it. Tune delivery to what your endpoint can handle. In the Beam editor, these settings are under Optional fields on the webhook sink. Leave a field blank to use its default. retry_period and timeout take durations such as 500ms, 30s, or 1m30s. Lower max_in_flight if your endpoint rate-limits or slows under load. Raise timeout if it takes longer than 5 seconds to respond.

​
NATS JetStream sinks

Publish records to a JetStream subject on a NATS broker you operate. Set the broker URL and the subject. The subject can include field interpolation, such as ${! json("chain") }. Authentication is token-based. Store the token as an organization secret and reference it with token_secret_id. If you omit it, Beam connects without authentication.
Don’t put the token in the url. When token_secret_id is set, Beam adds the token to the connection at deploy time: with a token of your-token, nats://your-broker.example.com:4222 becomes nats://[email protected]:4222.
If the token secret is rotated after deployment, the sink is flagged as stale. Redeploy to apply the new value.
See the NATS JetStream sink reference for every field.

​
Best practices

  1. Filter early. Put set filters before JavaScript transforms to cut volume.
  2. Start simple. Begin with a filter and add transforms one at a time.
  3. Check health after deploying. Confirm all workers are healthy.
  4. Handle edge cases in JavaScript. Guard against missing fields and unexpected types.
  5. Name things clearly. Descriptive pipeline and sink names make many pipelines easier to manage.
  6. Use lowercase. Allium data stores addresses, labels, and symbols in lowercase, and matching is exact.
  7. Redeploy instead of tearing down. Redeploys are idempotent and zero-downtime.
  8. Manage filter values separately. Use the UI or filter values API to change values without touching the config.

​
Troubleshooting

​
Unhealthy workers

  1. Check that the config is valid: the source chain and entity exist and filter expressions are correct.
  2. Confirm data is available for the selected chain and entity.
  3. Fix the config and redeploy. You don’t need to tear down.

​
Workers running out of memory

  1. Simplify JavaScript transforms to use less memory.
  2. Filter more aggressively, earlier in the pipeline.
  3. Increase the pipeline’s pod_size, or contact [email protected].

​
Crashing workers

  1. Check JavaScript transforms for syntax errors.
  2. Check that filter expressions are valid.
  3. Look for runtime errors in transform logic, such as reading properties of undefined.