Configuration reference

A Beam pipeline config describes the full data flow: where data comes from (source), how it’s processed (transforms), and where it goes (sinks).

source → transforms[] → sinks[]

BeamConfig

The top-level object returned by all config endpoints.

{
  "id": "abc123",
  "name": "USDC Transfer Monitor",
  "description": "Monitors USDC ERC20 transfers on Base",
  "tags": ["production", "base"],
  "static_egress_ip": false,
  "pod_size": "S",
  "delivery_guarantee": "at_least_once",
  "owner_org_team_user_id": "org_xyz",
  "created_at": "2025-01-15T10:30:00Z",
  "updated_at": "2025-01-15T10:30:00Z",
  "pipeline_config": {
    "source": { ... },
    "transforms": [ ... ],
    "sinks": [ ... ]
  }
}

Field	Type	Required	Description
`id`	`string`	Auto-generated	Pipeline ID. Omit when creating — the server generates it.
`name`	`string`	Yes	Pipeline name
`description`	`string`	Yes	Pipeline description
`tags`	`string[]`	No	Tags for organizing pipelines. Default: `null`
`static_egress_ip`	`boolean`	No	Enable a static outbound IP for webhook sinks. Default: `false`
`pod_size`	`string`	No	Worker pod size: `S`, `M`, `L`, or `XL`. Default: `S`
`delivery_guarantee`	`string`	No	Delivery mode. `at_least_once` gives at-least-once delivery: no message loss, the source is backpressured if the pipeline errors. `best_effort` gives at-most-once delivery: possible message loss, the source is not backpressured on pipeline errors. Default: `at_least_once`
`owner_org_team_user_id`	`string`	Auto-set	Owner ID, set from the authenticated API key
`created_at`	`datetime`	Auto-set	Creation timestamp
`updated_at`	`datetime`	Auto-set	Last update timestamp
`pipeline_config`	`PipelineConfig`	Yes	The pipeline’s source, transforms, and sinks

CreateBeamConfigRequest

Used as the request body for Create pipeline. Same as BeamConfig but without auto-generated fields.

{
  "name": "USDC Transfer Monitor",
  "description": "Monitors USDC ERC20 transfers on Base",
  "tags": ["production", "base"],
  "static_egress_ip": false,
  "pod_size": "S",
  "delivery_guarantee": "at_least_once",
  "pipeline_config": {
    "source": { ... },
    "transforms": [ ... ],
    "sinks": [ ... ]
  }
}

Field	Type	Required	Description
`name`	`string`	Yes	Pipeline name
`description`	`string`	Yes	Pipeline description
`tags`	`string[]`	No	Tags for organizing pipelines
`static_egress_ip`	`boolean`	No	Enable a static outbound IP. Default: `false`
`pod_size`	`string`	No	Worker pod size: `S`, `M`, `L`, or `XL`. Default: `S`
`delivery_guarantee`	`string`	No	Delivery mode. `at_least_once` gives at-least-once delivery: no message loss, the source is backpressured if the pipeline errors. `best_effort` gives at-most-once delivery: possible message loss, the source is not backpressured on pipeline errors. Default: `at_least_once`
`pipeline_config`	`PipelineConfig`	Yes	The pipeline’s source, transforms, and sinks

UpdateBeamConfigRequest

Used as the request body for Update pipeline. The shape is identical to CreateBeamConfigRequest — only the editable fields. The config_id is taken from the URL path, not the body. Server-managed fields (id, owner_org_team_user_id, organization_id, created_at, updated_at) are preserved from the existing record and ignored if present in the body.

{
  "name": "USDC Transfer Monitor (updated)",
  "description": "Updated description",
  "tags": ["production", "base"],
  "static_egress_ip": false,
  "pod_size": "S",
  "delivery_guarantee": "at_least_once",
  "pipeline_config": {
    "source": { ... },
    "transforms": [ ... ],
    "sinks": [ ... ]
  }
}

PipelineConfig

{
  "source": { ... },
  "transforms": [ ... ],
  "sinks": [ ... ]
}

Field	Type	Required	Description
`source`	`Source`	Yes	Where data comes from
`transforms`	`Transform[]`	Yes	Processing steps applied in order
`sinks`	`Sink[]`	Yes	Where processed data is delivered. Exactly one sink is allowed per pipeline

Source

Connects to Allium’s Datastreams. Select a blockchain and entity type.

{
  "type": "pubsub",
  "chain": "base",
  "entity": "erc20_token_transfer",
  "is_zerolag": false
}

Field	Type	Required	Description
`type`	`string`	Auto-set	Source type (set automatically based on chain)
`chain`	`string`	Yes	Blockchain to source data from
`entity`	`string`	Yes	Entity type to stream
`is_zerolag`	`boolean`	No	Stream from the tip of the blockchain before finality. Lower latency but may include reorged data. Default: `false`

Supported chains and entities

Chain	Entities	Zerolag
Polygon	`log`, `decoded_log`, `erc20_token_transfer`, `erc721_token_transfer`, `erc1155_token_transfer`	Yes
Base	`log`, `decoded_log`, `erc20_token_transfer`, `erc721_token_transfer`, `erc1155_token_transfer`	Yes
Solana	`nonvoting_transaction`	No
Hyperliquid	`block`, `trade`, `fill`, `order`, `misc_event`	No

More chains and entities are being added regularly. Check the Datastreams catalog for the latest availability, or use the GET /api/v1/beam/sources endpoint (requires a Beam-enabled organization and API key).

Transforms

Transforms process data in order. Two types are available. Each transform has an auto-generated uid — omit it when creating new transforms and the server will assign one.

Set filter
JavaScript (v8)

Filters data by matching a field value against a set. Only records whose extracted value exists in your set pass through. Sets support 10M+ values.

{
  "type": "redis_set_filter",
  "uid": "tf-001",
  "filter_expr": "root = this.token_address"
}

Field	Type	Required	Description
`type`	`"redis_set_filter"`	Yes	Must be `redis_set_filter`
`uid`	`string`	No	Auto-generated if omitted. Include to reference an existing transform.
`filter_expr`	`string`	Yes	Bloblang expression to extract the field value for filtering

Common filter_expr patterns:

Expression	Filters by
`root = this.address`	Contract address
`root = this.topic0`	Event signature
`root = this.from_address`	Transaction sender
`root = this.to_address`	Transaction recipient

Filter values are managed separately via the filter values endpoints. Changes take effect immediately — no redeploy needed.

Use lowercase values when filtering by addresses, labels, or symbols.

Transform data using JavaScript. Your function receives each record and can modify, enrich, or reshape it. Return null to drop a record.

{
  "type": "v8",
  "uid": "tf-002",
  "script": "function transform(record) { record.parsed = true; return record; }"
}

Field	Type	Required	Description
`type`	`"v8"`	Yes	Must be `v8`
`uid`	`string`	No	Auto-generated if omitted. Include to reference an existing transform.
`script`	`string`	Yes	JavaScript code that processes each record

Example — add a transfer size tag:

function transform(record) {
  const amount = parseFloat(record.amount || "0");
  if (amount > 1000000) {
    record.size_tag = "whale";
  } else if (amount > 10000) {
    record.size_tag = "medium";
  } else {
    record.size_tag = "small";
  }
  return record;
}

Sinks

Sinks define where processed data is delivered. A pipeline must have exactly one sink — the sinks array accepts a single entry. Each sink has an auto-generated uid — omit it when creating new sinks. Five types are available:

Kafka
SNS
External Kafka
Webhook
NATS JetStream

Delivers to an Allium-managed Kafka topic. After deployment, you receive connection credentials and consumer code snippets.

{
  "type": "kafka",
  "uid": "sk-001",
  "name": "usdc-transfers"
}

Field	Type	Required	Description
`type`	`"kafka"`	Yes	Must be `kafka`
`uid`	`string`	No	Auto-generated if omitted
`name`	`string`	Yes	Topic name suffix. Full topic: `beam.{config_id}.{name}`

Delivers to an Allium-managed SNS topic.

{
  "type": "sns",
  "uid": "sk-002",
  "name": "my-topic"
}

Field	Type	Required	Description
`type`	`"sns"`	Yes	Must be `sns`
`uid`	`string`	No	Auto-generated if omitted
`name`	`string`	Yes	Topic name suffix. Full topic: `beam-{config_id}-{name}`

Delivers to your own Kafka cluster.

{
  "type": "external_kafka",
  "uid": "sk-003",
  "name": "my-external-topic",
  "bootstrap_servers": "broker1.example.com:9092,broker2.example.com:9092",
  "username_secret_id": "my-kafka-username",
  "password_secret_id": "my-kafka-password",
  "sasl_mechanism": "PLAIN"
}

Field	Type	Required	Description
`type`	`"external_kafka"`	Yes	Must be `external_kafka`
`uid`	`string`	No	Auto-generated if omitted
`name`	`string`	Yes	Kafka topic name on your cluster
`bootstrap_servers`	`string`	Yes	Comma-separated broker addresses
`username_secret_id`	`string`	Yes	Organization secret id for the SASL username
`password_secret_id`	`string`	Yes	Organization secret id for the SASL password
`security_protocol`	`string`	No	Always `SASL_SSL` (not configurable)
`sasl_mechanism`	`string`	No	`PLAIN` or `SCRAM-SHA-512`. Default: `PLAIN`

If a referenced secret is updated after deployment, the sink is flagged as stale. You must redeploy for the new secret value to take effect.

Beam pipelines produce messages up to 1 MB in size. Make sure the target topic’s max.message.bytes is set to at least 1 MB, or deliveries may be rejected by your cluster.

Delivers data as POST requests to an HTTPS endpoint.

{
  "type": "webhook",
  "uid": "sk-004",
  "name": "my-webhook",
  "url": "https://example.com/webhook/receive",
  "hash_key": "MY_WEBHOOK_SIGNING_KEY",
  "enable_http_encoding": false
}

Field	Type	Required	Description
`type`	`"webhook"`	Yes	Must be `webhook`
`uid`	`string`	No	Auto-generated if omitted
`name`	`string`	Yes	Descriptive name for the webhook sink
`url`	`string`	Yes	HTTPS endpoint URL. Must use `https://`
`hash_key`	`string`	No	Organization secret id holding the HMAC signing key. When set, Beam signs each request — see Webhook authentication
`enable_http_encoding`	`boolean`	No	When `true`, Beam compresses each request body with zstd and sets `Content-Encoding: zstd` on the POST. Your endpoint must decompress the payload before parsing. Default: `false`

HTTP URLs are not supported — the URL must use HTTPS.

If the hash_key secret value is rotated after deployment, the sink is flagged as stale. You must redeploy for the new value to take effect.

Publishes each record to a NATS JetStream subject on your own broker.

{
  "type": "nats_jetstream",
  "uid": "sk-005",
  "name": "my-jetstream",
  "url": "nats://broker.example.com:4222",
  "subject": "allium.ethereum.blocks",
  "token_secret_id": "MY_NATS_TOKEN"
}

Field	Type	Required	Description
`type`	`"nats_jetstream"`	Yes	Must be `nats_jetstream`
`uid`	`string`	No	Auto-generated if omitted
`name`	`string`	Yes	Descriptive name for the sink
`url`	`string`	Yes	Broker URL, e.g. `nats://broker.example.com:4222`
`subject`	`string`	Yes	JetStream subject the pipeline publishes to, e.g. `allium.ethereum.blocks`
`token_secret_id`	`string`	No	Organization secret id holding the auth token. When omitted, the sink connects to the broker without authentication

Do not include the token in url. If token_secret_id is set, Beam injects the resolved token into the connection at deploy time. For example, with a token secret resolving to your-token, nats://your-broker.example.com:4222 becomes nats://your-token@your-broker.example.com:4222 before the publisher connects.

If the token_secret_id secret value is rotated after deployment, you must redeploy for the new value to take effect.

Contact support@allium.so if you need a specific sink type.

ID fields and creation behavior

When creating resources (pipelines, transforms, sinks), omit the id / uid field and the server will auto-generate one. When updating, include the id / uid to reference the existing resource.

Resource	ID field	Behavior when omitted
Pipeline	`id`	New pipeline created with auto-generated ID
Transform	`uid`	New transform created with auto-generated UID
Sink	`uid`	New sink created with auto-generated UID

Realtime API

Stream Transformation API

Beam API

Explorer API

Account API

Deep Dives

BeamConfig

CreateBeamConfigRequest

UpdateBeamConfigRequest

PipelineConfig

Source

Supported chains and entities

Transforms

Sinks

ID fields and creation behavior

​BeamConfig

​CreateBeamConfigRequest

​UpdateBeamConfigRequest

​PipelineConfig

​Source

​Supported chains and entities

​Transforms

​Sinks

​ID fields and creation behavior

BeamConfig

CreateBeamConfigRequest

UpdateBeamConfigRequest

PipelineConfig

Source

Supported chains and entities

Transforms

Sinks

ID fields and creation behavior