> ## Documentation Index
> Fetch the complete documentation index at: https://docs.allium.so/llms.txt
> Use this file to discover all available pages before exploring further.
# Configuration reference
> Schema reference for Beam pipeline configs, sources, transforms, and sinks.
A Beam pipeline config describes the full data flow: where data comes from (source), how it's processed (transforms), and where it goes (sinks).
```text theme={null}
source → transforms[] → sinks[]
```
## BeamConfig
The top-level object returned by all config endpoints.
```json theme={null}
{
"id": "abc123",
"name": "USDC Transfer Monitor",
"description": "Monitors USDC ERC20 transfers on Base",
"tags": ["production", "base"],
"static_egress_ip": false,
"pod_size": "S",
"owner_org_team_user_id": "org_xyz",
"created_at": "2025-01-15T10:30:00Z",
"updated_at": "2025-01-15T10:30:00Z",
"pipeline_config": {
"source": { ... },
"transforms": [ ... ],
"sinks": [ ... ]
}
}
```
| Field | Type | Required | Description |
| :----------------------- | :---------------------------------- | :------------- | :-------------------------------------------------------------- |
| `id` | `string` | Auto-generated | Pipeline ID. Omit when creating — the server generates it. |
| `name` | `string` | Yes | Pipeline name |
| `description` | `string` | Yes | Pipeline description |
| `tags` | `string[]` | No | Tags for organizing pipelines. Default: `null` |
| `static_egress_ip` | `boolean` | No | Enable a static outbound IP for webhook sinks. Default: `false` |
| `pod_size` | `string` | No | Worker pod size: `S`, `M`, `L`, or `XL`. Default: `S` |
| `owner_org_team_user_id` | `string` | Auto-set | Owner ID, set from the authenticated API key |
| `created_at` | `datetime` | Auto-set | Creation timestamp |
| `updated_at` | `datetime` | Auto-set | Last update timestamp |
| `pipeline_config` | [`PipelineConfig`](#pipelineconfig) | Yes | The pipeline's source, transforms, and sinks |
## CreateBeamConfigRequest
Used as the request body for [Create pipeline](/beam/api-reference/configs/create-pipeline). Same as `BeamConfig` but without auto-generated fields.
```json theme={null}
{
"name": "USDC Transfer Monitor",
"description": "Monitors USDC ERC20 transfers on Base",
"tags": ["production", "base"],
"static_egress_ip": false,
"pod_size": "S",
"pipeline_config": {
"source": { ... },
"transforms": [ ... ],
"sinks": [ ... ]
}
}
```
| Field | Type | Required | Description |
| :----------------- | :---------------------------------- | :------- | :---------------------------------------------------- |
| `name` | `string` | Yes | Pipeline name |
| `description` | `string` | Yes | Pipeline description |
| `tags` | `string[]` | No | Tags for organizing pipelines |
| `static_egress_ip` | `boolean` | No | Enable a static outbound IP. Default: `false` |
| `pod_size` | `string` | No | Worker pod size: `S`, `M`, `L`, or `XL`. Default: `S` |
| `pipeline_config` | [`PipelineConfig`](#pipelineconfig) | Yes | The pipeline's source, transforms, and sinks |
## UpdateBeamConfigRequest
Used as the request body for [Update pipeline](/beam/api-reference/configs/update-pipeline). The shape is **identical** to [`CreateBeamConfigRequest`](#createbeamconfigrequest) — only the editable fields. The `config_id` is taken from the URL path, not the body. Server-managed fields (`id`, `owner_org_team_user_id`, `organization_id`, `created_at`, `updated_at`) are preserved from the existing record and ignored if present in the body.
```json theme={null}
{
"name": "USDC Transfer Monitor (updated)",
"description": "Updated description",
"tags": ["production", "base"],
"static_egress_ip": false,
"pod_size": "S",
"pipeline_config": {
"source": { ... },
"transforms": [ ... ],
"sinks": [ ... ]
}
}
```
## PipelineConfig
```json theme={null}
{
"source": { ... },
"transforms": [ ... ],
"sinks": [ ... ]
}
```
| Field | Type | Required | Description |
| :----------- | :--------------------------- | :------- | :-------------------------------- |
| `source` | [`Source`](#source) | Yes | Where data comes from |
| `transforms` | [`Transform[]`](#transforms) | Yes | Processing steps applied in order |
| `sinks` | [`Sink[]`](#sinks) | Yes | Where processed data is delivered |
***
## Source
Connects to Allium's Datastreams. Select a blockchain and entity type.
```json theme={null}
{
"type": "pubsub",
"chain": "base",
"entity": "erc20_token_transfer",
"is_zerolag": false
}
```
| Field | Type | Required | Description |
| :----------- | :-------- | :------- | :------------------------------------------------------------------------------------------------------------------ |
| `type` | `string` | Auto-set | Source type (set automatically based on chain) |
| `chain` | `string` | Yes | Blockchain to source data from |
| `entity` | `string` | Yes | Entity type to stream |
| `is_zerolag` | `boolean` | No | Stream from the tip of the blockchain before finality. Lower latency but may include reorged data. Default: `false` |
### Supported chains and entities
| Chain | Entities | Zerolag |
| :-------------- | :---------------------------------------------------------------------------------------------- | :------ |
| **Polygon** | `log`, `decoded_log`, `erc20_token_transfer`, `erc721_token_transfer`, `erc1155_token_transfer` | Yes |
| **Base** | `log`, `decoded_log`, `erc20_token_transfer`, `erc721_token_transfer`, `erc1155_token_transfer` | Yes |
| **Solana** | `nonvoting_transaction` | No |
| **Hyperliquid** | `block`, `trade`, `fill`, `order`, `misc_event` | No |
More chains and entities are being added regularly. Check the [Datastreams catalog](https://app.allium.so/build/datastreams) for the latest availability, or use the [`GET /api/v1/beam/sources`](/beam/api-reference/configs/list-sources) endpoint (requires a Beam-enabled organization and API key).
***
## Transforms
Transforms process data in order. Two types are available. Each transform has an auto-generated `uid` — omit it when creating new transforms and the server will assign one.
Filters data by matching a field value against a set. Only records whose extracted value exists in your set pass through. Sets support **10M+ values**.
```json theme={null}
{
"type": "redis_set_filter",
"uid": "tf-001",
"filter_expr": "root = this.token_address"
}
```
| Field | Type | Required | Description |
| :------------ | :------------------- | :------- | :----------------------------------------------------------------------------------------------------------------- |
| `type` | `"redis_set_filter"` | Yes | Must be `redis_set_filter` |
| `uid` | `string` | No | Auto-generated if omitted. Include to reference an existing transform. |
| `filter_expr` | `string` | Yes | [Bloblang](https://www.benthos.dev/docs/guides/bloblang/about) expression to extract the field value for filtering |
**Common `filter_expr` patterns:**
| Expression | Filters by |
| :------------------------- | :-------------------- |
| `root = this.address` | Contract address |
| `root = this.topic0` | Event signature |
| `root = this.from_address` | Transaction sender |
| `root = this.to_address` | Transaction recipient |
Filter values are managed separately via the [filter values endpoints](/beam/api-reference/filter-values/list-filter-values). Changes take effect immediately — no redeploy needed.
Use **lowercase** values when filtering by addresses, labels, or symbols.
Transform data using JavaScript. Your function receives each record and can modify, enrich, or reshape it. Return `null` to drop a record.
```json theme={null}
{
"type": "v8",
"uid": "tf-002",
"script": "function transform(record) { record.parsed = true; return record; }"
}
```
| Field | Type | Required | Description |
| :------- | :------- | :------- | :--------------------------------------------------------------------- |
| `type` | `"v8"` | Yes | Must be `v8` |
| `uid` | `string` | No | Auto-generated if omitted. Include to reference an existing transform. |
| `script` | `string` | Yes | JavaScript code that processes each record |
**Example** — add a transfer size tag:
```javascript theme={null}
function transform(record) {
const amount = parseFloat(record.amount || "0");
if (amount > 1000000) {
record.size_tag = "whale";
} else if (amount > 10000) {
record.size_tag = "medium";
} else {
record.size_tag = "small";
}
return record;
}
```
***
## Sinks
Sinks define where processed data is delivered. Each sink has an auto-generated `uid` — omit it when creating new sinks. Five types are available:
Delivers to an Allium-managed Kafka topic. After deployment, you receive connection credentials and consumer code snippets.
```json theme={null}
{
"type": "kafka",
"uid": "sk-001",
"name": "usdc-transfers"
}
```
| Field | Type | Required | Description |
| :----- | :-------- | :------- | :------------------------------------------------------- |
| `type` | `"kafka"` | Yes | Must be `kafka` |
| `uid` | `string` | No | Auto-generated if omitted |
| `name` | `string` | Yes | Topic name suffix. Full topic: `beam.{config_id}.{name}` |
Delivers to an Allium-managed SNS topic.
```json theme={null}
{
"type": "sns",
"uid": "sk-002",
"name": "my-topic"
}
```
| Field | Type | Required | Description |
| :----- | :------- | :------- | :------------------------------------------------------- |
| `type` | `"sns"` | Yes | Must be `sns` |
| `uid` | `string` | No | Auto-generated if omitted |
| `name` | `string` | Yes | Topic name suffix. Full topic: `beam-{config_id}-{name}` |
Delivers to your own Kafka cluster.
```json theme={null}
{
"type": "external_kafka",
"uid": "sk-003",
"name": "my-external-topic",
"bootstrap_servers": "broker1.example.com:9092,broker2.example.com:9092",
"username_secret_id": "my-kafka-username",
"password_secret_id": "my-kafka-password",
"sasl_mechanism": "PLAIN"
}
```
| Field | Type | Required | Description |
| :------------------- | :----------------- | :------- | :------------------------------------------------------------------------------------- |
| `type` | `"external_kafka"` | Yes | Must be `external_kafka` |
| `uid` | `string` | No | Auto-generated if omitted |
| `name` | `string` | Yes | Kafka topic name on your cluster |
| `bootstrap_servers` | `string` | Yes | Comma-separated broker addresses |
| `username_secret_id` | `string` | Yes | [Organization secret](/app/organization-management/secrets) *id* for the SASL username |
| `password_secret_id` | `string` | Yes | [Organization secret](/app/organization-management/secrets) *id* for the SASL password |
| `security_protocol` | `string` | No | Always `SASL_SSL` (not configurable) |
| `sasl_mechanism` | `string` | No | `PLAIN` or `SCRAM-SHA-512`. Default: `PLAIN` |
If a referenced secret is updated after deployment, the sink is flagged as **stale**. You must **redeploy** for the new secret value to take effect.
Beam pipelines produce messages up to **1 MB** in size. Make sure the target topic's `max.message.bytes` is set to at least 1 MB, or deliveries may be rejected by your cluster.
Delivers data as POST requests to an HTTPS endpoint.
```json theme={null}
{
"type": "webhook",
"uid": "sk-004",
"name": "my-webhook",
"url": "https://example.com/webhook/receive",
"hash_key": "MY_WEBHOOK_SIGNING_KEY",
"enable_http_encoding": false
}
```
| Field | Type | Required | Description |
| :--------------------- | :---------- | :------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | `"webhook"` | Yes | Must be `webhook` |
| `uid` | `string` | No | Auto-generated if omitted |
| `name` | `string` | Yes | Descriptive name for the webhook sink |
| `url` | `string` | Yes | HTTPS endpoint URL. Must use `https://` |
| `hash_key` | `string` | No | [Organization secret](/app/organization-management/secrets) *id* holding the HMAC signing key. When set, Beam signs each request — see [Webhook authentication](/beam/webhook-authentication) |
| `enable_http_encoding` | `boolean` | No | When `true`, Beam compresses each request body with zstd and sets `Content-Encoding: zstd` on the POST. Your endpoint must decompress the payload before parsing. Default: `false` |
HTTP URLs are not supported — the URL must use HTTPS.
If the `hash_key` secret value is rotated after deployment, the sink is flagged as **stale**. You must **redeploy** for the new value to take effect.
Publishes each record to a NATS JetStream subject on your own broker.
```json theme={null}
{
"type": "nats_jetstream",
"uid": "sk-005",
"name": "my-jetstream",
"url": "nats://broker.example.com:4222",
"subject": "allium.ethereum.blocks",
"token_secret_id": "MY_NATS_TOKEN"
}
```
| Field | Type | Required | Description |
| :---------------- | :----------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type` | `"nats_jetstream"` | Yes | Must be `nats_jetstream` |
| `uid` | `string` | No | Auto-generated if omitted |
| `name` | `string` | Yes | Descriptive name for the sink |
| `url` | `string` | Yes | Broker URL, e.g. `nats://broker.example.com:4222` |
| `subject` | `string` | Yes | JetStream subject the pipeline publishes to, e.g. `allium.ethereum.blocks` |
| `token_secret_id` | `string` | No | [Organization secret](/app/organization-management/secrets) *id* holding the auth token. When omitted, the sink connects to the broker without authentication |
Do not include the token in `url`. If `token_secret_id` is set, Beam injects the resolved token into the connection at deploy time. For example, with a token secret resolving to `your-token`, `nats://your-broker.example.com:4222` becomes `nats://your-token@your-broker.example.com:4222` before the publisher connects.
If the `token_secret_id` secret value is rotated after deployment, you must **redeploy** for the new value to take effect.
Contact [support@allium.so](mailto:support@allium.so) if you need a specific sink type.
***
## ID fields and creation behavior
When creating resources (pipelines, transforms, sinks), **omit the `id` / `uid` field** and the server will auto-generate one. When updating, include the `id` / `uid` to reference the existing resource.
| Resource | ID field | Behavior when omitted |
| :-------- | :------- | :-------------------------------------------- |
| Pipeline | `id` | New pipeline created with auto-generated ID |
| Transform | `uid` | New transform created with auto-generated UID |
| Sink | `uid` | New sink created with auto-generated UID |