Getting Started With Google Pub/Sub
Integrating Allium’s Pub/Sub data streams into your systems is a straightforward process that allows you to access real-time data seamlessly. Follow the steps below to create and manage Pub/Sub streams for your organization.
You will need to set up a subscription to start receiving data. Allium supports two types of subscriptions:
- Pull Subscription: This allows your system to pull data from the stream. When setting up a pull subscription, provide a service account or user that will be authorized to pull data.
- Push Subscription: This option sends data directly to your specified webhook endpoint. When setting up a push subscription, ensure you provide the correct webhook URL where the data will be sent.
Step 1: Request access to data streams
To get started, email support@allium.so specifying the chains and schemas you’re interested in accessing.
For Pull Subscriptions: Include the email address you want to use for authentication. For Push Subscriptions: Provide the webhook URL where you want to receive the data.
Our team will confirm once your data streams are set up and ready to use.
Step 2: Receive the data
In order to receive data, you will need to configure your code to connect and pull data from the Pub/Sub subscription. Here’s an example Python snippet to pull the data:
The data should look something like:
You can read more about pull subscriptions and what each field means in the Google Pub/Sub docs.
In order to receive data, you will need to configure your code to connect and pull data from the Pub/Sub subscription. Here’s an example Python snippet to pull the data:
The data should look something like:
You can read more about pull subscriptions and what each field means in the Google Pub/Sub docs.
You should already be seeing data being sent to the endpoint you configured. The data you receive should look something like:
You can read more about push subscriptions and what each field means in the Google Pub/Sub docs.
Step 3: Decompress the data
Data in Allium’s streams are typically compressed for optimization reasons. We employ 3 different compression methods, depending on the shape, size and frequency of data coming through the stream:
The easiest way to determine which compression method was used, is to check the first few bytes of the data. Here are the bytes that the data will start with depending on the compression method used:
Method | First Bytes |
---|---|
gzip | \x1f\x8b |
zstandard | \x28\xb5\x2f\xfd |
lz4 | \x04\x22\x4d\x18 |
Here are sample scripts that will help you decompress the data if necessary: