We are thrilled to unveil Yellowstone Fumarole : a new persistent streaming solution that empowers you to subscribe to geyser event without losing any data, even in the event of network connection errors.

This blog post will go in depth on why we created Fumarole and why you would use it.

We Love Streaming ❤️🐉

The Solana ecosystem is filled with streaming solutions that allow developers to listen to blockchain events in real time. At Triton One, one of our most successful services is Yellowstone Dragon’s Mouth, a streaming solution widely used by traders, DEXs, APIs, and DeFi builders.

At Triton One, we strive to offer the best streaming service with the lowest possible latency. Building a gRPC interface on top of Geyser was a key decision, ensuring both high performance and maintainability for our customers.

Under the hood, we've gone through multiple iterations to optimize performance. We reduced unnecessary data copies, fine-tuned HTTP/2 settings, and rigorously tested different configurations to deliver the best possible performance.

While Yellowstone Dragon’s Mouth serves many use cases, we’ve noticed that many customers struggle with reliability and data persistence.

• What happens if I get disconnected from Dragon’s Mouth?

• How do I know if I missed data?

• How can I retrieve lost data?

These are all important questions, and, to be frank, answering them isn’t as straightforward as one might think.

High-Availabitity ☁️✅

The new Yellowstone Fumarole service is designed for high availability, not just in terms of multiple endpoints to connect to, but also in how it fundamentally sources its data.

Under the hood, Fumarole aggregates data from multiple RPC nodes, coalescing everything into a single, consistent timeline of events.

This ensures that developers no longer need to worry about low-level redundancy challenges, such as subscribing to multiple RPC nodes or handling global deduplication of events.

Once you’re connected to Fumarole, you’re already receiving a highly available, reliable stream—without the extra complexity.

Persistence ⛰️

Persistence is by far the most important feature of Fumarole—blockchain data is stored in a distributed storage system, allowing our customers to query past events.

With consumer groups, Fumarole tracks your position in the data stream, ensuring that network issues won’t cause data loss. When you reconnect, you seamlessly resume from where you left off.

This is a feature many of our customers have been waiting for, and we believe it will be a game-changer for years to come.

Horizontally-Scalable ↔️

The introduction of consumer groups allows developers to partition the delivery of Geyser events, enabling better parallelism and scalability for their applications.

When you activate consumer group parallelism, you create a list of members where each member will receive a stream of unique events that no other members will see.

Moreover, each member is allowed to connect once. Our backend ensures to track active sessions, preventing developers from listening to the same stream twice.

Fumarole 🤝 Dragon's mouth

We put a lot of effort into making Fumarole fully compatible with the Dragon’s Mouth API.

Not only does Yellowstone Fumarole output the same data as Dragon’s Mouth, but it also supports the same account and transaction filter APIs.

This ensures that developer integration requires minimal changes to existing code.

Here’s a preview of the protobuffer specification for SubscribeRequest in Yellowstone Fumarole :

message SubscribeRequest {
  string consumer_group_label = 1;
  optional uint32 consumer_id = 2;
  map<string, geyser.SubscribeRequestFilterAccounts> accounts = 3;
  map<string, geyser.SubscribeRequestFilterTransactions> transactions = 4;
}

Here’s a the original Dragon’s mouth Protobuffer specification:

message SubscribeRequest {
  map<string, SubscribeRequestFilterAccounts> accounts = 1;
  map<string, SubscribeRequestFilterSlots> slots = 2;
  map<string, SubscribeRequestFilterTransactions> transactions = 3;
  map<string, SubscribeRequestFilterTransactions> transactions_status = 10;
  map<string, SubscribeRequestFilterBlocks> blocks = 4;
  map<string, SubscribeRequestFilterBlocksMeta> blocks_meta = 5;
  map<string, SubscribeRequestFilterEntry> entry = 8;
  optional CommitmentLevel commitment = 6;
  repeated SubscribeRequestAccountsDataSlice accounts_data_slice = 7;
  optional SubscribeRequestPing ping = 9;
  optional uint64 from_slot = 11;
}

As you can both shared the accounts and transactions filter API. In fact, Fumarole import the exact same type from geyser spec.

Getting Started 🏃‍♀️🏃🏽‍♂️🏃🏿‍♀️

Start by downloading our yellowstone-fume CLI tool.

This tool will allow you to control your account in fumarole service and do necessary bookkeeping: create or delete consumer groups, list metadata and even streaming!

Download using pip:

pip install yellowstone-fume

Test it:

fume --version

The fume CLI is the tool you will be using as the control-plane of Fumarole.

This tool allow you to:

1. Create consumer groups;

2. List/Get consumer groups;

3. Delete consumer groups;

4. Streaming consume groups.

For more details visit our public repository where we host public Fumarole code. In this repo you will find documentation and example on how to use fume CLI tool.

Configuration File 📜

Fumarole CLI looks for a file in ~/.config/fume/config.toml by default, you can change the path location by using fume --config <PATH>.

The configuration needed is simple, all you need is an endpoint and a token (contact your customer support channel for a token if you’re already a Triton customer, if not get in touch over Telegram):

[fumarole]
endpoints = ["fumarole.endpoint.rpcpool.com"]
x-token = "<YOUR X-TOKEN secret here>"

You can test your configuration file with test-config subcommand:

$ fume test-config

or with custom config path:

$ fume --config path/to/config.toml test-config

Create Consumer Group

To create a consumer group that at the end of the log, that stream only "confirmed" commitment level transactions:

$ fume create-cg --name helloworld-1 \
--commitment confirmed \
--seek latest \
--include tx

To do the same but for account updates:

$ fume create-cg --name helloworld-2 \
--commitment confirmed \
--seek latest \
--include account

More details can be find using the --help option:

$ fume create-cg --help
Creates a consumer group

Options:
  --name TEXT                     Consumer group name to subscribe to, if none
                                  provided a random name will be generated
                                  following the pattern
                                  'fume-<random-6-character>'.
  --size INTEGER                  Size of the consumer group
  --commitment [processed|confirmed|finalized]
                                  Commitment level  [default: confirmed]
  --include [all|account|tx]      Include option  [default: all]
  --seek [earliest|latest|slot]   Seek option  [default: latest]
  --help                          Show this message and exit.

Consumer Group Staleness ⌛

Consumer groups can become stale if you are ingesting too slow.

Fumarole is a distributed log of blockchain events where each new event is appended to the end.

As Solana emits a lot of events in one hour, we cannot keep every blockchain event forever.

Fumarole evicts fragment of the log as they age and get old enough.

Depending on the Fumarole cluster you are connected to this time may vary. Connect with us to learn more.

When creating a Consumer Group, you must ingest what you are capable of. Otherwise your consumer group is destined to become stale.

A stale consumer group is a consumer group that haven't yet ingested blockchain events that have already been evicted by Fumarole’s vacuum process.

Consumer Group Size and Performance Guidelines 📈

Consumer group sizes allow you to shard a fumarole stream into multiple consumer group member. Sharded consumer group follow similar semantics as Kafka Static Consumer membership.

Here's a quick-recap of static group membership:

• The Fumarole log is already sharded in multiple partitions.

• When you create a consumer group with --size N, it creates N member with each # total fumarole partition / N partitions.

• Each member of the consumer group advances at its own pace.

• Your consumer group becomes stale as soon as one membership is stale.

As of this writing the maximum size of a consumer group is 6.

Each member can have their own dedicated TCP connection which offer better performance.

The processing you do in reception, your internet speed, network bandwidth and location will impact the size of the consumer group.

Ingesting everything Fumarole can output requires you to be in the same region as your assigned Fumarole cluster and multiple Gbits for internet Bandwidth, otherwise you will fall behind and become stale.

Limit your subscription feed by using the various filters over the accounts and transactions we offer.

As for the consumer group size goes, starting with a size of 1 is the simplest approach.

If you are falling behind because your receiving code adds too much processing overhead, you can try 2, 3 and so forth.

Fumarole is already redundant and load balanced inside our datacenters, increasing --size does not inherently add more redundancy. It is a tool for you to scale your read operation in case on instance is not sufficient.

To create a consumer group with 2 members you just have to provided --size options:

$ fume create-cg --name example --size 2

List all Consumer Groups

$ fume list-cg

Delete a Consumer Group

$ fume delete-cg --name helloworld

Delete all Consumer Groups

$ fume delete-all-cg

Stream Summary on Terminal

To stream out from the CLI, you can use the stream command and its various features!

$ fume stream --name helloworld

You can filter the stream content by adding one or multiple occurrence of the following options:

• --tx-account <base58 pubkey> : filter transaction by account keys.

• --owner <base58 pubkey> : filter account update based on its owner

• --account <base58 pubkey> : filter account update based on accout key.

Here is an example to get all account updates owned by Token SPL program:

$ fume stream --cg-name helloworld \
--owner TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA

Here is how to chain multiple filters together:

$ fume stream --cg-name helloworld \
--owner metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s \
--owner TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb \
--owner TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA \
--owner ATokenGPvbdGVxr1b2hvZbsiqW5xWH25efTNsLJA8knL \
--owner BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY \
--owner CoREENxT6tW1HoK8ypY1SxRMZTcVPm7R94rH4PZNhX7d \
--tx-account BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY

The above command stream all data required by DAS.

Note: This command serves more as a testing tool/playground for you to try it out as it only prints summarized data.

Even More Code: Rust client! 🔧🔨

We have also published the yellowstone-fumarole-client crate.

Here’s a small code example on how to connect and subscribe to Fumarole using Rust code:

use yellowstone_fumarole_client::{
    config::FumaroleConfig,
    SubscriberRequestBuilder,
    FumaroleClientBuilder,
}
let config = FumaroleConfig {
    endpoint: "<ENDPOINT>".to_string(),
    x_token: Some("<X_TOKEN>".to_string()),
    max_decoding_message_size_bytes: FumaroleConfig::default_max_decoding_message_size_bytes(),
};

let fumarole = FumaroleClientBuilder::default()
    .connect(config)
    .await
    .expect("Failed to connect to Fumarole service");

let request = SubscribeRequestBuilder::default()
    .build("my-consumer-group");

let rx = fumarole
        .subscribe_with_request(request)
        .await
        .expect("Failed to subscribe to Fumarole service");

while let Some(msg) = rx.next().await {
    // ...
}

Visit yellowstone-fumarole public repo for more documentation and code example with rust.

Wrapping up 🖖

In conclusion, Yellowstone Fumarole and Dragon’s Mouth are at the forefront of providing scalable, high-availability streaming solutions for blockchain data. With features like persistent data storage, consumer group partitioning, and seamless developer integration, we’re empowering our customers to build more robust and efficient applications with ease.

At Triton One, we are committed to continually enhancing the performance, reliability, and scalability of our services to meet the growing demands of blockchain developers.

We believe that with Fumarole, you now have the tools to ensure high-performance, fault-tolerant data streaming.

Checkout other cool stuff Triton has!