Transforming IPC in CKB On-Chain Script

The CKB blockchain is taking a significant leap forward with the introduction of the Spawn syscall in its upcoming Meepo hardfork. Think of Spawn as a bridge that enables different CKB Scripts to talk to each other securely and efficiently—much like how programs communicate in modern operating systems.

However, building Spawn from scratch represents a significant development effort, which is why we’ve developed ckb-script-ipc libraries to simplify this process. Together, Spawn and the ckb-script-ipc library transform how on-chain scripts communicate and share functionality, offering developers a streamlined solution for creating modular and reusable on-chain applications.

In this deep dive, we’ll explore:

• How the new Spawn syscall overcomes the limitations of existing code reuse methods

• The ckb-script-ipc library and how it simplifies complex IPC (Inter-Process Communication) implementations

• Practical examples of building client-server communication between scripts

• The technical details of the wire format protocol that makes it all possible

• Future possibilities, including developments in bridging on-chain scripts with native machine code

Whether you’re a blockchain developer looking to leverage these new capabilities or a technical enthusiast interested in understanding CKB’s evolution, this guide will provide you with both the conceptual framework and practical knowledge needed to work with CKB’s new IPC features. Let’s dive in and explore how these new tools can transform the way we build on CKB.

Why Spawn

The upcoming CKB hardfork Meepo introduces a new syscall called Spawn. This feature draws inspiration from the Unix/Linux operating system, functioning similarly to a combined fork and exec operation.

Along with Spawn, related syscalls such as pipe, read, and write are also implemented, following Unix/Linux conventions. For detailed specifications, refer to the RFC documentation.

Overcoming Previous Challenges with Spawn

Prior to Spawn, CKB supported three primary methods for code reuse:

• Static linking

• Dynamic linking

• exec

Each of these methods has distinct limitations. Static linking, while straightforward, only enables code reuse at the source code level, making binary-level reuse impossible. Both dynamic linking and exec offer binary-level reuse but come with significant drawbacks.

Dynamic linking faces several challenges:

• Security Vulnerabilities: Called functions can access and modify the caller’s memory space, creating potential security risks

• Resource Constraints: The shared memory space architecture can lead to memory resource limitations

• Language Restrictions: Current implementation primarily supports C, with incomplete support for other languages like Rust

The exec syscall also has notable limitations:

• Context Loss: Execution resets the current context information, making state preservation impossible

• Communication Barriers: No built-in mechanism for inter-process communication

The new Spawn syscall addresses these limitations, offering a more robust and flexible solution for code reuse in CKB by:

• Providing isolated memory spaces to improve security

• Preserving context information to enable state preservation

• Enabling inter-process communication while maintaining security boundaries

IPC vs RPC

In computer science, Inter-Process Communication (IPC) refers to the mechanisms that allow processes to share data and communicate with each other within a computer system. With the introduction of Spawn syscalls in CKB, we can now implement IPC functionality in CKB scripts.

You might wonder why we refer to this as IPC rather than RPC (Remote Procedure Call). The key distinction lies in the execution context:

• IPC: The script processes, where the code is executed, are part of a single transaction, all running locally on the same machine.

• RPC: RPC systems are designed for distributed computing and include sophisticated features, such as encryption and authentication, comprehensive error handling and propagation, retry mechanisms and timeout management, horizontal scaling capabilities, and network transport protocols.

Our implementation focuses specifically on the core IPC features needed for efficient process-to-process communication within CKB’s transaction context. This targeted approach keeps the system lightweight and appropriate for its use case.

Challenges of Developing with Spawn

Implementing IPC using Spawn requires a series of steps. Here’s a comprehensive overview of what developers need to consider:

1. Interface Definition: Design and define the service interfaces and methods that will be exposed.

2. Channel Establishment: Create communication channels between processes using pipes.

3. Parameter Serialization: Encode method parameters into a standardized format.

4. Wire Format Conversion: Transform the serialized parameters into a binary blob suitable for transmission.

5. Data Transmission: Send the encoded data blob to the target process.

6. Data Reception and Parsing: Receive and decode the transmitted data blob.

7. Method Dispatch: Route the decoded request to the appropriate function handler.

8. Response Handling: Encode the return values into a transmissible format.

9. Response Transmission: Send the encoded response back to the calling process.

It’s important to note that implementing a robust IPC system requires additional consideration for error handling and exception management. Building such a system from scratch represents a significant development effort, which is why we’ve developed libraries to simplify this process.

A Simplified IPC Solution: ckb-script-ipc

To streamline the implementation of IPC from scratch, we’ve developed ckb-script-ipc, a library that significantly simplifies the process. Inspired by Google’s tarpc, this library provides a straightforward, easy-to-use interface for IPC implementation. It abstracts away the complexities of serialization, message passing, and error management internally. Developers can focus on defining their service interfaces and implementing business logic rather than dealing with low-level IPC details.

Let’s walk through the implementation process step by step.

Step 1 Add required dependencies

Add the required dependencies to your Cargo.toml

ckb-script-ipc = { version = "..." }
ckb-script-ipc-common = { version = "..." }
serde = { version = "...", default-features = false, features = ["derive"] }

Remember to replace “…” with the latest available versions of these crates.

Step 2 Define the IPC interface

Define the IPC interface using a trait decorated with our service attribute:

#[ckb_script_ipc::service]
pub trait World {    
    fn hello(name: String) -> Result<String, u64>;
}

This trait should be placed in a shared library accessible to both client and server scripts. The #[ckb_script_ipc::service] attribute macro automatically generates the necessary implementations for IPC communication.

Step 3 Initialize the server

Initialize the server by creating communication pipes:

use ckb_script_ipc_common::spawn::spawn_server;

let (read_pipe, write_pipe) = spawn_server(
    0,
    Source::CellDep,
    &[CString::new("demo").unwrap().as_ref()],
)?;

Step 4 Implement the service logic and start the server

use crate::def::World;
use ckb_script_ipc_common::spawn::run_server;

struct WorldServer;

impl World for WorldServer {
    fn hello(&mut self, name: String) -> Result<String, u64> {
        if name == "error" {
            Err(1)
        } else {
            Ok(format!("hello, {}", name))
        }
    }
}

run_server(WorldServer.server()).map_err(|_| Error::ServerError)

Note that run_server operates as an infinite loop to handle incoming requests. The server() method is automatically implemented by our proc-macro.

Step 5 Set up and interact with the client

use crate::def::WorldClient;
let mut client = WorldClient::new(read_pipe, write_pipe);
let ret = client.hello("world".into()).unwrap();

The client uses the pipe handles obtained during server initialization to communicate with the server. For a complete working example, you can explore our ckb-script-ipc-demo repository.

Key Components: Procedural Macros and Wire Format

Procedural Macros

The implementation of client-server communication in ckb-script-ipc heavily relies on Rust’s procedural macros to eliminate boilerplate code. The #[ckb_script_ipc::service] attribute macro is particularly powerful, automatically generating the necessary code for client, server, and communication handling.

Let’s examine how this macro transforms a simple service definition into production-ready code:

First, define your service interface:

#[ckb_script_ipc::service]
pub trait World {    
    fn hello(name: String) -> Result<String, u64>;
}

The macro then generates the required implementation code, including client-side methods, request and response types, and communication handling.

Here’s a simplified version of the generated client code:

impl<R, W> WorldClient<R, W>
where
    R: ckb_script_ipc_common::io::Read,
    W: ckb_script_ipc_common::io::Write,
{
    pub fn hello(&mut self, name: String) -> Result<String, u64> {
        let request = WorldRequest::Hello { name };
        let resp: Result<_, ckb_script_ipc_common::error::IpcError> = self
            .channel
            .call::<_, WorldResponse>("World.hello", request);
        match resp {
            Ok(WorldResponse::Hello(ret)) => ret,
            Err(e) => {
                // Error handling code
            }
        }
    }
}

Here is a simplified version of generated server code:

impl<S> ckb_script_ipc_common::ipc::Serve for ServeWorld<S>
where
    S: World,
{
    type Req = WorldRequest;
    type Resp = WorldResponse;
    fn serve(
        &mut self,
        req: WorldRequest,
    ) -> ::core::result::Result<
        WorldResponse,
        ckb_script_ipc_common::error::IpcError,
    > {
        match req {
            WorldRequest::Hello { name } => {
                let ret = self.service.hello(name);
                Ok(WorldResponse::Hello(ret))
            }
        }
    }
}

The generated code handles several aspects:

• Type-safe request and response structures

• Proper error handling and propagation

• Serialization and deserialization of parameters

• Method routing and dispatch

This automatic code generation significantly reduces development time and potential errors while ensuring consistent implementation patterns across different services.

Wire Format

Another key component of ckb-script-ipc is its wire format, which defines how data is transmitted between processes. While the spawn syscall provides basic read/write stream operations, we needed a more structured approach to handle complex inter-process communications. This led us to implement a packet-based protocol.

We use Variable-length quantity (VLQ) to define the length information in the packet header. Compared to fixed-length representations, VLQ is more compact and suitable for this scenario. Packets are divided into the following two categories: Request and Response.

The Request contains the following fields without any format. That is, all fields are directly arranged without any additional header. Therefore, in the shortest case, version + method id + length only occupies 3 bytes. The complete structure includes:

• version (VLQ)

• method id (VLQ)

• length (VLQ)

• payload (variable length data)

The Response contains the following fields:

• version (VLQ)

• error code (VLQ)

• length (VLQ)

• payload (variable length data)

Let’s examine each field in detail:

All numeric fields (version, length, method_id, error_code) use VLQ encoding for efficient space utilization while supporting values up to 2^64. This provides a good balance between compact representation for common small values while maintaining support for larger values when needed.

For serialization and deserialization, we utilize serde_json as our primary library. This means any Rust structure that implements the Serialize and Deserialize traits (which can be automatically derived using the #[derive(Serialize, Deserialize)] attribute macro) can be seamlessly used as parameters and return values in your IPC communications. This provides great flexibility in the types of data you can transmit between processes while maintaining type safety. JSON is not the only option—any Serde framework that supports the Serialize and Deserialize traits can be used.

Potentiality Beyond On-Chain Communication

While the primary focus of ckb-script-ipc has been facilitating communication between on-chain scripts, its potential extends beyond that. One exciting development direction is bridging the gap between on-chain scripts and native off-chain machine code, enabling off-chain services to interact with on-chain functionality.

Let’s explore how this works. To interact with on-chain services from native code, follow these steps:

Step 1. Enable the std feature in ckb-script-ipc-common

Step 2. Initialize the server:

let script_binary = std::fs::read("path/to/on-chain-script-binary").unwrap();let (read_pipe, write_pipe) = ckb_script_ipc_common::native::spawn_server(&script_binary, &[]).unwrap();

Step 3. Create and interact with the client:

let mut client = UnitTestsClient::new(read_pipe, write_pipe);client.test_primitive_types(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, true);

These operations are executed on the native machine (off-chain), providing a bridge between off-chain applications and on-chain scripts.

The current implementation has two main limitations:

1. Transaction Context: The CKB-VM machine running in this mode cannot access transaction context data, as this information isn’t currently provided to the VM.

2. Integration Complexity: Integration with off-chain projects requires manual setup since the functionality is provided as a library rather than a complete solution.

We have a plans to enhance this functionality with two key features:

1. Native Node Integration: We’ll integrate the functionality directly into CKB nodes as an HTTP service, providing a “batteries included” solution that’s ready to use out of the box.

2. Context-Aware Execution: Future updates will enable access to transaction context data, allowing for more sophisticated interactions between off-chain and on-chain components.

These improvements will significantly expand the utility of ckb-script-ipc, making it a more powerful tool for bridging across on-chain and off-chain systems.

Final remarks

The introduction of Spawn and ckb-script-ipc marks a significant advancement in CKB’s script development capabilities. By providing robust IPC functionality and simplifying complex implementation details, these tools enable developers to build more sophisticated and modular on-chain applications. We encourage developers to explore these new capabilities and contribute to the growing ecosystem of CKB applications.

---

✍🏻 Written by Jiandong Xu

His previous posts include:

• Enable Bitcoin Taproot on CKB (Part I)

• Enable Bitcoin Taproot on CKB (Part II)

• Omnilock, a Universal Lock that Powers Interoperability