Benchmarking IBM Storage Ceph Object Gateway: A Deep Dive into Secure, Scalable Object Storage Performance. Part 1

Introduction: Why Performance Matters for Secure Object Storage

Organizations today are challenged not only with managing massive volumes of data — often in the tens of petabytes — but also with the responsibility of securing that data across hybrid and multicloud environments. Object storage systems, such as IBM Storage Ceph, provide the scalability and flexibility required to meet these challenges, offering S3-compatible access, native redundancy, and a growing set of enterprise features.

But security and scalability come with trade-offs. As encryption, load balancing, and internal daemon authentication are layered onto Ceph Object Gateway (RGW) deployments, it becomes essential to understand their impact on latency, throughput, and resource utilization.

This blog series presents the results of a comprehensive performance benchmarking effort on IBM Storage Ceph Object Gateway (RGW) running on all-flash IBM Ceph Ready nodes, conducted by the IBM Storage Ceph Performance and Interoperability team. Special thanks to @Jay Rajput for leading the execution of all test cases and data collection. Our evaluation focuses on how real-world workloads interact with different configurations of encryption, data protection, and horizontal scalability, offering practical insights for architects, administrators, and developers alike.

Hardware and Software Setup

We tested on a production-grade IBM Storage Ceph cluster, deployed with collocated daemons for RGW, OSDs, MONs, MGRs, and Ingress.

Hardware Specifications

Each test cluster configuration (4-node, 8-node, and 12-node) maintained consistent OSD density (24 per node) and 4 RGW daemons per node, with dedicated VIPs for Ingress-based load balancing.

Software version matrix

Ceph Cluster Configuration

Network Architecture & Hardware Connectivity

To complement the compute/storage setup, our network underpins the cluster's high-throughput performance:

• Leaf–Spine topology: We’re running a 100 Gbps leaf–spine network with one spine (QFX5120) and three leafs (QFX5120), enabling a scalable, low-latency design. This offers port density now and a future upgrade path (e.g., adding a true spine and repurposing the current one) without impacting performance.

• Dual 100 Gbps uplinks per server via LACP: Each Ceph node utilizes two 100 Gbps ports on a single NIC, bonded using LACP, to connect to both leaf switches for redundancy and link aggregation.

• Per-node limit: Each Ceph storage node is equipped with Intel NICs that support a maximum throughput of 100 Gbps, even though two ports are available and bonded via LACP. This means that per-node throughput is capped at ~12.5 GB/s in optimal conditions.

• Cluster-wide switching capacity: Our leaf–spine topology, built with one QFX5120 spine and three QFX5120 leaf switches, provides full line-rate connectivity across all 12 storage nodes. Each leaf connects to four nodes and uplinks to the spine at 100 Gbps. This results in a total cluster theoretical switching capacity of ~150 GB/s. In our large-object benchmarks, the system achieved an aggregate throughput of ~111 GB/s, demonstrating that we were reaching the physical network ceiling, particularly for large object read-intensive workloads.

Test Methodology

We designed our performance evaluation to answer foundational questions about how to deploy Ceph Object Gateway (RGW) for both performance and security:

• What’s the impact of TLS (SSL) on RGW throughput and latency?

• How much overhead does server-side encryption (SSE-S3/KMS) introduce?

• Does securing internal daemon communication (msgr v2) affect Performance and CPU utilization?

• How do EC profiles (2+2, 4+2, 8+3) compare to 3x replication?

• What are the performance implications of using HAProxy-based ingress vs direct access?

• How does performance scale with node count and concurrency?

Each test case was repeated across PUT and GET workloads with varying object sizes, ranging from 64 KB to 1 GB. Elbencho was used in client-server mode with thread counts of 128, running up to 8 concurrent clients(1024 Threads). Object buckets were pre-provisioned to avoid index initialization penalties, and multipart upload was used for objects of 1 GB.

Executive Summary

IBM Storage Ceph demonstrates exceptional performance and flexibility when deployed on cutting-edge, all-flash infrastructure with 100 GbE networking, such as the IBM Storage Ceph Ready Nodes. As enterprises scale to billions of objects and multi-petabyte workloads, Ceph's ability to handle diverse data patterns, from high IOPS, low-latency, metadata-heavy workloads to high-throughput, bandwidth-intensive workloads, becomes critical.

Large Object Workloads (Throughput Focus)

For objects exceeding 32MB, the cluster achieved near-linear scaling up to 12 storage nodes, peaking at an aggregate PUT throughput of ~65 GB/s and an aggregate GET throughput of ~115 GB/s. Beyond this point, 100 GbE NIC saturation on individual nodes became the primary constraint. This suggests that future benchmark testing will benefit from higher-bandwidth NICs, as large object workloads still have room to achieve more throughput from the current node's available resources.

For large objects, fully secured configurations (TLS + SSE + msgr v2) maintained high throughput with reasonable overhead, demonstrating that Ceph Object Gateway (RGW) is well-suited for secure data pipelines at scale.

Small Object Workloads (IOPS & Latency Focus)

Small object tests (64 KB–1 MB) demonstrated Ceph Object Gateway (RGW)’s ability to efficiently scale IOPS with increasing concurrency and cluster size, particularly in CPU-optimized environments. With 64 KB objects, the system achieved up to 391K GET IOPS and 86K PUT IOPS on a 12-node cluster using Erasure Coding.

These workloads benefit significantly from high RGW thread counts and strong CPU provisioning. At peak load, CPU became the primary resource constraint, particularly when SSE-S3 encryption at rest was enabled, underscoring the importance of sizing the hardware CPU resources when targeting high-throughput, small-object use cases.

What’s Next

This post introduces the testbed, methodology, and highlights key results. In the upcoming posts of this series, we’ll delve into each performance axis, exploring the impact of TLS and SSE on RGW throughput, scaling behaviors with erasure coding versus replication, how concurrency and daemon density affect latency, and more. Stay tuned. There’s much more to uncover.