Understanding Disk Slowness & Performance Bottlenecks

Scenario

Troubleshooting issues related to disk slowness or disk bottlenecks.

Overview of Disk Performance

We assess whether an application is performing well or not using performance indicators like:

· How fast an application is processing a user request

• How much data an application is processing per request

• How many requests is an application processing in a specific period of time

• How long a user has to wait to get a response after submitting their request

Most of the disk performance issues can be caused due high or low values of one or more than one of the following parameters. These parameters can also be called as Application Performance Indicators:

· IOPS

· Throughput (Bandwidth)

· Latency

The value of the above parameters at any given point in time depends on the IO Size (e.g., 4K or 64K or 1MB).

Understanding IO Size with Apple Analogy: How many apples we can eat in 10 minutes depends on the size of the apple. Quantity of smaller apple which is marked as 8K can be consumed more than the apple marked as 1MB. In this example, the size of the apple is IO Size and number of apples consumed in 10 minutes is IOPS*.*

Key Performance Indicators Explained

IOPS

· IOPS is an industry measurement of the hardware throughput of a disk or a collection of disks

• IOPS is number of requests that your application is sending to the storage disks in one second

• An input/output operation could be read or write, sequential or random

• OLTP applications like an online retail website need to process many concurrent user requests immediately

  Typical IOPS of 15K standard HDD: 200

  Typical IOPS of SSD: 5K to 95K

Throughput (Bandwidth)

· Throughput or Bandwidth is the amount of data that your application is sending to the storage disks in a specified interval

• When an application is performing input/output operations with large IO unit sizes, it requires high Throughput

• Example: Data warehouse Application

• Measured in MB/s or Gbps

Therefore, it is important to determine the optimal Throughput and IOPS values that your application requires. As you try to optimize one, the other also gets affected.

Latency

· Latency is the time it takes an application to receive a single request, send it to the storage disks and send the response to the client

• This is a critical measure of an application's performance in addition to IOPS and Throughput

• The Latency of a storage is the time it takes to retrieve the information for a request and communicate it back to your application

• Latency is measured in milliseconds

Important: When you are optimizing your application to get higher IOPS and Throughput, it will affect the Latency of your application. After tuning the application performance, always evaluate the Latency of the application to avoid unexpected high latency behaviour.

The Critical Role of I/O Size

· The IO size is the size of the input/output operation request generated by the application

• Identifying the nature of IO requests, random/sequential, read/write, small/large, helps to determine the performance requirements of application

• It has a significant impact on performance especially on the IOPS and Bandwidth that the application can achieve

• An OLTP application generates millions of small and random IO requests

• A data warehousing application generates large and sequential IO requests

IO Size × IOPS = Throughput

If the IO Size is higher (e.g., 1 MB) we will see high Throughput

If the IO Size is less (e.g., 4 KB) the IOPS will be higher

Nature of IO Requests

An IO request is a unit of input/output operation that your application will be performing. Identifying the nature of IO requests, random or sequential, read or write, small or large, will help you determine the performance requirements of your application.

It is very important to understand the nature of IO requests, to make the right decisions when designing your application infrastructure.

Performance Scenarios: A Practical Look

The following table illustrates how different I/O sizes affect performance based on application requirements.

Application Requirement	I/O Size	IOPS	Throughput/Bandwidth
Max IOPS	8 KB	5,000	40 MB per second
Max Throughput	1024 KB	200	200 MB per second
Max Throughput + high IOPS	64 KB	3,200	200 MB per second
Max IOPS + high Throughput	32 KB	5,000	160 MB per second

How to look at Performance Monitor Data for Disks - The Basics

Key Points:

• The PerfMon counters are available for processor, memory and, each logical disk and physical disk of your server

• Disk Performance Counters to measure application performance requirements

• The important counter mentioned under the PerfMon Column need to be looked at to start with

Performance Counters Reference

Counter	Description	PerfMon
IOPS or Transactions per second	Number of I/O requests issued to the storage disk per second.	Disk Reads/sec Disk Writes/sec
Disk Reads and Writes	% of Reads and Write operations performed on the disk.	% Disk Read Time % Disk Write Time
Throughput	Amount of data read from or written to the disk per second.	Disk Read Bytes/sec Disk Write Bytes/sec
Latency	Total time to complete a disk IO request.	Average Disk sec/Read and Average disk sec/Write
IO size	The size of I/O requests issues to the storage disks.	Average Disk Bytes/Read and Average Disk Bytes/Write
Queue Depth	Number of outstanding I/O requests waiting to be read from or written to the storage disk.	Current Disk Queue Length
Max. Memory	Amount of memory required to run application smoothly	% Committed Bytes in Use
Max. CPU	Amount CPU required to run application smoothly	% Processor time

Disk Performance Counter Summary

Counter	Description	PerfMon
IOPS or Transactions/second	Number of I/O requests issued to the storage disk per second.	Disk Reads/sec
Disk Writes/sec
Disk Reads and Writes	% of Reads and Write operations performed on the disk.	% Disk Read Time
% Disk Write Time
Throughput	Amount of data read from or written to the disk per second.	Disk Read Bytes/sec
Disk Write Bytes/sec
Latency	Total time to complete a disk IO request.	Average Disk sec/Read
Average disk sec/Write
IO size	The size of I/O requests issues to the storage disks.	Average Disk Bytes/Read
Average Disk Bytes/Write
Queue Depth	Number of outstanding I/O requests waiting to be read from or written to the storage disk.	Current Disk Queue Length