The character of your storage workload can be broken down into several traits - random vs. sequential I/O, large vs. small I/O request size, read vs. write ratio, and degree of parallelism.  The traits of your particular workload dictate how it interacts with the components of your storage system and ultimately determine the performance of your environment under a given configuration. This article will introduce the workload characterization, and how it affects your existing storage system.

    Note: This article is based on the traditional spinning hard disks. The solid-state disks (SSD) do not need to spin or seek to locate data, their workload characterization is quite different from the traditional disks.

Random vs. Sequential Access

    First let’s take a look at one formula:

    IOPS = 1000/(Seek Latency + Rotational Latency)

    We divide into 1000 to remove milliseconds from the equation, leaving (Seek Latency + Rotational Latency) as the important part of the equation.  Rotational latency is based on the spindle speed of the disk – 7.2k, 10k, or 15k RPM for standard server or SAN disks. The only way to change rotational latency is to buy faster (or slower) disks.  This essentially leaves seek latency as the only variable that we can “adjust”.  You’ll also recall that seek latency was the larger of the latencies and counts more against IOPS capability than does rotational latency.  Seeking is the most expensive operation in terms of performance.

    It is next to impossible to adjust seek latency on a disk because it is determined by the speed of the servos that move the heads across the platter.  We can, however, send workloads with different degrees of randomness to the platter.  The more sequential a workload is, the less time that will be spent in seek operations.  A high degree of sequentially ultimately leads to faster disk response and higher throughput rates. Sequential workloads may be candidates for slower disks or RAID levels.  Conversely, workloads that are highly randomized ought to be placed on fast spindles in fast RAID configurations.

    Examples of sequential workloads include backup-to-disk operations and the writing of SQL transaction log files.  Random workloads may include collective reads from Exchange Information Stores or OLTP database access.  Workloads are often a mix of random and sequential access, as is the case with most VMware vSphere implementations.  The degree to which they are random or sequential dictates the type of tuning you should perform to obtain the best possible performance for your environment.

I/O Request Size

    I/O request size is another important factor in workload characterization.  Generally speaking, larger reads/writes are more efficient than smaller I/O to a certain point.  The use of larger I/O requests (64KB instead of 2KB, for example) can result in faster throughput and reduced processor time.  Most workloads do not allow you to adjust your I/O request size.  However, knowing your I/O request size can help with appropriate configuration of certain parameters such as array stripe size and file system cluster size.  Check with your storage vendor for more information as it pertains to your specific configuration.

    If you are in a Windows shop, you can use perfmoncounters such as Avg. Disk Bytes/Read to determine average I/O size.  If you are running a VMware-virtualized workload, you can take advantage of a great tool – vscsiStats– to identify your I/O request size (see below).

Read vs. Write

    Every workload will display a differing amount of read and write activity.  Sometimes a specific workload, say Microsoft Exchange, can be broken down into sub-workloads for logging (write-heavy) and reading the database (read-heavy).  Understanding the read-to-write ratio may help with designing the underlying storage system.  For example, a write-heavy workload may perform better on a RAID10 LUN than a RAID5 array due to the write penalty associated with RAID5.  The ratio of read:write may also dictate caching strategies.  The read:write ratio, when combined with a degree of randomness measure, can be quite useful in architecting your storage strategy for a given application or workload.

Parallelism/Outstanding I/O’s

    Some workloads are capable of performing multi-threaded I/O.  These types of workloads can place a higher amount of stress on the storage system and should be understood when designing storage, both in terms of IOPS and throughput.  Multipathing may help with multi-threaded I/O workloads.  A typical VMware vSphere environment is a good example of a workload capable of queuing up outstanding I/O.

Measuring the Characteristics of Your Workload

    So how do we actually characterize storage workloads?  Start with the application vendor – many have published studies that can shed light on specific storage workloads in a standard implementation.  If you are interested in measuring your own for planning/architecture reasons, or performance troubleshooting reasons, read on….  There are several tools to measure storage characteristics, depending on your operating system and storage environment.  Standard OS performance counters, such as Windows Performance Monitor (perfmon) can reveal some of the characteristics.  Array based tools such as Unisphere Analyzer on EMC CLARiiON and VNX can also reveal statistics on the storage end of the equation. There are also many free disk performance benchmark tools at host side, such as IOZone, Iometer, FIO, etc.

    One of the most exciting tools for storage workload characterization comes from VMware in the form of vscsiStats.  vscsiStatsis a tool that has been included in VMware ESXi server since version 4.1.  Because all I/O commands pass through the Virtual Machine Monitor (VMM), the hypervisor can inspect and report on the I/O characteristics of a particular workload, down to a unique VM running on an ESX host. You can use vscsiStats to get the following performance metrics: ioLength, seekDistance, outstandingIOs, latency, interarrival, etc.

    Besides vscsiStats, VMware esxtopcan also be used to analyze storage workload and performance issue. But now it only supports Fibre Channel and iSCSI protocols. Refer to VMware KB1008205 for more details.

    Note: Before using vscsiStats and esxtop, you must enter ESXi Shell (formally Tech Support mode) first. Refer to VMware KB2004746 for the details.

Author: Roger