||Oracle Tips by Burleson
Solid-State Disk with RAC
Oracle Real Application Clusters(RAC) is a virtualized memory system that depends on a
high speed interconnect to provide a single memory image across
multiple host machines. This concept is illustrated in Figure 3.1.
Performance in a RAC environment is generally
maximized when a majority of data is cached within the virtualized
memory area. Once a database is
fully cached, the performance of the system then becomes dependent
upon the speed of the memory-to-memory transfers via the interconnect.
As a result, the overall performance of the entire system will be
dependent on the speed and latency of the cluster interconnect
When the system is not fully cached, performance
becomes codependent on the underlying disk I/O subsystem and the
interconnect with a convoy effect. The convoy effect is a term used to describe how a set
of dependent objects such as ships, cars, or computers becomes limited
by the speed of the slowest member. In most computer systems where
there is an interconnect and disks, the disks are the slowest
performers. Therefore, the speed at which an I/O subsystem can
service requests depends ultimately on the latency of the systems
disks, even when masked with memory caches.
The I/O subsystem consists of device types which
map to drivers. In turn, the drivers send the requests for I/O to a
host bus adapter (HBA). The HBA converts the request to the proper
format for the protocol being used, such as fiber, copper, or other
protocols. The request is then transferred over that protocol to the
storage subsystem I/O bus to the storage controller, depending on
whether the request can be satisfied via a cache read or must be read
from disk the controller either reads from cache or disk. A typical
subsystem is shown in Figure 3.2.
Of course the disk I/O subsystem also allows
virtualization of the true underlying disk structures as shown in
Figure 3.3. All of this I/O subsystem and disk activity can exact an
extreme penalty on Oracle and Oracle RAC performance if any piece of
the underlying structure is improperly configured.
Possible configuration issues include the SCSI
interface, the proper use of disk managers, proper application of
striping technology, proper stripe width and depths and the list goes
on and on. Many times the system administrator and DBA throw up their
hands and just accept whatever the vendor does to setup the disks and
I/O subsystem. This often leads to sub-optimal performance.
An example of a disk array parameter that directly
impacts performance is disk stripe width. For the purpose of this
text, disk stripe width defined to mean the amount of a stripe in a
RAID configuration that resides on one disk. In early systems, the
advice of many experts was to stripe shallow and wide. For example,
one might use an 8k stripe width across 32 drives. This was based on
rapidly retrieving a single large file. However, Oracle usually
retrieves small reads such as single data values after index lookups.
It does hundreds, if not thousands, of these in a single second in
large systems. This means Oracle disk arrays must be tuned for
concurrency. In Oracle systems, a stripe width that is too narrow is
guaranteed to generate I/O contention. Oracle recommends a stripe
width of between 128K (non-data) to 1 megabyte (data).
With any physical device that depends on the
mechanical motion of actuator arms and spinning disks, there is also
the issue of disk arm positional latency and disk rotational latency.
They severely limit the effective speed of I/O for physical disks.
Generally, non-linear I/O rates are only in the tens to hundreds of
megabits per second. Many disk array manufacturers are getting around
this through the implementation of huge memory caches, so not only do
users get to pay for the disks, they get to pay for the memory caches
as well. With disks running anywhere from $500-$3000 US, depending on
the vendor from which they are acquired and what firmware has been
loaded onto them, paying for both seems a bit redundant. In an SSD
environment, the disks are there strictly as a backing store, since no
large I/O rate is needed for this, there is no need to buy five or ten
disks to get the I/O needed. Just one can cover several units;
although for redundancy, the backing store is usually mirrored.
Solid-state disk (SSD) technology is replacing
ancient spinning platters of magnetic coated media with an array of
super fast solid-state RAM. Today’s SSD devices achieve tertiary
storage with software mechanisms that write the RAM frames to a back
end disk on the device.
With the cost of SSD at only $1k/gig, many Oracle
systems administrators are exploring how to leverage this powerful
performance tool for their environment. Smaller databases can now run
fully cached with SSD, yet there is a debate about the proper use of
SSD in an Oracle environment. Should it be used for redo? How about
undo segments? Maybe temp? This debate becomes even more muddled with
real application clusters (RAC) thrown into the picture.
With a standard Oracle database, the SSD array,
when not used for the entire database, is best used for the files
where the most I/O contention exists. For example, in an online
transaction system indexes, redo, or temp may be best placed on SSD,
while in a decision support or warehouse, temp and data files may be
the best choice.
As this study has determined, RAC files placed on
SSD may differ completely from those that might be considered in a
The above book excerpt is from:
Oracle RAC & Tuning with
Solid State Disk
Expert Secrets for High
Performance Clustered Grid Computing
Donald K. Burleson & Mike Ault