SCSI

Serial Attached SCSI or SAS is most often used to connect data drives within storage subsystems,
while SATA is often used within PCs.

Small Computer Systems Interface or SCSI was developed many years ago to create a standard set
of commands and connections to address physical disks. The original technology used parallel ribbon
cables but parallel SCSI has now been more or less replaced by Serial Attached SCSI, or SAS. The
other main technology that is used to connect disks to motherboards is SATA, but SAS has one major
advantage over SATA, it has a 'dual path' facility which means that if one path fails, the disk can still
be accessed down the other path.
The big storage vendors all connect disks together under a storage controller and use different
classes of storage so that data can be 'tiered' or placed on the correct storage type depending on its
current value. All the big vendors now use SAS for the lowest disk tier, and solid state for the highest
tier. Fiber channel disk was traditionally used as a middle tier, but SAS is now fast enough to compete
with FC disks for performance, and are about one third of the price.
The combination of all these factors means that SAS is competing with SATA in PC and appliance
applications, has replaced SATA in enterprise disks, and will almost certainly replace Fiber Channel
too. While SAS itself will probably be replaced by the faster protocols used to support SSD devices, it
will be around for years yet.

The Speeds of the various SAS release are

SAS-1: 3.0 Gbit/s, introduced in 2005

Serial Attached SCSI (SAS)

There are three types of SAS devices: initiators, targets and expanders. Initiators attach to one or
more targets and form a SAS domain and a SAS domain can consist of up to 128 devices. The
initiators also need a service delivery system to be able to communicate with the targets.

Initiator devices can be host bus adaptors (HBAs), controllers or an on-board component on a
motherboard. They have multiple ports for connecting to internal and external targets and each
initiator port can be narrow ports with a single physical link, or wide ports with two, four, or eight
physical links. If you want failover redundancy then you can connect SAS initiator ports to separate
domains.

Target devices include SAS disks (HDDs) and solid state drives, SATA disks and SAS tape drives.
SAS drives have two narrow ports and so can connect to two different domains for failover
redundancy. They use the same electrical and physical connection interface as SATA drives.
However, SATA drives have a single narrow port and so no failover. You can mix SAS and SATA
devices in the same domain.

Expanders are low-cost, high-speed switches that are used to increase the number of targets
attached to an initiator and so create a larger SAS domain. Expanders come with 12, 24 or 36 port
and can connect to initiators, targets and other expanders. They receive commands and data in one
port and route them to another port based on the SAS address of the target.
Expanders use three routing methods: direct, table, and subtractive. If the target device is directly
attached to the expander then the expander uses direct routing. Table routing is used to forward the
command and data to another expander and subtractive routing is used when the expander does not
recognise the address. In this case, the expander just forwards the command and data to another
expander that recognizes the address.
All these methods use routing tables that are created in each expander during the discovery process
known as self-configuration. The size of the expanders' routing tables determines how many initiators
and targets you can have in a domain, but in theory you could develop a SAS network that could
address as many as 16,256 physical devices.

It is tempting to say that the 'service delivery system' is just the cables that connect the SAS
components together but of course it is a bit more complicated than that. SAS-2.1 and above supports
active cables, which are thin cables with active circuitry including built-in drivers, repeaters, and an
equalizing filter. These components are designed to reduce signal distortion and transmission loss, so
thinner, longer cables can be used.
SAS also uses differential signaling (LVD), where the signaling is transmitted over two signal wires,
with one wire transmitting the inverse of the signal on the other wire. The receiver reads the
differential voltage between the two wires and so filters out noise, and ignores signal voltage drift.
SAS communication operates in full duplex mode, which means that it can send and receive
information simultaneously over two wire pairs, and the SAS interface lets vendors scale up
bandwidth by installing multiple physical links to create 2x, 3x, 4x, or 8x connections per port.
The actual SAS point-to-point connectors are small as they only need 4 wires, so they can be fitted to
2.5 in. hard disk drives.

SAS devices use port identifiers to ensure that communications are sent to the correct device. Each
SAS port in a SAS domain has a SCSI port identifier that is assigned by the device manufacturer and
identifies the port uniquely within the SAS domain (and usually world-wide unique too). This is usually
enough to identify a device, but every SAS device also has a SCSI device name, which identifies the
SAS device uniquely in the world.
Both the SCSI port identifiers and the SCSI device names consist of a 64 bit SAS address. For a SAS
expander device, the SCSI port identifier and SCSI device name are the same SAS address. The
SAS address is analogous to the WWN address in a Fibre Channel configuration.

It is possible to connect a SATA drive to a SAS interface, though you cannot connect a SAS drive to a
SATA interface. This is because the SATA connector is keyed with a small notch between the data
and power connections. SAS has no key or notch, so while a SATA cable cannot fit a SAS socket, a
SAS cable will fit a SATA socket. It is also possible to mix and match SAS and SATA on the same
SAS interface. To achieve this, the SAS designers made the SAS connector form-factor compatible
with SATA, and the SATA connector signals are a subset of SAS signals that enable compatibility
between SATA devices and SAS controllers. Upgrading from SATA to SAS is as simple as replacing
the disk drives.
SAS uses three different protocols to communicate with the different device types, and automatically
selects the correct one depending on the device type it is accessing

 Serial SCSI Protocol (SSP) communicates with standard SAS devices using SCSI commands
 SATA Tunneling Protocol (STP) wraps a SATA protocol into SAS packets for communication
with SATA devices. Check out the SATA page for details.
 Serial Management Protocol (SMP) communicates management information to expanders

SAS now comes with some of its logic embedded in chipsets. For example, a SAS RAID on chip
controller (ROC) which takes data from a PCI or PCI-X host interface and RAIDs it to 4 or 8 SAS
connections on a card. A SAS IO controller is similar to the above, but without the RAID facility. A
SAS / SATA MUX dual ports a SAS drive to allow connection to two RAID controllers for high
availability applications.

The picture below illustrates two SAS initiators connecting a mix of six SAS and SATA disks. The
upper initiator is connected to both the fan-out expander and one of the edge expanders, so it has
some failover resilience. The lower initiator is only connected to one edge expander so it has no
resilience if that expander fails. However the lower initiator can see all the disks. The upper initiator
can see all the disks except the one that in connected directly to the lower initiator.
 iSCSI

A Fibre channel SAN requires a dedicated data network, separate from the LAN and this is costly to
provision. It also requires that all the devices attached to the SAN be fitted with expensive Fibre
Channel HBAs. An IP SAN removes this requirement as it works over the same network as the user
traffic and used standard HBAs. This may be an issue as storage traffic will compete with user traffic,
so it is best to implement iSCSI on a VLAN to isolate the traffic. Standard SCSI is limited to short
distances, typically tens of meters. iSCSI encapsulates SCSI commands in TCP/IP packets and can
send them over more or less any distance. This is useful for compliance driven long distance
replication as this needs to transmit at block level. Of course, fibre does this but it is more expensive.
iSCSI also drives IO at block level and is relatively cheap.
Most SAN switches are multi-protocol, so they will support and combine IP and Fibre channel in a
SAN and can convert between SCSI and FC protocols.

SCSI Express

SCSI Express is a combination of a SCSI disk interface and PCI express connectivity. It was designed
to manage the high I/O requirements of solid state disks. However it is competing with NVMe, which is
much faster. SAS just supports a single IO queue, with a maximum of 256 queued commands,
because it was designed for spinning hard drives, which are restricted by the read head to only
transfer of one unit of data at a time. An SSD can retrieve many pieces of data simultaneously, as it
does not have this hardware restriction. NVMe can take advantage of this parallelism it almost has an
unlimited number of queues, though it typically uses 4 lane PCIe to get parallelism. So what does this
mean in real terms? To put some approximate numbers around this, a spinning hard drive will transfer
data at 200MBps, SSD over SCSI Express at 600MBps and SSD over NVMe 3GBps.

back to top

Disk Protocols

 NVMe protocol
 Parallel and Serial ATA
 Serial ATA 2
 SCSI

Enterprise Disk

 Disk Architecture
 Large storage disks
 3390 disks and ch. speeds
 FICON channels
 RAID levels
 PAV and MA
 PowerMAX overview
 VMAX overview
  VMAX CLI commands
 DS8K overview
 DS8K configuration
 ESS Config.
 NAS storage
 Flash Storage
 SCM storage

Lascon updTES

I retired 2 years ago, and so I'm out of touch with the latest in the data storage world. The Lascon site
has not been updated since July 2021, and probably will not get updated very much again. The site
hosting is paid up until early 2023 when it will almost certainly disappear.
Lascon Storage was conceived in 2000, and technology has changed massively over those 22 years.
It's been fun, but I guess it's time to call it a day. Thanks to all my readers in that time. I hope you
managed to find something useful in there.
All the best

DISCLAIMER - By entering and using this site, you accept the conditions and limitations of use

Click here to see the Full Site Index       Click here to see the Cookie Policy       Click here to see
the Privacy Policy                             ©2019 and later, A.J.Armstrong