LIO

The Linux SCSI Target Wiki

Latest revision as of 02:17, 12 March 2016

LinuxIO

LinuxIO
Original author(s)	Nicholas Bellinger
Developer(s)	Datera, Inc.
Initial release	January 14, 2011 (2011-01-14)
Stable release	4.1.0 / June 20, 2012; 8 years ago (2012-06-20)
Preview release	4.2.0-rc5 / June 28, 2012; 8 years ago (2012-06-28)
Development status	Production
Written in	C
Operating system	Linux
Type	SCSI target
License	GNU General Public License, version 2 (GPLv2)
Website	datera.io

architecture overview.

LIO iSCSI architecture diagram.

SCSI Portal Group and multipath architecture overview.

SCSI standards diagram.

QLogic Fibre Channel running at line rate in target mode with PCIe device passthrough and MSI-X polled interrupts across Linux/SCSI qla2xxx LLD request and response rings.

The IEEE Firewire driver exporting IBLOCK LUNs to a MacOS-X based IEEE-1394/FireWire client.

LinuxIO (LIO) has been the Linux SCSI target since kernel version 2.6.38.^[1][2] It supports a rapidly growing number of fabric modules, and all existing Linux block devices as backstores.

Overview

LinuxIO is based on a SCSI engine that implements the semantics of a SCSI target as described in the SCSI Architecture Model (SAM), and supports its comprehensive SPC-3/SPC-4 feature set in a fabric-agnostic way. The SCSI target core does not directly communicate with initiators and it does not directly access data on disk.

LIO has obtained VMware vSphere 4 Ready certification on Buffalo [PDF] and QNAP [PDF] systems, and vSphere 5 Ready certification on Netgear [PDF] and Pure Storage [PDF] systems.

Native support in OpenStack (setup, code), starting with the Grizzly release, makes LIO also an attractive storage option for cloud deployments.

The LIO core (target_core.ko, see Linux kernel driver database) was released with Linux kernel 2.6.38 on January 14, 2011 (2011-01-14).^[3]

Setup

targetcli provides a comprehensive, powerful and easy CLI tool to configure and manage LIO. targetcli was developed by Datera.

Fabric modules

Fabric modules implement the "frontend" of the SCSI target. They "speak" specific protocols that transport SCSI commands. The Fabric Hardware Abstraction Layer (F-HAL) allows all protocol-specific processing to be encapsulated in fabric modules. The following fabric modules are available:

• Fibre Channel: SCSI access over FC networks with QLogic HBAs (qla2xxx.ko, released).
• Fibre Channel over Ethernet (FCoE): FC access over Ethernet (tcm_fc.ko, released).
• FireWire: SCSI access over FireWire / iEEE 1394 networks (sbp.ko, released).
• IBM vSCSI: IBM virtual SCSI (vSCSI) module for IBM System p (ibmvscsis.ko, RC).
• iSCSI: SCSI access over IP networks (iscsi.ko, released).
• iSER: iSCSI access over InfiniBand networks with Mellanox HCAs (ib_sert.ko, released)
• SRP: SCSI access over InfiniBand networks with Mellanox HCAs (srpt.ko, released).
• tcm_loop: SCSI emulation device to serve any type of raw hardware to local applications and VMs as a fully SCSI SPC-4 compliant block device, including emulation of advanced functionality (loopback.ko, released).
• USB Gadget: SCSI access over USB networks (tcm_usb_gadget.ko, released).
• vHost: QEMU virtio and virtio-scsi PV guests (tcm_vhost.ko, released).

Backstores

Backstores implement the LIO "backend". They implement the methods of accessing data on disk. A backstore subsystem plugin is a physical storage object that provides the block device underlying a SCSI Endpoint.

Backstore objects can be added via the Storage Hardware Abstraction Layer (S-HAL) that brings storage hardware into LIO engine as raw block devices, on which the Linux stack just works (including complex functionality such as software RAID, LVM, snapshots, virtualization, etc.).

LIO supports the SCSI-3 standard for all backstore devices (block devices and/or VFS):

• FILEIO (Linux VFS devices): any file on a mounted filesystem. It may be backed by a file or an underlying real block device. FILEIO is using struct file to serve block I/O with various methods (synchronous or asynchronous) and (buffered or direct). The Linux kernel code for filesystems resides in linux/fs. By default, FILEIO uses non-buffered mode (O_SYNC set).

Do not use buffered FILEIO Creating a FILEIO backend with buffered=True enables the buffer cache. While this can provide significant performance increases, it also creates a serious data integrity hazard: If the system crashes for any reason, an unflushed buffer cache can cause the entire backstore to be irrecoverably corrupted.

• IBLOCK (Linux BLOCK devices): any block device that appears in /sys/block. The Linux kernel code for block devices is in linux/block.

Recommended for performance IBLOCK backstores usually provide the best performance.

• PSCSI (Linux pass-through SCSI devices): any storage object that supports direct pass-through of SCSI commands without SCSI emulation. This assumes an underlying SCSI device that appears with lsscsi in /proc/scsi/scsi, such as a SAS hard drive, etc. SCSI-3 and higher is supported with this subsystem, but only for control CDBs capable by the device firmware.

Do not use PSCSI Do not use PSCSI unless you know exactly how it will be used. Advanced SCSI CDBs such as for Persistent Reservations or ALUAs (used e.g. by VMware ESX and vSphere) are typically not implemented in the device firmware, and can cause malfunctions or crashes. Use IBLOCK for production setups instead.

• Memory Copy RAMDISK (Linux RAMDISK_MCP): Memory Copy ram disks (rd_mcp) provide ram disks with full SCSI emulation and separate memory mappings using memory copy for initiators, thus providing multi-session capability. This is most useful for fast volatile mass storage for production.

SCSI functionality is implemented directly in the LIO engine in a fabric-agnostic way, including a number of high-end features, such as Persistent Reservations (PRs), Asymmetric Logical Unit Assignment (ALUA), vStorage APIs for Array Integration (VAAI) following the SPC-4 standard.

The backstore devices (FILEIO, IBLOCK, pSCSI, RAMDISK, etc.) report the underlying HW limitiations for things like TCQ depth, MaxSectors, TaskAbortedStatus, UA Interlocking, etc. All of these values are available as attributes in the targetcli device context.

Specifications

The following specifications are available as T10 Working Drafts:

• SCSI Architecture Model - 2 (SAM-2): SAM-2 describes the second generation of the SCSI Architecture Model. Status: Published, 2002-09-11
• SCSI Architecture Model - 3 (SAM-3): SAM-3 describes the third generation of the SCSI Architecture Model. Status: Published, 2004-09-21
• SCSI Architecture Model - 4 (SAM-4): SAM-4 describes the fourth generation of the SCSI Architecture Model. Status: Published, 2008-05-08

• SCSI Primary Commands - 2 (SPC-2): contains the second-generation definition of the basic commands for all SCSI devices. SPC-2 is used in conjuction with a standard for the specific device type. In 2006, the ANSI SPC-2 standard was replaced with ISO/IEC 14776-452:2005, which is available through ANSI. Status: Published, 2001-07-18
• SCSI Primary Commands - 3 (SPC-3): contains the third-generation definition of the basic commands for all SCSI devices. SPC-3 is used in conjuction with a standard for the specific device type. Status: Published, 2005-05-04
• SCSI Primary Commands - 4 (SPC-4): contains the fourth-generation definition of the basic commands for all SCSI devices. SPC-4 is used in conjuction with a standard for the specific device type. Status: Development, 2010-11-16

• Small Computer System Interface - 2 (SCSI-2): SCSI-2 defines the second generation of the Small Computer System Interface. Status: Published, 1993-09-07

Glossary

• Access Control List (ACL): Used to specify the access rights for Initiators to TPGs.
• Backstore: A physical storage object that provides the actual storage underlying an SCSI Endpoint.
• Command Descriptor Block (CDB): The standard format for SCSI commands. CDBs are commonly 6, 10, or 12 bytes long, though they can be 16 bytes or of variable length.
• Challenge Handshake Authentication Protocol (CHAP): An authentication technique for confirming the identity of one computer to another. Described in RFC 1994.
• Connection Identifier (CID): A 16-bit number, generated by the Initiator, that uniquely identifies a connection between two iSCSI devices. This number is presented during the login phase.
• Endpoint: The combination of an iSCSI TargetName with an iSCSI TPG Tag (IQN + Tag).
• Converged Network Adapter (CNA): An Ethernet PCIe network adapter (NIC) that natively supports RDMA (via RoCE), also called RNIC.
• Demo Mode: Means disabling authentification for an iSCSI Endpoint, i.e. its ACLs are diabled. Demo Mode grants read-only access to all iSCSI Initiators that attempt to connect to that specific Endpoint. See the iSCSI entry on how to enable Demo Mode.
• Data Integrity Field (DIF): Is an approach to protect data integrity in computer data storage. It was proposed in 2003 by the T10 committee of the International Committee for Information Technology Standards.
• Extended Unique Identifier (EUI): A 64-bit number that uniquely identifies every device in the world. The format consists of 24 bits that are unique to a given company, and 40 bits assigned by the company to each device it builds.
• I_T Nexus: An I_T Nexus denotes a live session between an Initiator and a target.
• Initiator: The originating end of a SCSI session. Typically a controlling device such as a computer.
• Internet Protocol Storage (IPS): The class of protocols or devices that use the IP protocol to move data in a storage network. FCIP, iFCP, and iSCSI are all examples of IPS protocols.
• iSCSI Qualified Name (IQN): A name format for iSCSI that uniquely identifies every device in the world (e.g. iqn.5886.com.acme.tapedrive.sn-a12345678).
• Initiator Session Identifier (ISID): A 48-bit number, generated by the Initiator, that uniquely identifies a session between the Initiator and the target. This value is created during the login process, and is sent to the target with a Login PDU.
• Multipath I/O (MPIO): A method by which data can take multiple redundant paths between a server and storage.
• Network Portal: The combination of an iSCSI Endpoint with an IP address plus a TCP port. The TCP port number for the iSCSI protocol defined by IANA is 3260.
• Network Interface Card (NIC): An Ethernet PCIe network adapter.
• N_Port ID Virtualization (NPIV): is a Fibre Channel facility allowing multiple N_Port IDs to share a single physical N_Port. This allows multiple Fibre Channel initiators to occupy a single physical port, easing hardware requirements in Storage Area Network design, especially where virtual SANs are called for. NPIV is defined by the Technical Committee T11 in the Fibre Channel - Link Services (FC-LS) specification.
• Non-Transparent Bridging (NTB): Non-transparent bridges in PCI systems support intelligent adapters in enterprise systems and multiple processors in embedded systems. The Intel DrawBridge established the paradigm of the embedded bridge and became a defacto standard in such environments as Compact PCI and intelligent adapters for enterprise systems. In these systems, the non-transparent bridge functions as a gateway between the local subsystem and the backplane.^[4]
• Organizationally Unique Identifier (OUI) is a 24-bit number that is purchased from the IEEE Registration Authority. This identifier uniquely identifies a vendor, manufacturer, or other organization (referred to by the IEEE as the “assignee”) globally or worldwide and effectively reserves a block of each possible type of derivative identifier (such as MAC addresses, group addresses, Subnetwork Access Protocol protocol identifiers, etc.) for the exclusive use of the assignee, see OUI Wikipedia entry. The OUI is subsequently used by the assignee to create particular instances of these identifiers for various purposes, such as the identification of a particular piece of equipment.
• RDMA NIC (RNIC): An Ethernet PCIe network adapter (NIC) that natively supports RDMA (via RoCE), also called CNA.
• SCSI Architectural Model (SAM): A document that describes the behavior of SCSI in general terms, allowing for different types of devices communicating over various media.
• SCSI Target: The receiving end of a SCSI session, typically a device such as a disk drive, solid state drive, tape drive, or scanner.
• Target Group: A collection of fabric module SCSI target endpoints containing SCSI target ports that provide access to individual storage objects.
• Target Port: The combination of an iSCSI Endpoint with one or more LUNs.
• TCP/IP Offload Engine (TOE): An enhanced NIC that runs a full TCP/IP stack independent from the main CPU and OS.
• Target Portal Group (TPG): A list of IP addresses and TCP port numbers that determines which interfaces a specific iSCSI target will listen to.
• Target Session Identifier (TSID): A 16-bit number, generated by the target, that uniquely identifies a session between the initiator and the target. This value is created during the login process, and is sent to the initiator with a Login Response PDU.
• World Wide Name (WWN) or World Wide Identifier (WWID) is a unique identifier which identifies a particular Fibre Channel, ATA or SAS target. Each WWN is an 8 byte number derived from an IEEE OUI and vendor-supplied information, see also WWN Wikipedia entry. There are two formats of WWN defined by the IEEE:
  • Original format: addresses are assigned to manufacturers by the IEEE standards committee, and are built into the device at build time, similar to an Ethernet MAC address. The first 2 bytes are either hex 10:00 or 2x:xx (where the x's are vendor-specified) followed by the 3-byte vendor identifier and 3 bytes for a vendor-specified serial number
  • New addressing schema: the first nibble is either hex 5 or 6 followed by a 3-byte vendor identifier and 36 bits for a vendor-specified serial number

Inclusion in Linux distributions

LIO and targetcli are included in most Linux distributions per default. Here is an overview over the most popular distributions:

1. ↑ The distribution release where LIO was included first.
2. ↑ Technical support, and qualified backports to other kernels and distributions are available from Datera.

Timeline

LIO and fabric modules have gone upstream into the Linux kernel as follows:

• Linux 2.6.38 (2011-03-14^[5]): LIO SCSI target core^[6]
• Linux 2.6.39 (2011-05-18^[7]): tcm_loop (SCSI support on top of any raw hardware)
• Linux 3.0 (2011-07-21^[8]): FCoE (by Cisco)
• Linux 3.1 (2011-10-24^[9]): iSCSI^[10]
• Linux 3.3 (2012-03-18^[11]): InfiniBand/SRP^[12] (Mellanox HCAs)
• Linux 3.5 (2012-07-21^[13]): Fibre Channel (QLogic HBAs),^[14] USB Gadget^[15] and IEEE 1394^[16]
• Linux 3.6 (2012-10-01^[17]): vHost (QEMU virtio and virtio-scsi PV guests)^[18]
• Linux 3.9 (2013-04-28^[19]): 16 GFC (QLogic HBAs)
• Linux 3.10 (2013-06-30^[20]): InfiniBand/iSER (Mellanox HCAs and CNAs)
• Linux 3.12 (2013-11-03^[21]): VAAI
• Linux 3.14 (2014-03-30^[22]): T10 DIF core, T10 Referrals, NPIV
• Linux 3.15 (2014-06-08^[23]): T10 DIF iSER
• Linux 3.16 (2014-08-03^[24]): Qlogic T10 DIF, vhost T10 DIF
• Linux 3.17 (2014-10-05^[25]): Userspace backend (TCMU), Xen host paravirtualized driver (xen-back)
• Linux 3.20 (planned): Update vhost-scsi to support virtio v1.0 specification requirements
• Linux 3.22 (planned): NVMe-OF reference target driver

Wikipedia

• Fabrics: ATA over Ethernet (AoE), FibreChannel over Ethernet (FCoE), FibreChannel (FC), InfiniBand (IB), iSCSI, iSCSI Extensions for RDMA (iSER), Internet Wide Area RDMA Protocol (iWARP), and SCSI RDMA Protocol (SRP)
• Services: Service Location Protocol (SLP), Internet Storage Name Service (iSNS)
• Other: Storage Area Network (SAN), Serial attached SCSI (SAS), World Wide Name (WWN)

Contact

Please direct all technical discussion on targetcli to the target-devel mailing list (post, subscribe, list info, gmane archive). Please see Support for more information.

See also

• LIO, targetcli
• Fabric modules: FCoE, Fibre Channel, IBM vSCSI, iSCSI, iSER, SRP, vHost
• Distributions: RHEL 4/5/6, SLES11, CentOS, Debian, Fedora, openSUSE, Ubuntu, etc.
• Platforms: PC architecture (x86/ia64), PlayStation 2 and PlayStation 3 (Cell), etc.
• Low level kernel API: configFS
• Standards: SCSI

Notes

External links

• LIO Wikipedia entries: German English
• LIO Admin Manual
• RTSlib Reference Guide [HTML][PDF]
• Adapters by SCSI connector type
• Anatomy of the Linux SCSI subsystem
• Organizationally Unique Identifier (OUI) Wikipedia entry
• SCSI Wikipedia entry
• SCSI Tutorial
• SCSI/iSCSI/RAID/SAS Information Sheet
• T10 Technical Committee (SCSI standards)
• T10 Technical Committee - Protocols and Physical Layers
• T10 Primary Commands (SPC): using SPC-3/4
• T10 Block Commands (SBC) using SBC-3
• wetpaint.com Fun storage stuff (Ming Zhang)