SCSI RDMA Protocol

The Linux SCSI Target Wiki

(Difference between revisions)
Jump to: navigation, search
Line 165: Line 165:
modprobe ib_srpt
modprobe ib_srpt srp_max_req_size=4200

Revision as of 05:41, 28 March 2011

LIO Target
Mellanox Technologies, Ltd.
Mellanox Infiniband SRP fabric module
Original author(s) Vu Pham
Developer(s) Mellanox Technologies, Ltd.
Initial release March 18, 2012 (2012-03-18)
Stable release 4.1.0 / June 20, 2012;
9 years ago
Preview release 4.2.0-rc5 / June 28, 2012;
9 years ago
Development status Beta
Written in C
Operating system Linux
Type Fabric module
License GNU General Public License
See Target for a complete overview over all fabric modules.

An SCSI RDMA Protocol (SRP, aka SCSI Remote Protocol) fabric module (tcm_ib_srpt) for Mellanox HCAs is available upon request. SRP defines a SCSI protocol mapping onto the Infiniband architecture and/or functionally similar cluster protocols.

The use of RDMA generally allows higher throughput and lower latency than TCP/IP based communication. RDMA is only possible with network adapters that support RDMA in hardware. Examples of such network adapters are InfiniBand HCAs and 10 GbE network adapters with iWARP support. While the SRP protocol has been designed to use RDMA networks efficiently, it is also possible to implement the SRP protocol over networks that do not support RDMA.

SRP is easier to implement than iSER, but iSER offers more management functionality, e.g. the target discovery infrastructure enabled by the iSCSI protocol.



The SRP driver is currently being tested and optimized.


SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:


This code won't be included in Linux before kernel v2.6.40. However using Linux >= v2.6.38, this and other HW target mode modules will be able to function. The installation and setup instructions are below. Please just contact us for RTSadmin.

Getting the sources

Initial build: git setup

git clone lio-core from git:// as follows:

git clone git://
cd lio-core-2.6
git checkout --track -b tcm_ib_srpt-38 origin/tcm_ib_srpt-38
git pull origin tcm_ib_srpt-38
cp <your_working_config_file> .config
make menuconfig

In menuconfig:

 <M> Generic Target Core Mod (TCM) and ConfigFS Infrastructure  --->
            --- Generic Target Core Mod (TCM) and ConfigFS Infrastructure     
                   <M>   TCM/IBLOCK Subsystem Plugin for Linux/BLOCK              
                   <M>   TCM/FILEIO Subsystem Plugin for Linux/VFS           
                   <M>   TCM/pSCSI Subsystem Plugin for Linux/SCSI                                        
                   < >   TCM/STGT Subsystem Plugin                             
                   <M> iSCSI Target Mode Stack
                   [ ]   LIO-Target iscsi_debug.h ring buffer messages
                   [ ]   LIO-Target ErrorRecoveryLevel debug code                   
  <M> InfiniBand support  --->       
                 --- InfiniBand support                                                                   
                 <M>   InfiniBand userspace MAD support                                                  
                 <M>   InfiniBand userspace access (verbs and CM)                                       
                 <M>   Mellanox HCA support                                                             
                 < >   QLogic PCIe HCA support                                                          
                 < >   Ammasso 1100 HCA support                                                          
                 <M>   Mellanox ConnectX HCA support                                                    
                 < >   NetEffect RNIC Driver                                                          
                 <M>   IP-over-InfiniBand                                                                
                 [ ]     IP-over-InfiniBand Connected Mode support                                      
                 -*-     IP-over-InfiniBand debugging                                                    
                 [ ]       IP-over-InfiniBand data path debugging                                        
                 < >   InfiniBand SCSI RDMA Protocol                                                    
                 <M>   InfiniBand SCSI RDMA Protocol target support                                      
                 < >   iSCSI Extensions for RDMA (iSER)            

If CONFIGFS_FS=y is set, a manual mount is required after the kernel boots:

mount -t configfs configfs /sys/kernel/config

Subsequent builds: git pull

Go to the directory where your lio-core kernel is located, and:

cd <lio-core-2.6>
git pull origin tcm_ib_srpt-38
make clean
make oldconfig


Kernel and modules:

make modules
make modules_install
make install


New Linux kernel

Use lio-core-2.6 kernel:

Then reboot:


and verify that the live kernel is indeed your new lio-core-2.6.git/tcm_ib_srpt-38 kernel:

uname -a 

Loading LIO Target

Verify the presence of the following required modules with modprobe: ib_srpt, target_core_mod, configfs, e.g.:

modprobe ib_srpt srp_max_req_size=4200

Normally all these modules should be loaded. If one is missing, you can load it as follows:

modprobe configfs
# Or do the mount command as shown above (if .config file has CONFIGFS_FS=y)
modprobe target_core_mod
modprobe ib_srpt

Starting LIO Target

The target will be automatically started if lio-utils/RTSadmin-frozen is installed. Otherwise, start the target manually:

/etc/init.d/target start

To stop target:

/etc/init.d/target stop

To check the target status:

/etc/init.d/target status

User-space tools

There are two options: lio-utils and the RTSadmin-frozen packages; at the very minimum, lio-utils is required. RTSadmin requires the libreadline6 package to be installed, too. The links for these rpm packages are:

You can also build lio-utils.git in-house as follows. Install rpmbuild and a libsnmp*-dev package, then:

make user_rpms

The 32-bit package should be built in /usr/src/packages/RPMS/.

Configure from lio-utils

Please see Lio-utils_HOWTO#Configuring_the_Target for creating a /sys/kernel/config/target/core/$HBA/$DEV/ group using TCM_node for PSCSI, IBLOCK, FILEIO and RAMDISK backends.

Configuration from RTSAdmin

If RTSadmin is installed:

<rtsadmin> create hba fileio
<rtsadmin> create storage <mylun_name> /tmp/<myfile_name> <size>

Where, e.g.:

<rtsadmin> cfspath
  Current context object lives in
<rtsadmin> saveconfig
<rtsadmin> quit

At this point, there are still a few manual steps required, which will soon be incorporated into rtsadmin:

# Load the mlx4_ib driver and register HW as an IB HCA.
modprobe mlx4_ib
# Load the ib_srpt module, if not already done earlier
modprobe ib_srpt srp_max_req_size=4200
# Create the 'srpt/mlx4_0/mlx4_0' endpoint, and create 'lun_0' group
mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0
# Create the target core symlink from 'fileio_0/my_file/' to 'lun/lun_0/srpt_port'
ln -s /sys/kernel/config/target/core/fileio_0/my_file/ /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0/srpt_port
# Create the explict NodeACL for SRP Initiator:
mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/acls/0x00000000000000000002c903000e8be9/lun_0
# Create the MappedLUN=0 for TPG LUN=0 for explict NodeACL for SRP Initiator
ln -s /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0 /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/acls/0x00000000000000000002c903000e8be9/lun_0/mapped_lun

Substitute the current target WWN portname (mlx4_0/mlx4_0) with your actual driver name from /sys/class/infiniband/, which can be obtained by:

ls /sys/class/infiniband/

Substitute the Initiator WWN portname (0x00000000000000000002c903000e8be9) with your actual IB Port GUID in long form, which can be obtained by running ibstat on the SRP Initiator:

truelife:/usr/src/lio-core-2.6.git# ibstat
CA 'mlx4_0'
	CA type: MT26428
	Number of ports: 2
	Firmware version: 2.7.9230
	Hardware version: b0
	Node GUID: 0x0002c903000e8be8
	System image GUID: 0x0002c903000e8beb
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 1
		LMC: 0
		SM lid: 2
		Capability mask: 0x02510868
		Port GUID: 0x0002c903000e8be9
	Port 2:
		State: Down
		Physical state: Polling
		Rate: 70
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x02510868
		Port GUID: 0x0002c903000e8bea

Once the explict Node ACL and MappedLUN have been for the initiator configured, it's time to login from the initiator side.

This includes using ibsrpdm -c to obtain the necessary string:

truelife:/usr/src/lio-core-2.6.git# ibsrpdm -c

And then echoing this into sysfs add_target attribute on the SRP Initiator to perform the SRP_LOGIN:

truelife:/usr/src/lio-core-2.6.git# echo -n id_ext=0002c903000e8acc,ioc_guid=0002c903000e8acc,dgid=fe800000000000000002c903000e8acd,pkey=ffff,service_id=0002c903000e8acc > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target 


fileio0/my_file(/tmp/my_file)> ls attribute



Please see SCSI RDMA Protocol/configFS for the running layout of a /sys/kernel/config/target/srpt configuration

lsmod output

 Module                  Size  Used by
 ib_srpt                26028  6 
 ib_umad                 9422  8 
 iscsi_target_mod      201391  1 
 mlx4_ib                38241  0 
 target_core_pscsi      11045  0 
 target_core_file        6307  1 
 target_core_iblock      6606  1 
 target_core_mod       206442  28 ib_srpt,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
 configfs               19153  2 target_core_mod
 ib_iser                23172  0 
 rdma_cm                23833  1 ib_iser
 ib_cm                  24878  2 ib_srpt,rdma_cm
 iw_cm                   6019  1 rdma_cm
 ib_sa                  15537  2 rdma_cm,ib_cm
 ib_mad                 30330  5 ib_srpt,ib_umad,mlx4_ib,ib_cm,ib_sa
 ib_core                39716  9 ib_srpt,ib_umad,mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
 ib_addr                 3982  1 rdma_cm
 ipv6                  242427  21 ib_addr
 iscsi_tcp               7729  0 
 libiscsi_tcp           10485  1 iscsi_tcp
 libiscsi               29466  3 ib_iser,iscsi_tcp,libiscsi_tcp
 scsi_transport_iscsi    24624  4 ib_iser,iscsi_tcp,libiscsi
 dm_multipath           14016  0 
 scsi_dh                 4656  1 dm_multipath

Interrupt stats

MSI-X interrupt stats for target side operation:

 lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | grep mlx4
 43:   16864937   16864950   16864795   16864897   PCI-MSI-edge      mlx4-comp-0@pci:0000:00:05.0
 44:          0          0          0          0   PCI-MSI-edge      mlx4-comp-1@pci:0000:00:05.0
 45:          0          0          0          0   PCI-MSI-edge      mlx4-comp-2@pci:0000:00:05.0
 46:          0          0          0          0   PCI-MSI-edge      mlx4-comp-3@pci:0000:00:05.0
 47:       1377       1308       1338       1322   PCI-MSI-edge      mlx4-async@pci:0000:00:05.0


Kernel ring buffer

Initiator side ring buffer output during initiator login:

 scsi7 : SRP.T10:0002C903000E8ACC
 scsi 7:0:0:0: Direct-Access     LIO-ORG  RAMDISK-MCP      4.0  PQ: 0 ANSI: 5
 sd 7:0:0:0: Attached scsi generic sg2 type 0
 sd 7:0:0:0: [sdb] 524288 512-byte logical blocks: (268 MB/256 MiB)
 sd 7:0:0:0: [sdb] Write Protect is off
 sd 7:0:0:0: [sdb] Mode Sense: 2f 00 00 00
 sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
  sdb: unknown partition table
 sd 7:0:0:0: [sdb] Attached SCSI disk

Device identifiers

sg_inq -i initiator side output for EVPD=0x83 device identifiers:

 truelife:/usr/src/lio-core-2.6.git# sg_inq -i /dev/sdb
 VPD INQUIRY: Device Identification page
   Designation descriptor number 1, descriptor length: 20
     designator_type: NAA,  code_set: Binary
     associated with the addressed logical unit
       NAA 6, IEEE Company_id: 0x1405
       Vendor Specific Identifier: 0x929cc061d
       Vendor Specific Identifier Extension: 0xe082d4d98d810cdc
   Designation descriptor number 2, descriptor length: 61
     designator_type: T10 vendor identification,  code_set: ASCII
     associated with the addressed logical unit
       vendor id: LIO-ORG
       vendor specific: RAMDISK-MCP:929cc061-e082-4d98-810c-c90b421bd066
   Designation descriptor number 3, descriptor length: 8
     transport: Remote Direct Memory Access (RDMA)
     designator_type: Relative target port,  code_set: Binary
     associated with the target port
       Relative target port: 0x2
   Designation descriptor number 4, descriptor length: 8
     transport: Remote Direct Memory Access (RDMA)
     designator_type: Target port group,  code_set: Binary
     associated with the target port
       Target port group: 0x0
   Designation descriptor number 5, descriptor length: 8
     designator_type: Logical unit group,  code_set: Binary
     associated with the addressed logical unit
       Logical unit group: 0x0
   Designation descriptor number 6, descriptor length: 32
     transport: Remote Direct Memory Access (RDMA)
     designator_type: SCSI name string,  code_set: UTF-8
     associated with the target port
       SCSI name string:

ibsrpdm output

 truelife:/usr/src/lio-core-2.6.git# ibsrpdm 
 IO Unit Info:
     port LID:        0002
     port GID:        fe800000000000000002c903000e8acd
     change ID:       0001
     max controllers: 0x10
     controller[  1]
         GUID:      0002c903000e8acc
         vendor ID: 000002
         device ID: 00673c
         IO class : 0100
         ID:        Linux SRP target
         service entries: 1
             service[  0]: 0002c903000e8acc / SRP.T10:0002c903000e8acc


Timeline of the LinuxIO
Release Details 2011 2012 2013 2014 2015
123456789101112 123456789101112 123456789101112 123456789101112 123456789101112
4.x Version 4.0 4.1
Feature LIO Core Loop back FCoE iSCSI Perf SRP
vHost Perf Misc 16 GFC iSER Misc VAAI Misc DIF Core
DIF iSER DIF FC vhost TCMU Xen Misc Misc virtio 1.0 Misc NVMe OF
Linux 2.6.38 2.6.39 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22

See also


External links

Personal tools
Google AdSense