QLogic

The Linux SCSI Target Wiki

Revision as of 08:55, 10 April 2012

See Target for a complete overview over all fabric modules.

TCM_QLA2xxx running at line rate HW target mode from with PCIe device passthrough with MSI-X polled interrupts across Linux/SCSI qla2xxx LLD request and response rings

TCM v4.0 QLAXXX prototype using PCIe device passthrough for target mode MSI-X interrupts, and QEMU Megasas HBA emulation into Windows7 64-bit guest.

QLogic qla2xxx series Fibre Channel ASICs and HBAs are supported by a fabric module (tcm_qla2xxx.c) and a complementary qla2xxx target mode LLD, using the Target core infrastructure.

The latest version of the tcm_qla2xxx fabric module (master). The qla2xxx LLD target mode logic was refactored from the SCST qla2x00t driver, and contains a number of fixes and performance optimizations, including a modern LLD port (qla_target.c) of the qla2x00t target logic.

Status

The QLogic driver is in BETA, and currently being hardened in the QLogic test labs.

Here's an overview over the status:

The LLD changes and tcm_qla2xxx are now Beta status:

• ISP-2532 adapters have been tested in point to point (non NPIV) mode with ≥24xx code at 8 Gb/sec.
• As requested for mainline code, support for PCI-E <= 23xx has been removed in lio-core with the following commit

Setup

targetcli from Datera, Inc. is a comprehensive, powerful, yet easy storage management tool that can efficiently handle complex storage installations.

Getting the sources

Initial build: git setup

git clone lio-core from git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core.git as follows:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core.git 
cd lio-core
cp <your_working_config_file> .config
make menuconfig

In menuconfig:

• Select device drivers
• Select the TCM module
• Select the modules FILEIO, IBLOCK, PSCSI, Linux-iSCSI, then return to device drivers
• Select the SCSI device drivers, then SCSI low-level drivers.
• Select QLogic QLA2XXX Fibre Channel Support to build qla2xxx.ko, and TCM_QLA2XXX fabric modul to build tcm_qla2xxx.ko
• Save and return

Verify the following entries in the .config file (or make the corresponding changes):

CONFIG_SCSI_QLA_FC=m
CONFIG_TCM_QLA2XXX=m
CONFIG_INET=y
CONFIG_SCSI_FC_ATTRS=m
CONFIGFS_FS=m

If CONFIGFS_FS=y is set, a manual mount is required after the kernel boots:

mount -t configfs configfs /sys/kernel/config

Subsequent builds: git pull

Go to the directory where your lio-core kernel is located, and:

cd <lio-core-2.6>
git pull origin tcm_qla2xxx
make clean
make oldconfig

Building

Kernel and modules:

make
make modules
make modules_install
make install

Installing

New Linux kernel

Use lio-core-2.6 kernel:

• Change /boot/grub/menu.lst to select your lio-core-2.6 kernel.
• In menu.lst, make sure that vmalloc=256m is in the kernel line.

Normally, the latter is optional. But if /proc/meminfo shows vmalloc <128k, it fixes an issue with fcport_lport_map allocation seen by dmesg or in /var/log/messages. Note that 128m may be ok but too low, and 512m may be too high.

Then reboot:

reboot

and verify that the life kernel is indeed your new lio-core kernel:

uname -a 

Loading LIO Target

Verify the presence of the following required modules with modprobe: tcm_qla2xxx, target_core_mod, qla2xxx, iscsi_target_mod, configfs, e.g.:

modprobe tcm_qla2xxx

Normally all these modules should be loaded. If one is missing, you can load it as follows:

insmod fs/configfs/configfs.ko
# Or do the mount command as shown above (if .config file has CONFIGFS_FS=y)
insmod /drivers/target/target_core_mod.ko
insmod /drivers/target/iscsi_target_mod.ko
modprobe tcm_qla2xxx

Starting LIO Target

The target will be automatically started if targetcli-frozen is installed. Otherwise, start the target manually:

/etc/init.d/target start

To stop target:

/etc/init.d/target stop

To check the target status:

/etc/init.d/target status

Configuring

targetcli should be used to configure QLogic targets, please see QLogic/targetcli. The basic steps are as follows:

# targetcli
  ...
/> create hba fileio
  ...
/> create storage <mylun_name> /tmp/<myfile_name> <size>

Where, e.g.:

• size = 10G
• mylun_name could be my_lun
• myfile_name could be my_file
• This creates a storage object my_lun and a 10GB FILEIO backstore in /tmp/my_file.

Notes:

• By default, FILEIO uses O_SYNC. It allows both, exporting a shared filesystem for mounting, and creating files in the shared filesystem to be used as the backstore for raw block devices.
• Any number of virtual HBAs and assoicated devices is possible. The Target subsystem plugins (FILEIO, IBLOCK, pSCSI, RAMDISK, etc.) will report the underlying HW limitiations for things like TCQ depth, MaxSectors, TaskAbortedStatus, UA Interlocking, etc. All of these values are available as attributes in the targetcli device context:

# targetcli
  ...
/> fileio0/my_file(/tmp/my_file)> ls attribute
  ...

Results

ConfigFS

See QLogic/configFS for the ConfigFS and kernel ringbuffer output of a working qla2xxx driver.

Modules

lsmod output:

Module                  Size  Used by
tcm_qla2xxx            16822  8 
qla2xxx               327379  1 tcm_qla2xxx
iscsi_target_mod      201391  1 
target_core_pscsi      11045  0 
target_core_file        6307  1 
target_core_iblock      6606  0 
target_core_mod       207359  26 tcm_qla2xxx,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
configfs               19153  2 target_core_mod
ipv6                  242427  20 

Interrupt stats

MSI-X interrupt stats for target side operation:

lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | head -n 1 ; cat /proc/interrupts | grep qla2xxx
            CPU0       CPU1       CPU2       CPU3       
  43:      11697      11610      11951      11906   PCI-MSI-edge      qla2xxx (default)
  44:      14836      14922      14582      14628   PCI-MSI-edge      qla2xxx (rsp_q)

Initiator

Kernel ring buffer

Initiator side ring buffer output during initiator login:

scsi34 : qla2xxx
qla2xxx 0000:02:00.0: LIP reset occurred (f8f7).
qla2xxx 0000:02:00.0: LIP occurred (f8f7).
qla2xxx 0000:02:00.0: LOOP UP detected (8 Gbps).
scsi 34:0:0:0: Direct-Access     LIO-ORG  RAMDISK-MCP      4.0  PQ: 0 ANSI: 5
sd 34:0:0:0: Attached scsi generic sg2 type 0
sd 34:0:0:0: [sdb] 819200 512-byte logical blocks: (419 MB/400 MiB)
sd 34:0:0:0: [sdb] Write Protect is off
sd 34:0:0:0: [sdb] Mode Sense: 2f 00 00 00
sd 34:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 34:0:0:0: [sdb] Attached SCSI disk

Device identifiers

sg_inq -i initiator side output for EVPD=0x83 device identifiers:

truelife:/usr/src/qemu-kvm.git# sg_inq -i /dev/sdb
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    id_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x1405
      Vendor Specific Identifier: 0x7bb168f6d
      Vendor Specific Identifier Extension: 0x32edd42b3d9e26df
      [0x60014057bb168f6d32edd42b3d9e26df]
  Designation descriptor number 2, descriptor length: 61
    id_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: LIO-ORG
      vendor specific: RAMDISK-MCP:7bb168f6-32ed-42b3-9e26-fe6aab35460a
  Designation descriptor number 3, descriptor length: 8
    transport: Fibre Channel (FCP-2)
    id_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x1
  Designation descriptor number 4, descriptor length: 8
    transport: Fibre Channel (FCP-2)
    id_type: Target port group,  code_set: Binary
    associated with the target port
      Target port group: 0x0
  Designation descriptor number 5, descriptor length: 8
    id_type: Logical unit group,  code_set: Binary
      associated with the addressed logical unit
      Logical unit group: 0x0
  Designation descriptor number 6, descriptor length: 40
    transport: Fibre Channel (FCP-2)
    id_type: SCSI name string,  code_set: UTF-8
    associated with the target port
      SCSI name string:
      21:00:00:24:ff:31:4c:48,t,0x0001

Action Items

• Add configfs attribute in /sys/kernel/config/target/core/qla2xxx/ to show available struct scsi_qla_host available for target mode registration using mkdir -p /sys/kernel/config/target/qla2xxx/$QLA_FC_WWPN
• Enable proper target side SRR support in qla_target.c
• More testing of cable pull and shutdown with active I/O.
• Proper support for NPIV using the seperate /sys/kernel/config/target/qla2xxx_npiv fabric module logic.
• Proper testing with pre 24xx series adapters support

Acknowledgements

Kudos to the QLogic team, in particular Madhu Iyengar, for their excellent support and guidance with the qla2xxx driver.

See also

• targetcli: QLogic/targetcli
• ConfigFS: QLogic/configFS (low level kernel API)
• Other fabric modules: iSCSI, Fibre Channel, FCoE, InfiniBand, IBM vSCSI, tcm_loop
• Target

External links

• QLogic Wikipedia entry
• Fibre Channel Wikipedia entry
• QLogic official website