Installation notes for Linux iSCSI driver Copyright (C) 2001, 2002, Cisco Systems, Inc. maintained by linux-iscsi@cisco.com This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. See the file COPYING included with this distribution for more details. =============================================================================== CONTENTS =============================================================================== Before Installing the iSCSI Driver Installing And Configuring The iSCSI Driver =============================================================================== BEFORE INSTALLING THE iSCSI DRIVER =============================================================================== The iSCSI drivers, README files, and example configuration files are available on the Linux-iSCSI homepage at: http://linux-iscsi.sourceforge.com In addition, at the website you can get information about the availability of new drivers, updated drivers, driver compatibility, and other relevant information. =============================================================================== INSTALLING AND CONFIGURING THE iSCSI DRIVER =============================================================================== This section is the starting point for installing and configuring the iSCSI Driver for Linux and contains the following topics: o Product Overview o System Requirements o Installation Procedures o Device Names o Target Bindings o Mounting Filesystems o Unmounting Filesystems o Dynamic target and LUN discovery o /proc/scsi/iscsi o Uninstalling iSCSI o Kernel and driver upgrades ---------------- PRODUCT OVERVIEW ---------------- The iSCSI Driver for Linux provides an IP host with the ability to access storage through an IP network. The iSCSI driver acts as an iSCSI protocol initiator to transport SCSI requests and responses over an IP network between the IP host and an iSCSI target device such as a Cisco SN 5400 Series system. (The iSCSI protocol is an IETF-defined protocol for IP storage. For more information about the iSCSI protocol, refer to the IETF standards for IP storage at http://www.ietf.org.) Architecturally, the iSCSI driver combines with the IP host TCP/IP stack, network drivers, and NICs to provide the same functions as a SCSI or Fibre Channel adapter driver with an HBA. (See Figure 1.) FIGURE 1 iSCSI Driver Architecture ==================================================================== | IP host OS | Storage host OS and applications | and applications | ==================================================================== | +------------+ | +-----------+ | iSCSI | | | | | driver | | | SCSI | | | | | or | +---------+------------+ | | FC | | TCP/IP | | | adapter | +----------------------+ | | driver | | Network drivers | | | | +----------------------+ | +-----------+ | | | =============|=====================================|================ | | | +-----------+ | +-----------+ | NIC | | | HBA | +-----------+ | +-----------+ | | | =============|=====================================|================ | | | _|_ | | _( )_ | +-----------+ _( IP )_ | | SCSI or | (_ network _) | | FC media | (_ _) | +-----------+ (_ _) | | | | | +------------------+ | +------------------+ | iSCSI | | | Storage | | target | | +------------------+ | interface | | | ___ ___ ___ | +------------------+ | | (___)(___)(___) | | | | (___)(___)(___) | | | | (___)(___)(___) | +------------------+ | | (___)(___)(___) | | Storage | | +------------------+ +------------------+ | | ___ ___ ___ | | | (___)(___)(___) | | | (___)(___)(___) | | | (___)(___)(___) | | | (___)(___)(___) | | +------------------+ | | ==================================================================== The iSCSI driver provides a transport for SCSI requests and responses for storage devices; however, instead of providing a transport for directly attached devices, the driver transports the SCSI requests and responses between the IP host and an iSCSI target system via an IP network. The iSCSI target system, in turn, transports SCSI requests and responses between it and the storage devices attached to it. Once the iSCSI driver is installed and started, the IP host will proceed with a discovery process for iSCSI storage devices as follows: 1. The iSCSI driver requests available iSCSI targets from the iSCSI target system. 2. The iSCSI target system sends available iSCSI target names to the IP host. 3. The IP host logs in to the iSCSI targets. 4. The iSCSI target system accepts the IP host login. 5. The IP host queries targets for device information. 6. Targets respond with device information. 7. The IP host creates a table of internal devices. The iSCSI Driver for Linux provides IP access to a maximum of sixteen remote SCSI targets. Each target will be probed for up to 256 LUNs, until the Linux kernel's limit of SCSI devices has been reached. ------------------- SYSTEM REQUIREMENTS ------------------- This driver requires a 2.4.x Linux kernel version of 2.4.16 or later. Compilation will require the kernel header files matching the kernel version you wish to run the driver on. Kernels released after June 7, 2002 may or may not work with this river, depending on what if any changes have been made to the kernel's SCSI midlayer code. As of June 6, 2002 there are several issues with the Linux kernel code that can cause problems when using SCSI devices (including iSCSI devices). Linux kernels released after this date may or may not have fixed these problems. o Linux kernels 2.2.16 through 2.2.20 and 2.4.0 through 2.4.18 are known to have a problem in the SCSI error recovery process. In some cases, a successful device reset may be ignored and the SCSI layer will continue on to the later stages of the error recovery process. The problem occurs when multiple SCSI commands for a particular device are queued in the low-level SCSI driver when a device reset occurs. Even if the low-level driver correctly reports that all the commands for the device have been completed by the reset, Linux will assume only one command has been completed and continue the error recovery process. (If only one command has timed out or failed, Linux will correctly terminate the error recovery process following the device reset.) This action is undesirable because the later stages of error recovery may send other types of resets, which can affect other SCSI initiators using the same target or other targets on the same bus. It is also undesirable because there are more serious bugs in the later stages of the Linux SCSI error recovery process. The Linux iSCSI driver now attempts to avoid this problem by replacing the usual error recovery handler for SCSI commands that timeout or fail. o Linux kernels 2.2.16 through 2.2.20 and 2.4.0 through 2.4.2 may take SCSI devices offline after Linux issues a reset as part of the error recovery process. Taking a device offline causes all I/O to the device to fail until the HBA driver is reloaded. After the error recovery process does a reset, it sends a SCSI Test Unit Ready command to check if the the SCSI target is operational again. If this command returns SCSI sense data, instead of correctly retrying the command, Linux will treat it as a fatal error, and immediately take the SCSI device offline. The Test Unit Ready will almost always be returned with sense data because most targets return a deferred error in the sense data of the first command received after a reset. This is a way of telling the initiator that a reset has occurred. Therefore, the affected Linux kernel versions almost always take a SCSI device offline after a reset occurs. This bug is fixed in Linux kernels 2.4.3 and later. The Linux iSCSI driver now attempts to avoid this problem by replacing the usual error recovery handler for SCSI commands that timeout or fail. o Linux kernels 2.2.16 through 2.2.21 and 2.4.0 through 2.4.18 appear to have problems when SCSI commands are completed with a check condition containing sense data. This can result in applications receiving I/O errors, short reads or short writes. After receiving a SCSI command with sense data, the Linux SCSI midlayer checks if some sectors of an I/O request have been transferred and informs the application layer if an I/O request has partially completed. The SCSI midlayer then retries the request for any sectors that have not yet been transferred. Because of the partial I/O request completion, applications may receive short reads or writes. All UNIX applications should handle these conditions, but there may be some applications which do not. There are also some cases where the application receives an I/O error rather than a short read or write. The exact cause of the I/O errors is still being investigated, but it appears to be a bug in the Linux kernel's SCSI layer or block device layer. o Linux kernels 2.2.16 through 2.2.21 and 2.4.0 through 2.4.18 may crash on a NULL pointer if a SCSI device is taken offline while one of the Linux kernel's I/O daemons (e.g. kpiod, kflushd, etc.) is trying to do I/O to the SCSI device. The exact cause of this problem is still being investigated. Note that some of the other bugs in the Linux kernel's error recovery handling may result in a SCSI device being taken offline, thus triggering this bug and resulting in a Linux kernel crash. o Linux kernels 2.2.16 through 2.2.21 running on uniprocessors may hang if a SCSI disk device node is opened while the Linux SCSI device structure for that node is still being initialized. This occurs because the sd driver which controls SCSI disks will loop forever waiting for a device busy flag to be cleared at a certain point in the open routine for the disk device. Since this particular loop will never yield control of the processor, the process initializing the SCSI disk device is not allowed to run, and the initialization process can never clear the device busy flag which the sd driver is constantly checking. A similar problem exists in the SCSI generic driver in some 2.4 kernel versions. The sg driver may crash on a bad pointer if a /dev/sg* device is opened while it is being initialized. ---------- SETTING UP ---------- 1. Update /etc/iscsi.conf to include the IP addresses for your iSCSI targets. A sample configuration file might include entries like this: DiscoveryAddress=192.168.10.94 The iscsi.conf man page has a more detailed description of the configuration file format. To read the man page, type: man iscsi.conf 2. Manually start iSCSI services to test your configuration. Run: service iscsi start If there are problems loading the iSCSI kernel module, diagnostic information will be placed in /var/log/iscsi.log. The iSCSI initialization will report information on each detected device to the console or in dmesg(8) output. For example: Vendor: SEAGATE Model: ST39103FC Rev: 0002 Type: Direct-Access ANSI SCSI revision: 02 Detected scsi disk sda at scsi0, channel 0, id 0, lun 0 SCSI device sda: hdwr sector= 512 bytes. Sectors= 17783240 [8683 MB] [8.7 GB] sda: sda1 scsi singledevice 0 0 0 1 Normal disk commands like fdisk, mkfs, and fsck will work on the iSCSI devices like a local drive. /proc/scsi/iscsi will contain a file (the controller number) that contains information about the iSCSI devices. To manually stop the iSCSI driver enter: service iscsi stop 3. List your iSCSI partitions in /etc/fstab with the _netdev option. ------------ DEVICE NAMES ------------ Because Linux assigns SCSI device nodes dynamically whenever a SCSI logical unit is detected, the mapping from device nodes (e.g /dev/sda, /dev/sdb) to iSCSI targets and logical units may vary. Variations in process scheduling and network delay may result in iSCSI targets being mapped to different SCSI device nodes every time the driver is started. Because of this variability, configuring applications or operating system utilities to use the standard SCSI device nodes to access iSCSI devices may result in SCSI commands being sent to the wrong target or logical unit. To provide a more reliable namespace, the iSCSI driver will scan the system to determine the mapping from SCSI device nodes to iSCSI targets, and then create a tree of directories and symbolic links under /dev/iscsi to make it easier to use a particular iSCSI target's logical units. Under /dev/iscsi, there will be a directory tree containing subdirectories for each iSCSI bus number, each target id number on the bus, and each logical unit number for each target. For example, the whole disk device for bus 0, target id 0, LUN 0 would be /dev/iscsi/bus0/target0/lun0/disk In each logical unit directory there will be a symbolic link for each SCSI device node that may be connected to that particular logical unit. These symbolic links are modeled after the Linux devfs naming convention. The symbolic link 'disk' will map to the whole-disk SCSI device node (e.g. /dev/sda, /dev/sdb, etc). The symbolic links 'part1' through 'part15' will map to each partition of that SCSI disk (e.g. /dev/sda1, dev/sda15, etc). Note that these links will exists regardless of the number of disk partitions. Opening the partition devices will result in an error if the partition does not actually exist on the disk. The symbolic link 'mt' will map to the auto-rewind SCSI tape device node for this LUN (e.g. /dev/st0), if any. Additional links for 'mtl', 'mtm', and 'mta' will map to the other auto-rewind devices (e.g. /dev/st0l, /dev/st0m, /dev/st0a), regardless of whether these device nodes actually exist or could be opened. The symbolic link 'mtn' will map to the no-rewind SCSI tape device node for this LUN (e.g. /dev/nst0), if any. Additional links for 'mtln', 'mtmn', and 'mtan' will map to the other no-rewind devices (e.g. /dev/nst0l, /dev/nst0m, /dev/nst0a), regardless of whether those device nodes actually exist or could be opened. The symbolic link 'cd' will map to the SCSI cdrom device node for this LUN (e.g. /dev/scd0), if any. The symbolic link 'generic' will map to the SCSI generic device node for this LUN (e.g. /dev/sg0), if any. Because the symlink creation process must open all of the SCSI device nodes in /dev in order to determine which nodes map to iSCSI devices, you may see many modprobe messages logged to syslog indicating that modprobe could not find a driver for a particular combination of major and minor numbers. This is harmless, and can be ignored. The messages occur when Linux is unable to find a driver to associate with a SCSI device node that the iSCSI daemon is opening as part of it's symlink creation process. To prevent these messages, the SCSI device nodes with no associated high-level SCSI driver can be removed. --------------- TARGET BINDINGS --------------- The iSCSI driver automatically maintains a bindings file /var/iscsi/bindings. This file contains persistent bindings to ensure that the same iSCSI bus and target id number are used for every iSCSI session to a particular iSCSI TargetName, no matter how many times the driver is restarted. This feature ensures that the SCSI numbers in the device symlinks described above will always map to the same iSCSI target. Note that because of the way Linux dynamically allocates SCSI device nodes as SCSI devices are found, the driver does not and can not ensure that any particular SCSI device node (e.g. /dev/sda) will always map to the same iSCSI TargetName. The symlinks described in the section on Device Names are intended to provide a persistent device mapping for use by applications and fstab files, and should be used instead of direct references to particular SCSI device nodes. If the bindings file grows too large, lines for targets that no longer exist may be manually removed by editing the file. Manual editing should not normally be needed, since the driver can maintain up to 65535 different bindings. -------------------- MOUNTING FILESYSTEMS -------------------- Because the Linux boot process normally mounts filesystems listed in /etc/fstab before the network is configured, you need to add the _netdev option to each filesystem on an iSCSI device. Because of the variability of the mapping between SCSI device nodes and iSCSI targets, instead of directly mounting SCSI device nodes, it is recommended to either mount the /dev/iscsi tree symlinks, mount filesystem UUIDs or labels (see man pages for mke2fs, mount, and fstab), or use logical volume management (see Linux LVM) to avoid mounting the wrong device due to device name changes resulting from iSCSI target configuration changes or network delays. ---------------------- UNMOUNTING FILESYSTEMS ---------------------- It is very important to unmount all filesystems on iSCSI devices before the iSCSI driver stops. If the iSCSI driver stops while iSCSI devices are mounted, buffered writes may not be committed to disk and filesystem corruption may occur. For this reason, the system automatically umounts all file systems with the _netdev option before the iSCSI driver stops and before the network is stopped. Since Linux will not unmount filesystems that are being used by a running process, before iSCSI devices can be unmounted, any processes using those devices must be stopped (see fuser(1)). To avoid filesystem corruption, the netfs shutdown script will automatically kill all processes using _netdev devices in /etc/fstab, first by sending them SIGTERM, and then by sending any remaining processes SIGKILL. It will then unmount all iSCSI filesystems and kill the iSCSI daemon, terminating all connections to iSCSI devices. -------------------------------- DYNAMIC TARGET AND LUN DISCOVERY -------------------------------- The driver can be told to rediscover iSCSI devices and probe for LUNs by running: service iscsi reload This will cause the iSCSI daemon to restart all iSCSI discovery processes and probe LUNs on all iSCSI targets. In addition, when using iSCSI targets that support long-lived iSCSI discovery sessions, such as the Cisco 5400 Series, the driver will keep a discovery session open waiting for change notifications from the target. When a notification is received, the driver will rediscover targets, add any new targets, and reprobe LUNs on all targets that were discovered. ----------------- /proc/scsi/iscsi ----------------- The directory /proc/scsi/iscsi will contain a special file that can be used to get status from your iSCSI HBA. The name of the file will be the iSCSI HBA's host number, which is assigned to the driver by Linux. When the file is read, it will show the driver's version number, followed by a list all iSCSI targets and LUNs the driver has found and can use. Each line will show the Linux host number, channel number, target id number, and logical unit number, as well as the IP address, TCP port, and iSCSI TargetName. If no LUNs have yet been found for a target, the LUN number field will contain a question mark. ------------ UNINSTALLING ------------ rpm -e iscsi -------------------------- KERNEL AND DRIVER UPGRADES -------------------------- Use up2date to get newer kernels and iscsi packages when they are available.