iscsi

Proxy Iscsi Initiator

* pcit


* One time Config
* Config for each initiator
* CSM mapping
* show iscsi commands
* isid
* ethereal trace
* FC trace of two iscsi sessions with proxy initiator and Xiotech target
* FC trace for two iscsi sessions with different MTU
* proxy iscsi initiator in multiple vsans

Benefits:

- simple zoning ( no need to configure all the hosts wwns)
- simple lun mapping at the storage.
- it is in certain way like SN5428 , where we configure storage lun mapping for the internal
HBAs and there is no individual wwns for each initiator.
MDS Config:

One time Config:
MDS9509-B1-sup1(config)# iscsi interface-vsan-member-enable

by default proxy wwn goes to vsan 1 to change that, we need to do these steps.
MDS9509-B1-sup1(config)# vsan 40 interface iscsi 4/1
( where my targets are).

interface iscsi 4/1
switchport proxy-initiator ( if you want you can configure wwns manual too!)

show interface iscsi 4/1
Proxy Initiator Mode : enabled
nWWN is 20:15:00:05:30:00:24:60 (system-assigned)
pWWN is 20:16:00:05:30:00:24:60 (system-assigned)

Add this pwwn to Zoning/ and configure storage Lun mapping

for CSM:
MDS9509-B1-sup1(svc)# show cluster tacCluster host proxy-iscsi
Host proxy-iscsi:
Number of port is 2
Port WWN is 20:16:00:05:30:00:24:60
LUN 0 : vdisk piscsi1
LUN 1 : vdisk piscsi2
LUN 2 : vdisk piscsi3
LUN 3 : vdisk piscsi4
LUN 4 : vdisk piscsi5
LUN 5 : vdisk piscsi6
LUN 6 : vdisk piscsi7
LUN 7 : vdisk piscsi8
-----------

Configuration for each host (this config on MDS host - configuring two virtual targets
(multipathing) for ip 172.69.122.104)

You can not do lunzoning/zoning based on ip address with proxy initiator.(even after setting
switch initiator id ip-address on iscsi port.( verify CSCed82704 ))
All the subsequent initiators will have access to same luns/storages as the first initiator.
In the MDS 1.3.3 config guide, under the sections "Configuring iSCSI proxy initiators"
(Configuring iSCSI/Configuring IP Storage), it is clearly mentioned that when in proxy
initiator mode, you cannot use iSCSI attributes in the FC access control mechanisms.
You have to use iSCSI based access control to accomplish the same.


(still needs to verify above).

ip route 171.69.122.104 255.255.255.255 interface gig 4/1

iscsi virtual-target name csm
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit
iscsi virtual-target name csm2
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit <<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator

Linux host config on MDS:
ip route 172.69.122.104 255.255.255.255 interface GigabitEthernet4/1
iscsi virtual-target name csm-linux
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator

iscsi virtual-target name csm-linux2
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
advertise interface GigabitEthernet4/1
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator



----------

on the iscsi PC host , I could see two luns for each targets.

VSAN 40:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x790000 N 20:04:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790001 N 20:09:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790005 N 20:16:00:05:30:00:24:60 (Cisco) scsi-fcp:init isc..w (only one initiator)
0x790100 N 21:00:00:e0:8b:0b:fc:0d (QLogic) scsi-fcp:init
0x790200 N 21:00:00:e0:8b:08:f6:18 (QLogic) ipfc scsi-fcp:init
0x790300 N 21:01:00:e0:8b:28:f6:18 (QLogic) ipfc scsi-fcp:init

---
MDS9509-B1-sup1# show iscsi session
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
Session #1
Target csm
VSAN 40, ISID 400001370018, Status active, no reservation

Session #2
Target csm2
VSAN 40, ISID 400001370019, Status active, no reservation

Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1
Target csm-linux
VSAN 40, ISID 801234567800, Status active, no reservation

Session #2
Target csm-linux2
VSAN 40, ISID 801234567801, Status active, no reservation

MDS9509-B1-sup1# show iscsi initiator
iSCSI Node name is iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
iSCSI alias name:
Node WWN is 20:02:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005

iSCSI Node name is dhcp-173-228
Initiator ip addr (s): 171.69.104.104
iSCSI alias name:
Node WWN is 20:00:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005


On the storage , let us see which session is logged (proxy or the actual iscsi host)- it is proxy.
MDS9509-B1-sup1# show svc session svc 2/1 (we don't see 20:02 ....60 wwn of actual iscsi host
being logged on to the storage)
svc2/1:
Target N-port WWN is 20:04:00:05:30:00:24:60, vsan is 40, FCID is 0x790000
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:01:00:05:30:00:24:60, vsan is 30, FCID is 0x780000
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:05:00:05:30:00:24:60, vsan is 50, FCID is 0xd40000
pWWN 20:14:00:05:30:00:24:60, nWWN 20:0f:00:05:30:00:24:60, FCID 0xd40001

MDS9509-B1-sup1# show svc session svc 2/2
svc2/2:
Target N-port WWN is 20:09:00:05:30:00:24:60, vsan is 40, FCID is 0x790001
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:08:00:05:30:00:24:60, vsan is 30, FCID is 0x780002
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:14:00:05:30:00:24:60, vsan is 50, FCID is 0xd40001
pWWN 20:05:00:05:30:00:24:60, nWWN 20:0e:00:05:30:00:24:60, FCID 0xd40000

------
debug ips iscsi flow
(with only pc iscsi connection)
Debugs:
MDS9509-B1-sup1# Dec 15 12:18:51 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:18:51 ips: Created initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Initiator(8) got nwwn 2002000530002460
Dec 15 12:18:51 ips: Initiator(8) got vsan list
Dec 15 12:18:51 ips: no:1 vsan_id 1
Dec 15 12:18:51 ips: Created an fc_port(7) pgt 384 iscsi-if-index 0x0b180000 intf 0x02180000 ip-addr: 172.16.34.10 for initiator(8)
Dec 15 12:18:51 ips: Created session(39) target name isid 400001370016 for initiator(8)
Dec 15 12:18:51 ips: fc_port(7) has a pwwn 0, mode: 1
Dec 15 12:18:51 ips: Put iscsi4/1 in vsan 40 status: 0
Dec 15 12:18:51 ips: Fc_port(7) pwwn 2016000530002460 member of 1 vsans registered 0
Dec 15 12:18:51 ips: fc_port(7) sent 1 flogi requests
Dec 15 12:18:51 ips: Flogi response: fc_port(7) fcid 00790005 in vsan 40
Dec 15 12:18:51 ips: fc_port(7) pwwn 2016000530002460 sent 1 NS reg requests
Dec 15 12:18:51 ips: NS reg resp: fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 fcid 00790005 vsan 40
Dec 15 12:18:51 ips: Discovery session.. no need to check target
Dec 15 12:18:51 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370016]
Dec 15 12:18:51 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:18:51 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:18:51 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:18:51 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:18:51 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:18:51 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:18:51 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:18:51 ips: Removing session(39) tgt-name: isid: 400001370016 failure code: 1
Dec 15 12:19:12 ips: Node 2016000530002460, vsan 40 is not discovered as init or target
Dec 15 12:19:12 ips: Initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com processing tgt_online 2016000530002460 vsan 40

Refresh on MS initiator

Dec 15 12:19:54 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:19:54 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:19:54 ips: Created session(40) target name isid 400001370017 for initiator(8)
Dec 15 12:19:54 ips: Discovery session.. no need to check target
Dec 15 12:19:54 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370017]
Dec 15 12:19:54 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:19:54 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:19:54 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:19:54 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:19:54 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:19:54 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:19:54 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:19:54 ips: Removing session(40) tgt-name: isid: 400001370017 failure code:
1
Logon CSM:
Dec 15 12:20:36 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm
Dec 15 12:20:36 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:20:36 ips: Created session(41) target name csm isid 400001370018 for initiator(8)
Dec 15 12:20:36 ips: Target csm a virtual target checking access
Dec 15 12:20:36 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:20:36 ips: fc_port(7) Querying NS for target pwwn:[2004000530002460] sec pwwn:[0] wait 1
Dec 15 12:20:36 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790000
Dec 15 12:20:36 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm] isid:[400001370018]

Logon CSM2
Dec 15 12:21:18 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm2
Dec 15 12:21:18 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:21:18 ips: Created session(42) target name csm2 isid 400001370019 for initiator(8)
Dec 15 12:21:18 ips: Target csm2 a virtual target checking access
Dec 15 12:21:18 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:21:18 ips: fc_port(7) Querying NS for target pwwn:[2009000530002460] sec pwwn:[0] wait 1
Dec 15 12:21:18 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790001
Dec 15 12:21:18 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm2] isid:[400001370019]
--------

Etherreal Trace from Linux:
Linux host 171.69.104.104 to proxy iscsi initiator using unh_iscsi with two virtual-targets to two CSM nodes
(one each) and these virtual targets have been mapped to one iscsi lun 0 (fc-lun 3).

configured unh iscsi conf : initiator dhcp-173-228 ,
target =csm-linux and csm-linux2 and the ipaddress
of target 172.16.34.10.

cat /proc/scsi/scsi showed two disks which are same.
[root@dhcp-173-228 root]# cat /proc/scsi/scsi
(only iscsi devices displayed)
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04


etherreal trace

MDS9509-B1-sup1# show iscsi session detail
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.48
Session #1 (index 4)
Target xiotech
VSAN 40, ISID 400001370004, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 20104, MaxCmdSN 20119, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 2
Stats:
PDU: Command: 933, Response: 933
Bytes: TX: 20158288, RX: 19995648
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.122.48
CID 1, State: Full-Feature
StatSN 937, ExpStatSN 0
MaxRecvDSLength 65536, our_MaxRecvDSLength 1024
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: Yes

Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1 (index 2)
Target xiotech-linux
VSAN 40, ISID 801234567800, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 24429, MaxCmdSN 24443, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 1
Stats:
PDU: Command: 2207, Response: 2206
Bytes: TX: 128568, RX: 202783744
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.104.104
CID 0, State: Full-Feature
StatSN 2209, ExpStatSN 0
MaxRecvDSLength 1392, our_MaxRecvDSLength 1392
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: No

Just for recap: here are the virtual targets defined.
target: xiotech
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 2
iSCSI LUN: 0x0000, FC LUN: 0x0000
iSCSI LUN: 0x0001, FC LUN: 0x0001
No. of initiators permitted: 1
initiator 171.69.122.48/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled

target: xiotech-linux
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 1
iSCSI LUN: 0x0000, FC LUN: 0x0002
No. of initiators permitted: 1
initiator 171.69.104.104/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled

MDS9509-B1-sup1# show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
Connection Stats
0 active openings, 109 accepts
0 failed attempts, 0 reset received, 109 established
Segment stats
4564268 received, 1961372 sent, 1133 retransmitted
43 bad segments received, 0 reset sent

TCP Active Connections
Local Address Remote Address State Send-Q Recv-Q
172.16.34.10:3260 171.69.122.48:1593 ESTABLISH 0 0
172.16.34.10:3260 171.69.104.104:32779 ESTABLISH 0 0
0.0.0.0:3260 0.0.0.0:0 LISTEN 0 0

Traces are in this directory

Here is the trace snapshots of proxy_linux1_win2_logoff.

a. no iscsi sessions were logged on before the taking the trace.
b. started /etc/init.d/unh_iscsi start ( Linux has fc-lun 2 mapped to iscsi-lun 0)
- you see prli from the proxy initiator
- lun inquiry proxied for Linux.
prli_linux_inquiry

3. mount /dev/sdb1 /xiotech1 and deleted some files in /xiotech1
4. Using microsoft initiator, I connect to virtual target xiotech , u will see microsoft's inquiry, no new plogi or prli session
initiated. (probably if PDU of this session is lower, then we might reinitiate , so that PMTU Is reneogiated.

windows_inquiry

and finally I removed windows session and then the linux session, so you will prlo as last iscsi session is cleared up.

prlo

ip payload size is 1460. (+ 20 byte TCP Options from ethereal trace )
MSS in PC is 1460 bytes
iscsi payload is 1440 ( iscsi header is 48 bytes)
FC data size is 1392

Example II:

win iscsi initiator logs in first with default MTU size (MSS 1460)
linux iscsi initiator logs in second with mtu size of 800.
We expect LOGO And PRLO to happen because proxy initiator relogs to target with lower Receive data field Size.

here is the picture with PLOGI when the win2k initiator comes in. Note the Class 3 receive data field size.
win2k_prli

after a little bit, here comes the linux session with lower MTU ( 800)
(setting up Linux mtu)
at> ifconfig eth0 mtu 800
at> ifconfig eth0 down
at> ifconfig eth0 up
at> route add default gw 171.69.104.1

second


iSCSI initiator 20.1.2.12 will be in VSAN 40, 41, 50 and 51. Not in VSAN 30. All initiators without "iscsi initiator" command or without vsan command will be in VSAN 30.

iscsi initiator ip-address 20.1.2.12
vsan 40
vsan 41
vsan 50
vsan 51

interface iscsi3/3
switchport initiator id ip-address
switchport proxy-initiator nWWN 11:11:11:11:11:11:11:00 pWWN 11:11:11:11:11:11:11:11

vsan database
vsan 30 interface iscsi 3/3