To check the thro'put via fcip pipe.
MDS9509-A1(config)# san-ext-tuner enable
MDS9509-A1# san-ext-tuner
MDS9509-A1(san-ext)#
MDS9509-A1(san-ext)# nwwn 10:00:00:0d:3f:2c:11:22
MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1# show flogi database
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/5 100 0x660300 21:00:00:e0:8b:08:f6:18 20:00:00:e0:8b:08:f6:18
iscsi4/1 100 0x660003 10:00:00:0d:3f:2c:11:21 10:00:00:0d:3f:2c:11:22
Total number of flogi = 2.
-----
without iscsi , I got this error
MDS9509-B1(san-ext)# nport pwwn 21:08:00:d0:b2:00:82:c0 vsan 100 interface gigabitethernet 2/1
Error: flogi for the virtual nport failed(0x40be0009)
i------
SAN EXT is not persistent between the reboots.
iscsi enable
interface iscsi 2/1 enable
MDS9509-B1(config)# san-ext-tuner enable
MDS9509-B1(config)# exit
MDS9509-B1# san-ext-tuner MDS9509-B1(config)# interface iscsi 2/1
MDS9509-B1(config-if)# no shut
MDS9509-B1(config-if)# exit
MDS9509-B1(san-ext)# nwwn 20:00:00:d0:b2:00:82:d0
MDS9509-B1(san-ext)# nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 interface gigabitethernet 2/1
MDS9509-B1# show flogi database
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/5 100 0x4e0001 21:00:00:d0:b2:00:82:c0 20:00:00:d0:b2:00:82:c0
[xio-hab0]
iscsi2/1 100 0x4e0004 20:08:00:d0:b2:00:82:d0 20:00:00:d0:b2:00:82:d0
Total number of flogi = 2.
-----
testing
Zone:
MDS9509-B1# show zoneset active vsan 100
zoneset name ZoneSet1 vsan 100
zone name Zone1 vsan 100
attribute qos priority high
pwwn 21:00:00:e0:8b:0b:fc:0d [dell6450]
* fcid 0x4e0001 [pwwn 21:00:00:d0:b2:00:82:c0] [xio-hab0]
* fcid 0x660300 [pwwn 21:00:00:e0:8b:08:f6:18]
zone name SanExtVirtualHosts vsan 100
* fcid 0x660003 [pwwn 10:00:00:0d:3f:2c:11:21]
* fcid 0x4e0004 [pwwn 20:08:00:d0:b2:00:82:d0]
MDS9509-B1# show fcns database
VSAN 100:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x4e0001 N 21:00:00:d0:b2:00:82:c0 scsi-fcp:both
[xio-hab0]
0x4e0004 N 20:08:00:d0:b2:00:82:d0 scsi-fcp
0x660003 N 10:00:00:0d:3f:2c:11:21 scsi-fcp
0x660300 N 21:00:00:e0:8b:08:f6:18 (Qlogic) ipfc scsi-fcp:init
Total number of entries = 4
MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1(san-ext-nport)# write command-id 100 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1# show san-ext-tuner nports
----------------------------------------------------------------------------
Interface NODE NAME PORT NAME VSAN
----------------------------------------------------------------------------
GigabitEthernet4/1 10:00:00:0d:3f:2c:11:22 10:00:00:0d:3f:2c:11:21 100
MDS9509-B1# show san-ext-tuner nports
----------------------------------------------------------------------------
Interface NODE NAME PORT NAME VSAN
----------------------------------------------------------------------------
GigabitEthernet2/1 20:00:00:d0:b2:00:82:d0 20:08:00:d0:b2:00:82:d0 100
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 230267
Reads : 66%
Writes : 33%
Egress throughput : 96.30 MBs/sec (Max - 205.18 MBs/sec)
Ingress throughput : 176.91 MBs/sec (Max - 177.43 MBs/sec)
Average response time : Read - 651 us, Write - 1297 us
Minimum response time : Read - 480 us, Write - 335 us
Maximum response time : Read - 990 us, Write - 1764 us
Errors : 0
MDS9509-B1# show san-ext-tuner interface gigabitethernet 2/1 nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 counters
Statistics for nport
Node name 20:00:00:d0:b2:00:82:d0 Port name 20:08:00:d0:b2:00:82:d0
I/Os per sec : 229946
Reads : 66%
Writes : 33%
Egress throughput : 176.65 MBs/sec (Max - 177.78 MBs/sec)
Ingress throughput : 96.17 MBs/sec (Max - 205.18 MBs/sec)
Average response time : Read - 1 us, Write - 656 us
Minimum response time : Read - 1 us, Write - 113 us
Maximum response time : Read - 7 us, Write - 1012 us
Errors : 0
MDS9509-A1(san-ext-nport)# write command-id 103 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 104 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 206454
Reads : 39%
Writes : 60%
Egress throughput : 142.51 MBs/sec (Max - 205.18 MBs/sec)
Ingress throughput : 105.52 MBs/sec (Max - 177.43 MBs/sec)
Average response time : Read - 1212 us, Write - 2417 us
Minimum response time : Read - 480 us, Write - 335 us
Maximum response time : Read - 1395 us, Write - 2561 us
Errors : 0
----
removed write/tape and ip compression .....shut/no shut the gige, fcip cameup
but sanext config went away , so recreated it again.
MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1(san-ext-nport)# write command-id 103 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 104 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 100 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# read command-id 101 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# read command-id 102 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# exit
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 205242
Reads : 57%
Writes : 42%
Egress throughput : 105.85 MBs/sec (Max - 204.80 MBs/sec)
Ingress throughput : 138.77 MBs/sec (Max - 139.05 MBs/sec)
Average response time : Read - 1708 us, Write - 3399 us
Minimum response time : Read - 893 us, Write - 354 us
Maximum response time : Read - 2199 us, Write - 4178 us
Errors : 0
----
Let me enable the write/tape and ip compression
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 203654
Reads : 57%
Writes : 42%
Egress throughput : 105.05 MBs/sec (Max - 105.30 MBs/sec)
Ingress throughput : 137.69 MBs/sec (Max - 213.75 MBs/sec)
Average response time : Read - 1721 us, Write - 3425 us
Minimum response time : Read - 259 us, Write - 2017 us
Maximum response time : Read - 2304 us, Write - 4240 us
Errors : 0
MDS9509-B1# show san-ext-tuner interface gigabitethernet 2/1 nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 counters
Statistics for nport
Node name 20:00:00:d0:b2:00:82:d0 Port name 20:08:00:d0:b2:00:82:d0
I/Os per sec : 203896
Reads : 57%
Writes : 42%
Egress throughput : 137.87 MBs/sec (Max - 213.85 MBs/sec)
Ingress throughput : 105.15 MBs/sec (Max - 105.42 MBs/sec)
Average response time : Read - 1 us, Write - 1720 us
Minimum response time : Read - 1 us, Write - 993 us
Maximum response time : Read - 14 us, Write - 2475 us
Errors : 0
-------------------------------
no compression
MDS9509-B1# config t
Enter configuration commands, one per line. End with CNTL/Z.
MDS9509-B1(config)# interface fcip 2
MDS9509-B1(config-if)# no write-accelerator
MDS9509-B1(config-if)# no write-accelerator tape-accelerator
MDS9509-B1(config-if)# no ip-compression
MDS9509-A1(config)# interface fcip 2
MDS9509-A1(config-if)# no write-accelerator
MDS9509-A1(config-if)# no write-accelerator tape-accelerator
MDS9509-A1(config-if)# no ip-compression
MDS9509-B1(config)# interface gigabitethernet 2/1
MDS9509-B1(config-if)# shut
MDS9509-B1(config-if)# no shut
MDS9509-A1(config-if)# interface gigabitethernet 4/1
MDS9509-A1(config-if)# shut
MDS9509-A1(config-if)# no shutdown
MDS9509-B1# show interface fcip2
fcip2 is trunking
Hardware is GigabitEthernet
Port WWN is 20:42:00:05:30:00:24:5e
Peer port WWN is 20:c2:00:05:30:00:24:1e
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,100)
Trunk vsans (operational) (1,100)
Trunk vsans (up) (1,100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet2/1)
Peer Information
Peer Internet address is 10.1.1.1 and port is 3225
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is 256 KBytes
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
Time Stamp is disabled
MDS9509-A1# show interface fcip2
fcip2 is trunking
Hardware is GigabitEthernet
Port WWN is 20:c2:00:05:30:00:24:1e
Peer port WWN is 20:42:00:05:30:00:24:5e
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,100)
Trunk vsans (operational) (1,100)
Trunk vsans (up) (1,100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet4/1)
Peer Information
Peer Internet address is 10.1.1.2 and port is 3225
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is 256 KBytes
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
Time Stamp is disabled
------
MDS9509-B1# san-ext-tuner
MDS9509-B1(san-ext)# nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 interface gigabitethernet 2/1
MDS9509-B1(san-ext-nport)# exit
MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1(san-ext-nport)# read command-id 102 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# read command-id 101 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 100 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 104 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 103 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# exit
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 210123
Reads : 57%
Writes : 42%
Egress throughput : 108.37 MBs/sec (Max - 108.46 MBs/sec)
Ingress throughput : 142.07 MBs/sec (Max - 213.73 MBs/sec)
Average response time : Read - 1668 us, Write - 3320 us
Minimum response time : Read - 197 us, Write - 1725 us
Maximum response time : Read - 2214 us, Write - 4109 us
Errors : 0
MDS9509-B1# show san-ext-tuner interface gigabitethernet 2/1 nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 counters
Statistics for nport
Node name 20:00:00:d0:b2:00:82:d0 Port name 20:08:00:d0:b2:00:82:d0
I/Os per sec : 210175
Reads : 57%
Writes : 42%
Egress throughput : 142.08 MBs/sec (Max - 213.73 MBs/sec)
Ingress throughput : 108.42 MBs/sec (Max - 108.63 MBs/sec)
Average response time : Read - 1 us, Write - 1668 us
Minimum response time : Read - 1 us, Write - 815 us
Maximum response time : Read - 13 us, Write - 2205 us
Errors : 0
Saturday, April 12, 2008
TACACS Config:
Config on MDS is very simple
tacacs+ enable
tacacs-server timeout 4
tacacs-server host 171.69.89.198 key 7 fewhg
aaa group server tacacs+ secteam
server 171.69.89.198
aaa authentication login default group secteam
MDS9216i# show user-acc sanremote
user:sanremote
expires on Wed Oct 12 23:59:59 2005
roles:network-admin
account created through REMOTE authentication
Local login not possible
Iscsi Via CHAP authentication:
tacacs+ enable
tacacs-server timeout 4
tacacs-server host 171.69.89.198 key 7 fewhg
aaa group server tacacs+ secteam
server 171.69.89.198
aaa authentication login default group secteam
MDS9216i# show user-acc sanremote
user:sanremote
expires on Wed Oct 12 23:59:59 2005
roles:network-admin
account created through REMOTE authentication
Local login not possible
Iscsi Via CHAP authentication:
VRRP/ipfc and mgmt on MDS
When you have to access remote MDS via fcip link or fc link, ie., access
FM via that IPFc, here is a design for you!
SAmple Config:
DS9120-A1# show vrrp
Interface VR Status
-------------------------------------------------------
mgmt0 1 backup
MDS9120-A1# show run int mgmt 0
interface mgmt0
ip address 172.16.33.82 255.255.255.128
switchport speed 100
vrrp 1
address 172.16.33.82
address 172.16.33.120 secondary
no shutdown
MDS9216i# show run int mgmt 0
version 2.1(1)
interface mgmt0
ip address 172.16.33.86 255.255.255.128
vrrp 1
address 172.16.33.86
address 172.16.33.120 secondary
no shutdown
MDS9216i# show vrrp
Interface VR Status
-------------------------------------------------------
mgmt0 1 master
the problem is uou have to vrrp address same as that mgmt 0.. It has
been confusing for me.. how does the remote switch know the ip address
of other switche's vrrp enabled address ?
Problem Statement:
a. Customer wants to use ISL link to FM/DM related management in a
Two Switch
Scenario, when one mgmt interface is messed up.
b. also the wants minimum traffic
to ISL and confirmed that no host or storage will talk to the storage or
host on the
remote switch. (localization).
Design:
a. both 9216 will have console access for better recovery.
-configure vrrp on the mgmt interface. , this is to resolve
problem with two static entries for same network. As
static route entries load balances , if there are two paths,
and it does not know when when one path fails.
So we configure only one static entry with vrrp interface 172.16.33.120and
vrrp will take care of routing thro' active path).
( for eg, configuration of 9506 with mgmt of 172.16.33.79
MDS9506-B1-sup1(config)# interface mgmt 0
MDS9506-B1-sup1(config-if)# vrrp 1
MDS9506-B1-sup1(config-if-vrrp)# address 172.16.33.79
MDS9506-B1-sup1(config-if-vrrp)# address 172.16.33.120 secondary
MDS9506-B1-sup1(config-if-vrrp)# no shutdown
MDS9506-B1-sup1# show vrrp
Interface VR Status
-------------------------------------------------------
mgmt0 1 master
and do the same thing for other switch with mgmt ip address 172.16.33.77
MDS9509-B1-sup1(config)# interface mgmt 0
MDS9509-B1-sup1(config-if)# vrrp 1
MDS9509-B1-sup1(config-if-vrrp)# address 172.16.33.77
MDS9509-B1-sup1(config-if-vrrp)# address 172.16.33.120 secondary
MDS9509-B1-sup1(config-if-vrrp)# no shutdown
MDS9509-B1-sup1# show vrrp
Interface VR Status
-------------------------------------------------------
mgmt0 1 backup
So even if mgmt 0 172.16.33.79 goes down, the vrrp 172.16.33.120 will route
thro' 172.16.33.77, so we need only static route path on host or the
router. )
- create ISL (TE port ) between 9216 in vsan 1
- configure ip for each 9216's vsan 1. (config t ; interface vsan 1)
- create vsan 10 with interfaces on one switch and empty vsan 20 on
the same switch.
- create empty vsan 10 on second switch and vsan 20 with interfaces on the
second switch.
- create second default routers with a different metric using vsan 1's ip
addresses.
- configure zoning from a single switch/ you can do either full zoneset to
propagate aliases
as well as non-active zonesets from that switch.
- you can do copy merge or leave it as it is on the second switch.
- if first 9216's mgmt fails, you can login to 9216 cli and connect to other
switche's vsan ip
and correct the problem. or thro' console.
- or configure static route to vsan's network to go via active mgmt (one
of the mgmt should be
up) on the mgmt wkstation and run fabric manager .
Also note that , even without VSAN 1 interface being configured for network,
you can
do zone editing etc thro' normal ISL because it uses FC-CT from one
switch.
Let me know if this suffice your requirements.
FCIP commands
show ips stats buffer interface gig 1/1 --- look free clusters
show ips stats tcp interface gig 1/1 det --- look for SACK/retrans
show ips stats dma inter gig 1/1 --- look for timestamp errors
show int gig 1/1 ----- mtu
show int fcip 1 ----- retransmits and other config cwm/retrans times
attach mod 1
show port internal port-control
show port internal link-events
exit
show tech internal link-events
show ips status module 1
show ips stats hw-comp int gig 1/1
show ips stats ip int gig 1/1 ----> reassembly/packets/dropped
show ips stats mac int gig 1/1
-----CRC erros -hw related stuff
extended ping with DF (y)
-
show ips internal eth-trace-logs gigabitethernet 1/1 (on NWMDS02 and
APMDS01)
- show ips internal eth-trace-logs gigabitethernet 1/2 (on NWMDS01 and
APMDS01)
-----------------------------
Internal Commands: don't give to customer
system core tftp://ip/
ips core dump full
Problem: show cores shows sibyte crashing and not able to
run any fcip commands.
show ips status --- said port1/1 failed (gig1/1 down)
show fcip-lock -- had a lock
show ips internal fcip-trace-log
show break-lock (feww times and show fcip-lock did not have any
locks)
show port internal event fcip showed SW-failed.
ips reset module 1 port 1 ( in ips it will reset in pairs, etherchannel
reason - same memory). ---- use with CARE.. very disruptive.
we could not get fcip link up - link failure error, so
and then killed and restarted ips manager,
show system internal mts buffer (attac)
debug ips fcip error port 1 (attach)
attach mod 1 port 1
show system int mts buffers sap 60 (Csceg 82721)
/open window - start capture
attach mod 1
debug ips fcip write-acc-err port 1
//open another window and start capture
attach mod 2
debug ips fcip write-acc-err port 2
And do the same on the other switch.
show int fcip X counters
show fcip target-tape-session
show fcip sum
show fcip host-tape-session 30
show fcip target-map
show fcip host-map
term len 0
show port internal even errors
show port internal even interface fcip 3
show port internal info global
show port internal info interface fcip 3
show hard internal errors all
show hard internal sup-fc0 errstats
show ips internal even epp interface fcip 3
show ips stats dma interface gig 4/1
show ips stats tcp interface gig 4/1 details
show ips stats buffer interface gig 4/1
show tech internal modul 4
from both switches in addition to
show tech details
show tech fcip
show ips stats tcp interface gig 1/1 det --- look for SACK/retrans
show ips stats dma inter gig 1/1 --- look for timestamp errors
show int gig 1/1 ----- mtu
show int fcip 1 ----- retransmits and other config cwm/retrans times
attach mod 1
show port internal port-control
show port internal link-events
exit
show tech internal link-events
show ips status module 1
show ips stats hw-comp int gig 1/1
show ips stats ip int gig 1/1 ----> reassembly/packets/dropped
show ips stats mac int gig 1/1
-----CRC erros -hw related stuff
extended ping with DF (y)
-
show ips internal eth-trace-logs gigabitethernet 1/1 (on NWMDS02 and
APMDS01)
- show ips internal eth-trace-logs gigabitethernet 1/2 (on NWMDS01 and
APMDS01)
-----------------------------
Internal Commands: don't give to customer
system core tftp://ip/
ips core dump full
Problem: show cores shows sibyte crashing and not able to
run any fcip commands.
show ips status --- said port1/1 failed (gig1/1 down)
show fcip-lock -- had a lock
show ips internal fcip-trace-log
show break-lock (feww times and show fcip-lock did not have any
locks)
show port internal event fcip showed SW-failed.
ips reset module 1 port 1 ( in ips it will reset in pairs, etherchannel
reason - same memory). ---- use with CARE.. very disruptive.
we could not get fcip link up - link failure error, so
and then killed and restarted ips manager,
show system internal mts buffer (attac)
debug ips fcip error port 1 (attach)
attach mod 1 port 1
show system int mts buffers sap 60 (Csceg 82721)
/open window - start capture
attach mod 1
debug ips fcip write-acc-err port 1
//open another window and start capture
attach mod 2
debug ips fcip write-acc-err port 2
And do the same on the other switch.
show int fcip X counters
show fcip target-tape-session
show fcip sum
show fcip host-tape-session 30
show fcip target-map
show fcip host-map
term len 0
show port internal even errors
show port internal even interface fcip 3
show port internal info global
show port internal info interface fcip 3
show hard internal errors all
show hard internal sup-fc0 errstats
show ips internal even epp interface fcip 3
show ips stats dma interface gig 4/1
show ips stats tcp interface gig 4/1 details
show ips stats buffer interface gig 4/1
show tech internal modul 4
from both switches in addition to
show tech details
show tech fcip
FCIP!
Before I get more details
Note this:
- retransmit failure, is because IP network not able to handle the fc traffic
( change CWM burstsize, reduce it tcp cwm burst 10) to so that when there is
fc traffic, it is not that bursty, reducing cwm might cause less traffic
to be sent in bursts when there is increase in fctraffic)
- if you see a lot B2b credit starvation on FC ports, (for eg, ports
which use fcip link, for eg, fc port where storage array is connected,
which replicates to remote storage array), increase send-buffer-size
in fcip profile.
- compression between various different cards might be different
ips-8, 18+4,etc.
- write accelaration/tape accelaration and ivr, please look for transit
vsan and might break fcip WA/TA because of equal cost paths available
via IVR., anyway fcip TA certainly has issues with multiple equal cost paths.
# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
b#wm Enable congestion window monitoring
keepalive-timeout Set keep alive timeout in sec
max-bandwidth-kbps Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps Configure maximum available path bandwidth in Mbps
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisecond
pmtu-enable Enable PMTU Discovery
sack-enable Enable SACK option for TCP
send-buffer-size Send buffer size in KBytes
The CWM parameter : The default value is 10K and should be left untouched under normal conditions. CWM is a way of controlling burstiness after long idle times or loss of Acks. CWM stands for congestion window monitoring .
The keepalive-timeout is the TCP keepalive timeout value and is set to 60 sec. by default . The configurable values range between 1 and 7200 sec.
The max- and min-bandwidth parameter programs the TCP Maximum Window Size (scaling factor) and engages an internal "shaper" functionality . These values should be carefully chosen and requires understanding of intermediate network's end-to-end topology .The default values are to be changed according to the aforementioned requirements.The Round-trip-time can be derived once you have your FCIP tunnel up and running as follows :
bison# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
bison#
Always add an additional margin of a few Microseconds to this value as a minium.
The max-retransmissions counter is set to 4 by default - in a healthy network environment this value should be left unchanged.
The max-retransmission timer is set to 200msec - If you experience extreme high retransmission counters this value might be increased -but in general this would not be required unless the RTT is above the 200msec value .
The PMTU (path mtu discovery) is enabled by default - best practice is to know which is the maximum MTU size supported by all interfaces along the logical path between both peers . Refer to RFC1191 for more details .
The SACK feature (selective acknowledgment) is not enabled by default - it could be considered when you have a lot of retransmissions going on between the two peers - SACK will allow selective retransmissions of your window - which is beneficial if larger maximum window sizes are configured and retransmissions are experienced frequently . In our sample config we have enabled it - when you do that make sure it is enabled at either side of the link .
The send-buffer-size is the amount of buffers in addition to the TCP window we allow to be transmitted out before we start to flow control the FC sources.The default value is set to 0 .
Configuration from MDS9216
c# sh run
Building Configuration ...
fcip profile 200
ip address 200.200.200.1
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10
fcip profile 201
ip address 200.200.200.5
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10
!.....the TCP parameters are identical to what we had configured on the peering FCIP interfaces , only in very specific cases we should consider different values , e.g. if the return-path(s) are running across a different part of the
interface fcip1
channel-group 2 force
no shutdown
use-profile 200
peer-info ipaddr 100.100.100.1
interface fcip2
channel-group 2 force
no shutdown
use-profile 201
peer-info ipaddr 100.100.100.5
!..both fcip1 and fcip2 are bound to the same Channel-group 2 - also note that we have no strict relationship between profile-id and fcip interface numbering here as this is not a requirement. However , from a management and troubleshooting perspective a "strict" relationship of both values is recommended...
FCIP Generic ( outputs needed to troubleshoot)
· show interface gig - Displays status of the relevant gig interface bound to the FCIP profile
· show ips stats tcp int gig details- Displays TCP stats and active connections for the relevant gig interface
. show ips stats dma-bridge int gig x/y - displays timestamp errors
. show ips stats buffer int gig x/y --- any buffer relted issue
- slow fcip connections if buffer is less than 70K
- show int fcip counters -- and show int fcip and show ips stats hw-comp
- hw compression /compression ratio , WA stats, TA stats
· show ips arp int gig - Dispalys all arp entries for the relevant gig interface , next hop or peer should be present in this list
· show ips ip route int gig- displays the specific routes going across the relevant gig interface
· show interface fcip - Displays the fcip interface status and all details related to this fcip tunnel
· show profile fcip - Displays ip address the profile is bound to and all configured TCP parameters
· show interface port - Displays the specified Port-channel number's information
· show int fcipcounters - verify here if there are any frames going through the FCIP tunnel
· show fcdomain vsan - Lists all domain related details - verify here if the fabric is formed cross the fcip tunnel(s)
· show fcns da vsan - Displays all pwwn , FC4-Types and FCID's of the relevant vsan - verify here that all expected entries are distributed across the fcip tunnel(s)
· show
http://www-tac.cisco.com/Teams/SAN/Bru/IPS8_elaborate.htm
show ips - tcp stats/fcip
show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
9506# show ips internal fcip-trace-log
ips measure-rtt ip-address
---------------
show int fcip X counters ( any WA issues like ABTS)
show ips stats tcp/dma
If FC traffic is bursty, you may want to increase sendbuffer size
max 8M ( 3.0 it is 16K), so that FC will dump the frames on to
this buffer, then fcip can process the frames as per bw availability.
This will reduce back filling fc causing lack of b2b credits
If RTT time and TCP window ( how many bytes within RTT) might cause
timestamp errors, if the send buffer is not cleared within fcdrop
latency ( 500 ms)
If RTT is 40 ms and TCP window ( show ips stats) and show int fcip
will give tcp window., is 1256 K (1.2)and if send buffer is 8Mb,
then 8MB send buffer will be cleared with approx 40ms x 8/1.2 =320 ms
but if RTT is 80 ms, then it might take about 640 ms, that means some
frames might get dropped (dma-bridge timestamp errors)
CWM changes the burstiness of TCP side, default 50K means 50K frames
sent without ACK at a burst, this might clean up fcip traffic or
send buffer, but might cause TCP retrans if network can't handle it.
- small send buffer will cause lack of b2b credit and congestion
on FC side
- bigger send buffer will cause time stamperrors if the buffer
is not cleared within fc drop latency
- small cwm might cause time stamp errors because fc frames are
queued (etherenet send queue) and might stay longer in fc/switch
- higher cwm might cause bursty tcp traffic and might lead to
higher retrans.
- higher b2b credit might cause filling up of send buffer quickly.
so good luck tunning fcip params.!
Look for any hardware errors on fcip blade as well.
Requested Send buffer - configured params ( tcp send-buffer)
Allocated send buffer - configured + tcp window
8000+ 1257 = 9257 (for eg.)
fcip profile ---- max/min bw is post compression.
so show int fcip can have more thro'put than max bw configured in fcip
profile
show int gig 's thro'put is deteermined by max/min bw configured on fcip
profile.
Note this:
- retransmit failure, is because IP network not able to handle the fc traffic
( change CWM burstsize, reduce it tcp cwm burst 10) to so that when there is
fc traffic, it is not that bursty, reducing cwm might cause less traffic
to be sent in bursts when there is increase in fctraffic)
- if you see a lot B2b credit starvation on FC ports, (for eg, ports
which use fcip link, for eg, fc port where storage array is connected,
which replicates to remote storage array), increase send-buffer-size
in fcip profile.
- compression between various different cards might be different
ips-8, 18+4,etc.
- write accelaration/tape accelaration and ivr, please look for transit
vsan and might break fcip WA/TA because of equal cost paths available
via IVR., anyway fcip TA certainly has issues with multiple equal cost paths.
# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
b#wm Enable congestion window monitoring
keepalive-timeout Set keep alive timeout in sec
max-bandwidth-kbps Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps Configure maximum available path bandwidth in Mbps
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisecond
pmtu-enable Enable PMTU Discovery
sack-enable Enable SACK option for TCP
send-buffer-size Send buffer size in KBytes
The CWM parameter : The default value is 10K and should be left untouched under normal conditions. CWM is a way of controlling burstiness after long idle times or loss of Acks. CWM stands for congestion window monitoring .
The keepalive-timeout is the TCP keepalive timeout value and is set to 60 sec. by default . The configurable values range between 1 and 7200 sec.
The max- and min-bandwidth parameter programs the TCP Maximum Window Size (scaling factor) and engages an internal "shaper" functionality . These values should be carefully chosen and requires understanding of intermediate network's end-to-end topology .The default values are to be changed according to the aforementioned requirements.The Round-trip-time can be derived once you have your FCIP tunnel up and running as follows :
bison# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
bison#
Always add an additional margin of a few Microseconds to this value as a minium.
The max-retransmissions counter is set to 4 by default - in a healthy network environment this value should be left unchanged.
The max-retransmission timer is set to 200msec - If you experience extreme high retransmission counters this value might be increased -but in general this would not be required unless the RTT is above the 200msec value .
The PMTU (path mtu discovery) is enabled by default - best practice is to know which is the maximum MTU size supported by all interfaces along the logical path between both peers . Refer to RFC1191 for more details .
The SACK feature (selective acknowledgment) is not enabled by default - it could be considered when you have a lot of retransmissions going on between the two peers - SACK will allow selective retransmissions of your window - which is beneficial if larger maximum window sizes are configured and retransmissions are experienced frequently . In our sample config we have enabled it - when you do that make sure it is enabled at either side of the link .
The send-buffer-size is the amount of buffers in addition to the TCP window we allow to be transmitted out before we start to flow control the FC sources.The default value is set to 0 .
Configuration from MDS9216
c# sh run
Building Configuration ...
fcip profile 200
ip address 200.200.200.1
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10
fcip profile 201
ip address 200.200.200.5
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10
!.....the TCP parameters are identical to what we had configured on the peering FCIP interfaces , only in very specific cases we should consider different values , e.g. if the return-path(s) are running across a different part of the
interface fcip1
channel-group 2 force
no shutdown
use-profile 200
peer-info ipaddr 100.100.100.1
interface fcip2
channel-group 2 force
no shutdown
use-profile 201
peer-info ipaddr 100.100.100.5
!..both fcip1 and fcip2 are bound to the same Channel-group 2 - also note that we have no strict relationship between profile-id and fcip interface numbering here as this is not a requirement. However , from a management and troubleshooting perspective a "strict" relationship of both values is recommended...
FCIP Generic ( outputs needed to troubleshoot)
· show interface gig
· show ips stats tcp int gig
. show ips stats dma-bridge int gig x/y - displays timestamp errors
. show ips stats buffer int gig x/y --- any buffer relted issue
- slow fcip connections if buffer is less than 70K
- show int fcip counters -- and show int fcip and show ips stats hw-comp
- hw compression /compression ratio , WA stats, TA stats
· show ips arp int gig
· show ips ip route int gig
· show interface fcip
· show profile fcip
· show interface port
· show int fcip
· show fcdomain vsan
· show fcns da vsan
· show
http://www-tac.cisco.com/Teams/SAN/Bru/IPS8_elaborate.htm
show ips - tcp stats/fcip
show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
9506# show ips internal fcip-trace-log
ips measure-rtt ip-address
---------------
show int fcip X counters ( any WA issues like ABTS)
show ips stats tcp/dma
If FC traffic is bursty, you may want to increase sendbuffer size
max 8M ( 3.0 it is 16K), so that FC will dump the frames on to
this buffer, then fcip can process the frames as per bw availability.
This will reduce back filling fc causing lack of b2b credits
If RTT time and TCP window ( how many bytes within RTT) might cause
timestamp errors, if the send buffer is not cleared within fcdrop
latency ( 500 ms)
If RTT is 40 ms and TCP window ( show ips stats) and show int fcip
will give tcp window., is 1256 K (1.2)and if send buffer is 8Mb,
then 8MB send buffer will be cleared with approx 40ms x 8/1.2 =320 ms
but if RTT is 80 ms, then it might take about 640 ms, that means some
frames might get dropped (dma-bridge timestamp errors)
CWM changes the burstiness of TCP side, default 50K means 50K frames
sent without ACK at a burst, this might clean up fcip traffic or
send buffer, but might cause TCP retrans if network can't handle it.
- small send buffer will cause lack of b2b credit and congestion
on FC side
- bigger send buffer will cause time stamperrors if the buffer
is not cleared within fc drop latency
- small cwm might cause time stamp errors because fc frames are
queued (etherenet send queue) and might stay longer in fc/switch
- higher cwm might cause bursty tcp traffic and might lead to
higher retrans.
- higher b2b credit might cause filling up of send buffer quickly.
so good luck tunning fcip params.!
Look for any hardware errors on fcip blade as well.
Requested Send buffer - configured params ( tcp send-buffer)
Allocated send buffer - configured + tcp window
8000+ 1257 = 9257 (for eg.)
fcip profile ---- max/min bw is post compression.
so show int fcip can have more thro'put than max bw configured in fcip
profile
show int gig 's thro'put is deteermined by max/min bw configured on fcip
profile.
RSPAN!
Caveats:
- trunking interface needed.
- ip routing needs to be enabled
on switches
- fc-tunnel needs to be enabled.
- vsan interface in same subnet in all three switches.
( participating switches)
Switch 1: (intermediate switch).
fc-tunnel enable
interface vsan100
ip address 10.1.1.75 255.255.255.0
no shutdown
ip routing
-----
SWitch II : ( rspan desitnation)
fc-tunnel enable
fc-tunnel tunnel-id-map 100 interface fc1/12
interface fc1/12
switchport mode SD
switchport speed 2000
connect DS_PAA here
interface vsan100
ip address 10.1.1.81 255.255.255.0
no shutdown
ip routing
Switch III: Rspan Source
fc-tunnel enable
interface fc-tunnel 100
destination 10.1.1.81
source 10.1.1.82
explicit-path rspan
no shutdown
fc-tunnel explicit-path rspan
next-address 10.1.1.75 strict
interface vsan100
ip address 10.1.1.82 255.255.255.0
no shutdown
ip routing
interface fc1/8^M
switchport mode ST
switchport speed 2000
rspan-tunnel interface fc-tunnel 100
no shut
----
span session 1^M
destination interface fc-tunnel 100
source interface fc1/10 rx
source interface fc1/10 tx
----
Testing:
show int fc-tunnel 100 ( it should be up, if
it wating for RESV, it means either path
is not reachabled, fspf cost is messed up
( if up and down cost for specific path is
different on vsan 100, or ST is not up or
SD is not configured ).
fc-tunnel 100 is up^M
Dest IP Addr: 10.1.1.81 Tunnel ID: 100^M
Source IP Addr: 10.1.1.82 LSP ID: 1^M
Explicit Path Name: rspan^M
Outgoing interface: port-channel 2^M
Outgoing Label(s) to Insert: 10008:0:1:ff'h ^M
Record Routes:^M
10.1.1.75 ^M
10.1.1.81 ^M
^M
- trunking interface needed.
- ip routing needs to be enabled
on switches
- fc-tunnel needs to be enabled.
- vsan interface in same subnet in all three switches.
( participating switches)
Switch 1: (intermediate switch).
fc-tunnel enable
interface vsan100
ip address 10.1.1.75 255.255.255.0
no shutdown
ip routing
-----
SWitch II : ( rspan desitnation)
fc-tunnel enable
fc-tunnel tunnel-id-map 100 interface fc1/12
interface fc1/12
switchport mode SD
switchport speed 2000
connect DS_PAA here
interface vsan100
ip address 10.1.1.81 255.255.255.0
no shutdown
ip routing
Switch III: Rspan Source
fc-tunnel enable
interface fc-tunnel 100
destination 10.1.1.81
source 10.1.1.82
explicit-path rspan
no shutdown
fc-tunnel explicit-path rspan
next-address 10.1.1.75 strict
interface vsan100
ip address 10.1.1.82 255.255.255.0
no shutdown
ip routing
interface fc1/8^M
switchport mode ST
switchport speed 2000
rspan-tunnel interface fc-tunnel 100
no shut
----
span session 1^M
destination interface fc-tunnel 100
source interface fc1/10 rx
source interface fc1/10 tx
----
Testing:
show int fc-tunnel 100 ( it should be up, if
it wating for RESV, it means either path
is not reachabled, fspf cost is messed up
( if up and down cost for specific path is
different on vsan 100, or ST is not up or
SD is not configured ).
fc-tunnel 100 is up^M
Dest IP Addr: 10.1.1.81 Tunnel ID: 100^M
Source IP Addr: 10.1.1.82 LSP ID: 1^M
Explicit Path Name: rspan^M
Outgoing interface: port-channel 2^M
Outgoing Label(s) to Insert: 10008:0:1:ff'h ^M
Record Routes:^M
10.1.1.75 ^M
10.1.1.81 ^M
^M
iscsi
Proxy Iscsi Initiator
* pcit
* One time Config
* Config for each initiator
* CSM mapping
* show iscsi commands
* isid
* ethereal trace
* FC trace of two iscsi sessions with proxy initiator and Xiotech target
* FC trace for two iscsi sessions with different MTU
* proxy iscsi initiator in multiple vsans
Benefits:
- simple zoning ( no need to configure all the hosts wwns)
- simple lun mapping at the storage.
- it is in certain way like SN5428 , where we configure storage lun mapping for the internal
HBAs and there is no individual wwns for each initiator.
MDS Config:
One time Config:
MDS9509-B1-sup1(config)# iscsi interface-vsan-member-enable
by default proxy wwn goes to vsan 1 to change that, we need to do these steps.
MDS9509-B1-sup1(config)# vsan 40 interface iscsi 4/1
( where my targets are).
interface iscsi 4/1
switchport proxy-initiator ( if you want you can configure wwns manual too!)
show interface iscsi 4/1
Proxy Initiator Mode : enabled
nWWN is 20:15:00:05:30:00:24:60 (system-assigned)
pWWN is 20:16:00:05:30:00:24:60 (system-assigned)
Add this pwwn to Zoning/ and configure storage Lun mapping
for CSM:
MDS9509-B1-sup1(svc)# show cluster tacCluster host proxy-iscsi
Host proxy-iscsi:
Number of port is 2
Port WWN is 20:16:00:05:30:00:24:60
LUN 0 : vdisk piscsi1
LUN 1 : vdisk piscsi2
LUN 2 : vdisk piscsi3
LUN 3 : vdisk piscsi4
LUN 4 : vdisk piscsi5
LUN 5 : vdisk piscsi6
LUN 6 : vdisk piscsi7
LUN 7 : vdisk piscsi8
-----------
Configuration for each host (this config on MDS host - configuring two virtual targets
(multipathing) for ip 172.69.122.104)
You can not do lunzoning/zoning based on ip address with proxy initiator.(even after setting
switch initiator id ip-address on iscsi port.( verify CSCed82704 ))
All the subsequent initiators will have access to same luns/storages as the first initiator.
In the MDS 1.3.3 config guide, under the sections "Configuring iSCSI proxy initiators"
(Configuring iSCSI/Configuring IP Storage), it is clearly mentioned that when in proxy
initiator mode, you cannot use iSCSI attributes in the FC access control mechanisms.
You have to use iSCSI based access control to accomplish the same.
(still needs to verify above).
ip route 171.69.122.104 255.255.255.255 interface gig 4/1
iscsi virtual-target name csm
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit
iscsi virtual-target name csm2
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit <<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator
Linux host config on MDS:
ip route 172.69.122.104 255.255.255.255 interface GigabitEthernet4/1
iscsi virtual-target name csm-linux
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator
iscsi virtual-target name csm-linux2
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
advertise interface GigabitEthernet4/1
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator
----------
on the iscsi PC host , I could see two luns for each targets.
VSAN 40:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x790000 N 20:04:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790001 N 20:09:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790005 N 20:16:00:05:30:00:24:60 (Cisco) scsi-fcp:init isc..w (only one initiator)
0x790100 N 21:00:00:e0:8b:0b:fc:0d (QLogic) scsi-fcp:init
0x790200 N 21:00:00:e0:8b:08:f6:18 (QLogic) ipfc scsi-fcp:init
0x790300 N 21:01:00:e0:8b:28:f6:18 (QLogic) ipfc scsi-fcp:init
---
MDS9509-B1-sup1# show iscsi session
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
Session #1
Target csm
VSAN 40, ISID 400001370018, Status active, no reservation
Session #2
Target csm2
VSAN 40, ISID 400001370019, Status active, no reservation
Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1
Target csm-linux
VSAN 40, ISID 801234567800, Status active, no reservation
Session #2
Target csm-linux2
VSAN 40, ISID 801234567801, Status active, no reservation
MDS9509-B1-sup1# show iscsi initiator
iSCSI Node name is iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
iSCSI alias name:
Node WWN is 20:02:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005
iSCSI Node name is dhcp-173-228
Initiator ip addr (s): 171.69.104.104
iSCSI alias name:
Node WWN is 20:00:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005
On the storage , let us see which session is logged (proxy or the actual iscsi host)- it is proxy.
MDS9509-B1-sup1# show svc session svc 2/1 (we don't see 20:02 ....60 wwn of actual iscsi host
being logged on to the storage)
svc2/1:
Target N-port WWN is 20:04:00:05:30:00:24:60, vsan is 40, FCID is 0x790000
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:01:00:05:30:00:24:60, vsan is 30, FCID is 0x780000
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:05:00:05:30:00:24:60, vsan is 50, FCID is 0xd40000
pWWN 20:14:00:05:30:00:24:60, nWWN 20:0f:00:05:30:00:24:60, FCID 0xd40001
MDS9509-B1-sup1# show svc session svc 2/2
svc2/2:
Target N-port WWN is 20:09:00:05:30:00:24:60, vsan is 40, FCID is 0x790001
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:08:00:05:30:00:24:60, vsan is 30, FCID is 0x780002
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:14:00:05:30:00:24:60, vsan is 50, FCID is 0xd40001
pWWN 20:05:00:05:30:00:24:60, nWWN 20:0e:00:05:30:00:24:60, FCID 0xd40000
------
debug ips iscsi flow
(with only pc iscsi connection)
Debugs:
MDS9509-B1-sup1# Dec 15 12:18:51 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:18:51 ips: Created initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Initiator(8) got nwwn 2002000530002460
Dec 15 12:18:51 ips: Initiator(8) got vsan list
Dec 15 12:18:51 ips: no:1 vsan_id 1
Dec 15 12:18:51 ips: Created an fc_port(7) pgt 384 iscsi-if-index 0x0b180000 intf 0x02180000 ip-addr: 172.16.34.10 for initiator(8)
Dec 15 12:18:51 ips: Created session(39) target name isid 400001370016 for initiator(8)
Dec 15 12:18:51 ips: fc_port(7) has a pwwn 0, mode: 1
Dec 15 12:18:51 ips: Put iscsi4/1 in vsan 40 status: 0
Dec 15 12:18:51 ips: Fc_port(7) pwwn 2016000530002460 member of 1 vsans registered 0
Dec 15 12:18:51 ips: fc_port(7) sent 1 flogi requests
Dec 15 12:18:51 ips: Flogi response: fc_port(7) fcid 00790005 in vsan 40
Dec 15 12:18:51 ips: fc_port(7) pwwn 2016000530002460 sent 1 NS reg requests
Dec 15 12:18:51 ips: NS reg resp: fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 fcid 00790005 vsan 40
Dec 15 12:18:51 ips: Discovery session.. no need to check target
Dec 15 12:18:51 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370016]
Dec 15 12:18:51 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:18:51 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:18:51 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:18:51 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:18:51 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:18:51 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:18:51 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:18:51 ips: Removing session(39) tgt-name: isid: 400001370016 failure code: 1
Dec 15 12:19:12 ips: Node 2016000530002460, vsan 40 is not discovered as init or target
Dec 15 12:19:12 ips: Initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com processing tgt_online 2016000530002460 vsan 40
Refresh on MS initiator
Dec 15 12:19:54 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:19:54 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:19:54 ips: Created session(40) target name isid 400001370017 for initiator(8)
Dec 15 12:19:54 ips: Discovery session.. no need to check target
Dec 15 12:19:54 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370017]
Dec 15 12:19:54 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:19:54 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:19:54 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:19:54 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:19:54 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:19:54 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:19:54 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:19:54 ips: Removing session(40) tgt-name: isid: 400001370017 failure code:
1
Logon CSM:
Dec 15 12:20:36 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm
Dec 15 12:20:36 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:20:36 ips: Created session(41) target name csm isid 400001370018 for initiator(8)
Dec 15 12:20:36 ips: Target csm a virtual target checking access
Dec 15 12:20:36 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:20:36 ips: fc_port(7) Querying NS for target pwwn:[2004000530002460] sec pwwn:[0] wait 1
Dec 15 12:20:36 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790000
Dec 15 12:20:36 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm] isid:[400001370018]
Logon CSM2
Dec 15 12:21:18 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm2
Dec 15 12:21:18 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:21:18 ips: Created session(42) target name csm2 isid 400001370019 for initiator(8)
Dec 15 12:21:18 ips: Target csm2 a virtual target checking access
Dec 15 12:21:18 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:21:18 ips: fc_port(7) Querying NS for target pwwn:[2009000530002460] sec pwwn:[0] wait 1
Dec 15 12:21:18 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790001
Dec 15 12:21:18 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm2] isid:[400001370019]
--------
Etherreal Trace from Linux:
Linux host 171.69.104.104 to proxy iscsi initiator using unh_iscsi with two virtual-targets to two CSM nodes
(one each) and these virtual targets have been mapped to one iscsi lun 0 (fc-lun 3).
configured unh iscsi conf : initiator dhcp-173-228 ,
target =csm-linux and csm-linux2 and the ipaddress
of target 172.16.34.10.
cat /proc/scsi/scsi showed two disks which are same.
[root@dhcp-173-228 root]# cat /proc/scsi/scsi
(only iscsi devices displayed)
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
etherreal trace
MDS9509-B1-sup1# show iscsi session detail
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.48
Session #1 (index 4)
Target xiotech
VSAN 40, ISID 400001370004, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 20104, MaxCmdSN 20119, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 2
Stats:
PDU: Command: 933, Response: 933
Bytes: TX: 20158288, RX: 19995648
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.122.48
CID 1, State: Full-Feature
StatSN 937, ExpStatSN 0
MaxRecvDSLength 65536, our_MaxRecvDSLength 1024
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: Yes
Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1 (index 2)
Target xiotech-linux
VSAN 40, ISID 801234567800, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 24429, MaxCmdSN 24443, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 1
Stats:
PDU: Command: 2207, Response: 2206
Bytes: TX: 128568, RX: 202783744
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.104.104
CID 0, State: Full-Feature
StatSN 2209, ExpStatSN 0
MaxRecvDSLength 1392, our_MaxRecvDSLength 1392
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: No
Just for recap: here are the virtual targets defined.
target: xiotech
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 2
iSCSI LUN: 0x0000, FC LUN: 0x0000
iSCSI LUN: 0x0001, FC LUN: 0x0001
No. of initiators permitted: 1
initiator 171.69.122.48/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled
target: xiotech-linux
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 1
iSCSI LUN: 0x0000, FC LUN: 0x0002
No. of initiators permitted: 1
initiator 171.69.104.104/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled
MDS9509-B1-sup1# show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
Connection Stats
0 active openings, 109 accepts
0 failed attempts, 0 reset received, 109 established
Segment stats
4564268 received, 1961372 sent, 1133 retransmitted
43 bad segments received, 0 reset sent
TCP Active Connections
Local Address Remote Address State Send-Q Recv-Q
172.16.34.10:3260 171.69.122.48:1593 ESTABLISH 0 0
172.16.34.10:3260 171.69.104.104:32779 ESTABLISH 0 0
0.0.0.0:3260 0.0.0.0:0 LISTEN 0 0
Traces are in this directory
Here is the trace snapshots of proxy_linux1_win2_logoff.
a. no iscsi sessions were logged on before the taking the trace.
b. started /etc/init.d/unh_iscsi start ( Linux has fc-lun 2 mapped to iscsi-lun 0)
- you see prli from the proxy initiator
- lun inquiry proxied for Linux.
prli_linux_inquiry
3. mount /dev/sdb1 /xiotech1 and deleted some files in /xiotech1
4. Using microsoft initiator, I connect to virtual target xiotech , u will see microsoft's inquiry, no new plogi or prli session
initiated. (probably if PDU of this session is lower, then we might reinitiate , so that PMTU Is reneogiated.
windows_inquiry
and finally I removed windows session and then the linux session, so you will prlo as last iscsi session is cleared up.
prlo
ip payload size is 1460. (+ 20 byte TCP Options from ethereal trace )
MSS in PC is 1460 bytes
iscsi payload is 1440 ( iscsi header is 48 bytes)
FC data size is 1392
Example II:
win iscsi initiator logs in first with default MTU size (MSS 1460)
linux iscsi initiator logs in second with mtu size of 800.
We expect LOGO And PRLO to happen because proxy initiator relogs to target with lower Receive data field Size.
here is the picture with PLOGI when the win2k initiator comes in. Note the Class 3 receive data field size.
win2k_prli
after a little bit, here comes the linux session with lower MTU ( 800)
(setting up Linux mtu)
at> ifconfig eth0 mtu 800
at> ifconfig eth0 down
at> ifconfig eth0 up
at> route add default gw 171.69.104.1
second
iSCSI initiator 20.1.2.12 will be in VSAN 40, 41, 50 and 51. Not in VSAN 30. All initiators without "iscsi initiator" command or without vsan command will be in VSAN 30.
iscsi initiator ip-address 20.1.2.12
vsan 40
vsan 41
vsan 50
vsan 51
interface iscsi3/3
switchport initiator id ip-address
switchport proxy-initiator nWWN 11:11:11:11:11:11:11:00 pWWN 11:11:11:11:11:11:11:11
vsan database
vsan 30 interface iscsi 3/3
* pcit
* One time Config
* Config for each initiator
* CSM mapping
* show iscsi commands
* isid
* ethereal trace
* FC trace of two iscsi sessions with proxy initiator and Xiotech target
* FC trace for two iscsi sessions with different MTU
* proxy iscsi initiator in multiple vsans
Benefits:
- simple zoning ( no need to configure all the hosts wwns)
- simple lun mapping at the storage.
- it is in certain way like SN5428 , where we configure storage lun mapping for the internal
HBAs and there is no individual wwns for each initiator.
MDS Config:
One time Config:
MDS9509-B1-sup1(config)# iscsi interface-vsan-member-enable
by default proxy wwn goes to vsan 1 to change that, we need to do these steps.
MDS9509-B1-sup1(config)# vsan 40 interface iscsi 4/1
( where my targets are).
interface iscsi 4/1
switchport proxy-initiator ( if you want you can configure wwns manual too!)
show interface iscsi 4/1
Proxy Initiator Mode : enabled
nWWN is 20:15:00:05:30:00:24:60 (system-assigned)
pWWN is 20:16:00:05:30:00:24:60 (system-assigned)
Add this pwwn to Zoning/ and configure storage Lun mapping
for CSM:
MDS9509-B1-sup1(svc)# show cluster tacCluster host proxy-iscsi
Host proxy-iscsi:
Number of port is 2
Port WWN is 20:16:00:05:30:00:24:60
LUN 0 : vdisk piscsi1
LUN 1 : vdisk piscsi2
LUN 2 : vdisk piscsi3
LUN 3 : vdisk piscsi4
LUN 4 : vdisk piscsi5
LUN 5 : vdisk piscsi6
LUN 6 : vdisk piscsi7
LUN 7 : vdisk piscsi8
-----------
Configuration for each host (this config on MDS host - configuring two virtual targets
(multipathing) for ip 172.69.122.104)
You can not do lunzoning/zoning based on ip address with proxy initiator.(even after setting
switch initiator id ip-address on iscsi port.( verify CSCed82704 ))
All the subsequent initiators will have access to same luns/storages as the first initiator.
In the MDS 1.3.3 config guide, under the sections "Configuring iSCSI proxy initiators"
(Configuring iSCSI/Configuring IP Storage), it is clearly mentioned that when in proxy
initiator mode, you cannot use iSCSI attributes in the FC access control mechanisms.
You have to use iSCSI based access control to accomplish the same.
(still needs to verify above).
ip route 171.69.122.104 255.255.255.255 interface gig 4/1
iscsi virtual-target name csm
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit
iscsi virtual-target name csm2
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit <<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator
Linux host config on MDS:
ip route 172.69.122.104 255.255.255.255 interface GigabitEthernet4/1
iscsi virtual-target name csm-linux
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator
iscsi virtual-target name csm-linux2
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
advertise interface GigabitEthernet4/1
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator
----------
on the iscsi PC host , I could see two luns for each targets.
VSAN 40:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x790000 N 20:04:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790001 N 20:09:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790005 N 20:16:00:05:30:00:24:60 (Cisco) scsi-fcp:init isc..w (only one initiator)
0x790100 N 21:00:00:e0:8b:0b:fc:0d (QLogic) scsi-fcp:init
0x790200 N 21:00:00:e0:8b:08:f6:18 (QLogic) ipfc scsi-fcp:init
0x790300 N 21:01:00:e0:8b:28:f6:18 (QLogic) ipfc scsi-fcp:init
---
MDS9509-B1-sup1# show iscsi session
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
Session #1
Target csm
VSAN 40, ISID 400001370018, Status active, no reservation
Session #2
Target csm2
VSAN 40, ISID 400001370019, Status active, no reservation
Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1
Target csm-linux
VSAN 40, ISID 801234567800, Status active, no reservation
Session #2
Target csm-linux2
VSAN 40, ISID 801234567801, Status active, no reservation
MDS9509-B1-sup1# show iscsi initiator
iSCSI Node name is iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
iSCSI alias name:
Node WWN is 20:02:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005
iSCSI Node name is dhcp-173-228
Initiator ip addr (s): 171.69.104.104
iSCSI alias name:
Node WWN is 20:00:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005
On the storage , let us see which session is logged (proxy or the actual iscsi host)- it is proxy.
MDS9509-B1-sup1# show svc session svc 2/1 (we don't see 20:02 ....60 wwn of actual iscsi host
being logged on to the storage)
svc2/1:
Target N-port WWN is 20:04:00:05:30:00:24:60, vsan is 40, FCID is 0x790000
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:01:00:05:30:00:24:60, vsan is 30, FCID is 0x780000
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:05:00:05:30:00:24:60, vsan is 50, FCID is 0xd40000
pWWN 20:14:00:05:30:00:24:60, nWWN 20:0f:00:05:30:00:24:60, FCID 0xd40001
MDS9509-B1-sup1# show svc session svc 2/2
svc2/2:
Target N-port WWN is 20:09:00:05:30:00:24:60, vsan is 40, FCID is 0x790001
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:08:00:05:30:00:24:60, vsan is 30, FCID is 0x780002
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:14:00:05:30:00:24:60, vsan is 50, FCID is 0xd40001
pWWN 20:05:00:05:30:00:24:60, nWWN 20:0e:00:05:30:00:24:60, FCID 0xd40000
------
debug ips iscsi flow
(with only pc iscsi connection)
Debugs:
MDS9509-B1-sup1# Dec 15 12:18:51 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:18:51 ips: Created initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Initiator(8) got nwwn 2002000530002460
Dec 15 12:18:51 ips: Initiator(8) got vsan list
Dec 15 12:18:51 ips: no:1 vsan_id 1
Dec 15 12:18:51 ips: Created an fc_port(7) pgt 384 iscsi-if-index 0x0b180000 intf 0x02180000 ip-addr: 172.16.34.10 for initiator(8)
Dec 15 12:18:51 ips: Created session(39) target name isid 400001370016 for initiator(8)
Dec 15 12:18:51 ips: fc_port(7) has a pwwn 0, mode: 1
Dec 15 12:18:51 ips: Put iscsi4/1 in vsan 40 status: 0
Dec 15 12:18:51 ips: Fc_port(7) pwwn 2016000530002460 member of 1 vsans registered 0
Dec 15 12:18:51 ips: fc_port(7) sent 1 flogi requests
Dec 15 12:18:51 ips: Flogi response: fc_port(7) fcid 00790005 in vsan 40
Dec 15 12:18:51 ips: fc_port(7) pwwn 2016000530002460 sent 1 NS reg requests
Dec 15 12:18:51 ips: NS reg resp: fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 fcid 00790005 vsan 40
Dec 15 12:18:51 ips: Discovery session.. no need to check target
Dec 15 12:18:51 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370016]
Dec 15 12:18:51 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:18:51 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:18:51 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:18:51 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:18:51 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:18:51 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:18:51 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:18:51 ips: Removing session(39) tgt-name: isid: 400001370016 failure code: 1
Dec 15 12:19:12 ips: Node 2016000530002460, vsan 40 is not discovered as init or target
Dec 15 12:19:12 ips: Initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com processing tgt_online 2016000530002460 vsan 40
Refresh on MS initiator
Dec 15 12:19:54 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:19:54 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:19:54 ips: Created session(40) target name isid 400001370017 for initiator(8)
Dec 15 12:19:54 ips: Discovery session.. no need to check target
Dec 15 12:19:54 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370017]
Dec 15 12:19:54 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:19:54 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:19:54 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:19:54 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:19:54 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:19:54 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:19:54 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:19:54 ips: Removing session(40) tgt-name: isid: 400001370017 failure code:
1
Logon CSM:
Dec 15 12:20:36 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm
Dec 15 12:20:36 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:20:36 ips: Created session(41) target name csm isid 400001370018 for initiator(8)
Dec 15 12:20:36 ips: Target csm a virtual target checking access
Dec 15 12:20:36 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:20:36 ips: fc_port(7) Querying NS for target pwwn:[2004000530002460] sec pwwn:[0] wait 1
Dec 15 12:20:36 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790000
Dec 15 12:20:36 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm] isid:[400001370018]
Logon CSM2
Dec 15 12:21:18 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm2
Dec 15 12:21:18 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:21:18 ips: Created session(42) target name csm2 isid 400001370019 for initiator(8)
Dec 15 12:21:18 ips: Target csm2 a virtual target checking access
Dec 15 12:21:18 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:21:18 ips: fc_port(7) Querying NS for target pwwn:[2009000530002460] sec pwwn:[0] wait 1
Dec 15 12:21:18 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790001
Dec 15 12:21:18 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm2] isid:[400001370019]
--------
Etherreal Trace from Linux:
Linux host 171.69.104.104 to proxy iscsi initiator using unh_iscsi with two virtual-targets to two CSM nodes
(one each) and these virtual targets have been mapped to one iscsi lun 0 (fc-lun 3).
configured unh iscsi conf : initiator dhcp-173-228 ,
target =csm-linux and csm-linux2 and the ipaddress
of target 172.16.34.10.
cat /proc/scsi/scsi showed two disks which are same.
[root@dhcp-173-228 root]# cat /proc/scsi/scsi
(only iscsi devices displayed)
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
etherreal trace
MDS9509-B1-sup1# show iscsi session detail
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.48
Session #1 (index 4)
Target xiotech
VSAN 40, ISID 400001370004, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 20104, MaxCmdSN 20119, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 2
Stats:
PDU: Command: 933, Response: 933
Bytes: TX: 20158288, RX: 19995648
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.122.48
CID 1, State: Full-Feature
StatSN 937, ExpStatSN 0
MaxRecvDSLength 65536, our_MaxRecvDSLength 1024
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: Yes
Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1 (index 2)
Target xiotech-linux
VSAN 40, ISID 801234567800, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 24429, MaxCmdSN 24443, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 1
Stats:
PDU: Command: 2207, Response: 2206
Bytes: TX: 128568, RX: 202783744
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.104.104
CID 0, State: Full-Feature
StatSN 2209, ExpStatSN 0
MaxRecvDSLength 1392, our_MaxRecvDSLength 1392
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: No
Just for recap: here are the virtual targets defined.
target: xiotech
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 2
iSCSI LUN: 0x0000, FC LUN: 0x0000
iSCSI LUN: 0x0001, FC LUN: 0x0001
No. of initiators permitted: 1
initiator 171.69.122.48/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled
target: xiotech-linux
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 1
iSCSI LUN: 0x0000, FC LUN: 0x0002
No. of initiators permitted: 1
initiator 171.69.104.104/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled
MDS9509-B1-sup1# show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
Connection Stats
0 active openings, 109 accepts
0 failed attempts, 0 reset received, 109 established
Segment stats
4564268 received, 1961372 sent, 1133 retransmitted
43 bad segments received, 0 reset sent
TCP Active Connections
Local Address Remote Address State Send-Q Recv-Q
172.16.34.10:3260 171.69.122.48:1593 ESTABLISH 0 0
172.16.34.10:3260 171.69.104.104:32779 ESTABLISH 0 0
0.0.0.0:3260 0.0.0.0:0 LISTEN 0 0
Traces are in this directory
Here is the trace snapshots of proxy_linux1_win2_logoff.
a. no iscsi sessions were logged on before the taking the trace.
b. started /etc/init.d/unh_iscsi start ( Linux has fc-lun 2 mapped to iscsi-lun 0)
- you see prli from the proxy initiator
- lun inquiry proxied for Linux.
prli_linux_inquiry
3. mount /dev/sdb1 /xiotech1 and deleted some files in /xiotech1
4. Using microsoft initiator, I connect to virtual target xiotech , u will see microsoft's inquiry, no new plogi or prli session
initiated. (probably if PDU of this session is lower, then we might reinitiate , so that PMTU Is reneogiated.
windows_inquiry
and finally I removed windows session and then the linux session, so you will prlo as last iscsi session is cleared up.
prlo
ip payload size is 1460. (+ 20 byte TCP Options from ethereal trace )
MSS in PC is 1460 bytes
iscsi payload is 1440 ( iscsi header is 48 bytes)
FC data size is 1392
Example II:
win iscsi initiator logs in first with default MTU size (MSS 1460)
linux iscsi initiator logs in second with mtu size of 800.
We expect LOGO And PRLO to happen because proxy initiator relogs to target with lower Receive data field Size.
here is the picture with PLOGI when the win2k initiator comes in. Note the Class 3 receive data field size.
win2k_prli
after a little bit, here comes the linux session with lower MTU ( 800)
(setting up Linux mtu)
at> ifconfig eth0 mtu 800
at> ifconfig eth0 down
at> ifconfig eth0 up
at> route add default gw 171.69.104.1
second
iSCSI initiator 20.1.2.12 will be in VSAN 40, 41, 50 and 51. Not in VSAN 30. All initiators without "iscsi initiator" command or without vsan command will be in VSAN 30.
iscsi initiator ip-address 20.1.2.12
vsan 40
vsan 41
vsan 50
vsan 51
interface iscsi3/3
switchport initiator id ip-address
switchport proxy-initiator nWWN 11:11:11:11:11:11:11:00 pWWN 11:11:11:11:11:11:11:11
vsan database
vsan 30 interface iscsi 3/3
Subscribe to:
Posts (Atom)