MDS Fc port stuck in initializing

Reasons:
- flogi server not responding correctly
show flogi internal event interface fc x/y

- persistent fcid limit exceeded ( 255 entries with all area unique fcid because
of qlogic hbas)

MDS9513-83-SJ# show run | include "vsan Z" | include fcid | count
where Z is vsan #.
- conflict in persistent fcids ( MDS can't assign the fcid allocated
to WWN)

- # of devices more than 255 if qlogic or devices that need area unique fcid
is allocated

- supervisor failure or module issue, flogi command not reaching the supervisor
or flogi process.

show fc2 internal even errors

these are few causes.

Flex Attach in MDS!

MDS gives virtual pwwn that is used for zoning and lun masking, so when a host
connected a port dies, you can either connect a new host to same port or reconfigure
flexattach config on port where spare host is connected. So this is a security concern
because anyone can come and attach another host to port and get all the access to luns.
Similar way someone can replace or remove hba and connect it to a different server.

This can be reduced by using port-security.

http://www.cisco.com/en/US/docs/storage/san_switches/mds9000/sw/rel_3_x/command/reference/CR03_f.html#wp1393061

MDS callhome for bootflash errors!

When a bootflash fails, if callhome is configured correctly, we would expect
XML destination to send the callhome, it does not happen even if Cisco_TAC alert group
is added because of the bug
CSCso71302.
Workaround is defined in the above Cisco MDS bug.

Symptom:

BOOTFLASH failure does not generate callhome to default XML destination profile.

Workaround:
Default XML destination profile is configured to receive alerts from Cisco-Tac alert group by default. You need to add linecard-hardware and supervisor-hardware alert groups to that destination profile to get callhome messages like BOOTFLASH failure


for eg.
callhome
destination-profile xml message-level 2
destination-profile xml email-addr auto-notify@cisco.com
destination-profile xml alert-group linecard-hardware
destination-profile xml alert-group supervisor-hardware
destination-profile xml alert syslog-group-port

NPIV sample config

Sample config:
9513(NPIV enable)--fc2/2---- ---fc1/1(NP port)---9134---(fc1/10 host)

All the ports are in vsan 30
NPV switch 9134 config:

Make sure switch password and console access is there before you do this,
npv enable.

npv enable and set the port to NP mode

MDS9134-SJ# show run int fc1/1
version 3.2(1a)

interface fc1/1
port-license acquire
switchport mode NP
no shutdown

This port is upstream to NPIV enabled switch and

fc1/10 is where host is connected
MDS9134-SJ# show run int fc1/10
version 3.2(1a)

interface fc1/10
port-license acquire
switchport mode F
no shutdown

They both have to be in same vsan else

NPIV upstream not availabe error might be there.

sh npv internal errors
192) Event:E_DEBUG, length:186, at 683624 usecs after Fri Apr 25 16:17:04 2008
[102] npivp_mts_hdlr_fwd_internal_flogi_update(1165): Unable to match the fw
d response for internal FLOGI with any of the outstanding responses, ignoring th
e resp, error: fu unknown error

193) Event:E_DEBUG, length:136, at 652218 usecs after Fri Apr 25 16:17:04 2008
[112] E(1,fc1/1) Upstream Port VSAN(30) for this interface is different from
the local port VSAN(1)Failing this external interface: fc1/1

MDS9134-SJ# show npv flogi-table
--------------------------------------------------------------------------------
SERVER EXTERNAL
INTERFACE VSAN FCID PORT NAME NODE NAME INTERFAC
E
--------------------------------------------------------------------------------
fc1/10 1 0x0a0400 21:00:00:e0:8b:0b:38:0e 20:00:00:e0:8b:0b:38:0e fc1/1
Total number of flogi = 1.

MDS9134-SJ# show npv status

npiv is enabled

External Interfaces:
====================
Interface: fc1/1, VSAN: 1, FCID: 0x0a000d, State: Up

Number of External Interfaces: 1

Server Interfaces:
==================
Interface: fc1/2, State: Pre-Initialized
Interface: fc1/10, VSAN: 1, State: Up

Number of Server Interfaces: 2


Eari

Earlier Error

MDS9134-SJ(config-if)# do show int fc1/10
fc1/10 is down (NPV upstream port not available)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:0a:00:0d:ec:51:05:40
Admin port mode is F
snmp link state traps are enabled
Port vsan is 30

Ve
Becasuse upstream port fc1/1 is in vsan 1 and fc1/10 is vsan 30..?

What if upstream port in vsan 30 but remote switch with npiv does
not have vsan 30.

MDS9134-SJ# show npv sta

npiv is enabled

External Interfaces:
====================
Interface: fc1/1, State: Failed(Mismatch in VSAN for this upstream port)

Number of External Interfaces: 1

Server Interfaces:
==================
Interface: fc1/2, State: Waiting for External Interface
Interface: fc1/10, State: Waiting for External Interface

Number of Server Interfaces: 2

Created vsan 30 on 9513
then

MDS9134-SJ# show npv sta

npiv is enabled

External Interfaces:
====================
Interface: fc1/1, VSAN: 30, FCID: 0xa00000, State: Up

Number of External Interfaces: 1

Server Interfaces:
==================
Interface: fc1/2, State: Waiting for External Interface
Interface: fc1/10, VSAN: 30, State: Up

Number of Server Interfaces: 2

ee
No flogi commands are there...

Commands

MDS9134-SJ# show npv internal event- flogi-fsm interface fc1/10


7) FSM: Transition at 922358 usecs after Fri Apr
25 16:50:55 2008
Previous state: [NPIVP_FLOGI_ST_WAIT_ON_FCID_ADD]
Triggered event: [NPIVP_FLOGI_EV_FCID_UPDATE_SUCCESS_RESP]
Next state: [NPIVP_FLOGI_ST_STEADY_STATE]

8) FSM: Transition at 922717 usecs after Fri Apr
25 16:50:55 2008
Previous state: [NPIVP_FLOGI_ST_STEADY_STATE]
Triggered event: [NPIVP_FLOGI_EV_SEND_FLOGI_ACC]
Next state: [FSM_ST_NO_CHANGE]

MDS9134-SJ# show npv internal event- ext-if-fsm int fc1/1\
497) FSM: Transition at 106900 usecs after Fri Apr 25 16:50:55
2008
Previous state: [NPIVP_EXT_IF_ST_WAITING_NS_REGISTRATION]
Triggered event: [NPIVP_EXT_IF_EV_NS_RSNN_RESPONSE_SUCCESSFUL]
Next state: [FSM_ST_NO_CHANGE]

498) FSM: Transition at 110396 usecs after Fri Apr 25 16:50:55
2008
Previous state: [NPIVP_EXT_IF_ST_WAITING_NS_REGISTRATION]
Triggered event: [NPIVP_EXT_IF_EV_NS_RSPN_RESPONSE]
Next state: [FSM_ST_NO_CHANGE]

499) FSM: Transition at 110406 usecs after Fri Apr 25 16:50:55
2008
Previous state: [NPIVP_EXT_IF_ST_WAITING_NS_REGISTRATION]
Triggered event: [NPIVP_EXT_IF_EV_NS_RSPN_RESPONSE_SUCCESSFUL]
Next state: [FSM_ST_NO_CHANGE]

500) FSM: Transition at 110412 usecs after Fri Apr 25 16:50:55
2008
Previous state: [NPIVP_EXT_IF_ST_WAITING_NS_REGISTRATION]
Triggered event: [NPIVP_EXT_IF_EV_VALIDATE_INT_FLOGI_ACC_SUCCESS]
Next state: [NPIVP_EXT_IF_ST_UP]


Curr state: [NPIVP_EXT_IF_ST_UP]



NPIV switch config and commands:

MDS9513-83-SJ(config)# do show run int fc2/2
version 3.2(2c)

interface fc2/2
no shutdown
switchport mode F


MDS9513-83-SJ(config)# do show int fc2/2
fc2/2 is up
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:42:00:0d:ec:2c:54:c0
Admin port mode is F
snmp link state traps are enabled
Port mode is F, FCID is 0xa00000
Port vsan is 30
Speed is 4 Gbps
Rate mode is dedicated
Transmit B2B Credit is 16
Receive B2B Credit is 16
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 128 bits/sec, 16 bytes/sec, 0 frames/sec
5 minutes output rate 104 bits/sec, 13 bytes/sec, 0 frames/sec
231 frames input, 22980 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
257 frames output, 19556 bytes
0 discards, 0 errors
5 input OLS, 5 LRR, 5 NOS, 0 loop inits

MDS9513-83-SJ(config)# do show flogi database interface fc2/2
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc2/2 30 0xa00000 20:01:00:0d:ec:51:05:40 20:1e:00:0d:ec:51:05:41
fc2/2 30 0xa00100 21:00:00:e0:8b:0b:38:0e 20:00:00:e0:8b:0b:38:0e
[Win_HBA0]

MDS9513-83-SJ# show fcns database npv NOde_wwn 20:1e:00:0d:ec:51:05:41

VSAN 30:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0xa00100 N 21:00:00:e0:8b:0b:38:0e (Qlogic) scsi-fcp:init
[Win_HBA0]

Total number of entries = 1



MDS9513-83-SJ# show flogi database details
---------------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME FLAGS
---------------------------------------------------------------------------------------
fc2/1 5 0xbd0000 21:00:00:e0:8b:08:dd:22 20:00:00:e0:8b:08:dd:22 ADOP
fc2/2 30 0xa00000 20:01:00:0d:ec:51:05:40 20:1e:00:0d:ec:51:05:41
fc2/2 30 0xa00100 21:00:00:e0:8b:0b:38:0e 20:00:00:e0:8b:0b:38:0e AVDO
[Win_HBA0]
fc2/8 1 0x0a000c 50:06:01:68:88:02:90:ce 50:06:01:60:11:02:90:ce
fc2/10 777 0x490300 21:00:00:d0:b2:00:82:c0 20:00:00:d0:b2:00:82:c0 ADOP
fc2/11 777 0x490200 21:02:00:d0:b2:00:82:c0 20:02:00:d0:b2:00:82:c0 ADOP
fv4/1/1 1 0x0a0000 26:02:00:0d:ec:2c:54:c2 26:0b:00:0d:ec:2c:54:c2 P
fv4/1/2 1 0x0a0001 26:03:00:0d:ec:2c:54:c2 26:0c:00:0d:ec:2c:54:c2 P
fv4/2/1 1 0x0a0002 26:04:00:0d:ec:2c:54:c2 26:0d:00:0d:ec:2c:54:c2 P
fv4/2/2 1 0x0a0009 25:02:00:0d:ec:2c:54:c2 25:03:00:0d:ec:2c:54:c2 P
fv4/3/1 1 0x0a0003 26:05:00:0d:ec:2c:54:c2 26:0e:00:0d:ec:2c:54:c2 P
fv4/4/1 1 0x0a0004 26:06:00:0d:ec:2c:54:c2 26:0f:00:0d:ec:2c:54:c2 P
fv4/5/1 1 0x0a0005 26:07:00:0d:ec:2c:54:c2 26:10:00:0d:ec:2c:54:c2 P
fv4/6/1 1 0x0a0006 26:08:00:0d:ec:2c:54:c2 26:11:00:0d:ec:2c:54:c2 P
fv4/7/1 1 0x0a0007 26:09:00:0d:ec:2c:54:c2 26:12:00:0d:ec:2c:54:c2 P
fv4/8/1 1 0x0a0008 26:0a:00:0d:ec:2c:54:c2 26:13:00:0d:ec:2c:54:c2 P

Total number of flogi = 16.
FLAGS:
A area FCID allocation
L loop device
V FDISC
D the wwn matches the default OUI list
O the wwn matches the configured OUI list
P allocation was done based on the persistency table

--------------

MDS9513-83-SJ# show run | include npi
npiv enable


MDS9513-83-SJ# show fcns database npv

VSAN 30:
-------------------------------------------------------------------------------
NPV NODE-NAME NPV IP_ADDR NPV IF CORE SWITCH WWN CORE IF
-------------------------------------------------------------------------------
20:1e:00:0d:ec:51:05:41 172.16.33.23 fc1/1 20:00:00:0d:ec:2c:54:c0 fc2/2

172.16.33.23 is 9134 switch.
which has npv enabled

MDS9513-83-SJ# show fcns database npv de
------------------------------------------------------------
VSAN:30 NPV Node Name: 20:1e:00:0d:ec:51:05:41
------------------------------------------------------------
NPV Fabric Port-WWN :20:01:00:0d:ec:51:05:40
class :2,3
NPV IP Address :172.16.33.23
ipa :ff ff ff ff ff ff ff ff
fc4-types:fc4_features :npv
NPV Switch Name:Interface :MDS9134-SJ:fc1/1
port-type :NP
Core Switch fabric-port-wwn :20:42:00:0d:ec:2c:54:c0
permanent-port-wwn (vendor) :20:01:00:0d:ec:51:05:40 (Cisco)


Total number of entries = 1
======================================================================

NPIV

Cisco SAN-OS release 3.0(1) supports the industry-standard N-port
identifier virtualization (NPIV), which allows a single Fibre Channel
HBA port to be assigned multiple Fibre Channel IDs. Under virtual
operating environments such as VMware, NPIV enables access control,
zoning, and port security to be configured for each virtual machine.

N Port virtualization (NPV) reduces the number of Fibre Channel domain IDs in SANs. Switches operating in the NPV mode do not join a fabric; rather, they pass traffic between NPV core switch links and end devices, which eliminates the domain IDs for these edge switches.




While NPV is similar to N port identifier virtualization (NPIV), it does not offer exactly the same functionality. NPIV provides a means to assign multiple FC IDs to a single N port, and allows multiple applications on the N port to use different identifiers. NPIV also allows access control, zoning, and port security to be implemented at the application level. NPV makes use of NPIV to get multiple FCIDs allocated from the core switch on the NP port.



NP Ports

An NP port (proxy N port) is a port on a device that is in NPV mode and connected to the NPV core switch using an F port. NP ports behave like N ports except that in addition to providing N port behavior, they also function as proxies for multiple, physical N ports.
NP Links

An NP link is basically an NPIV uplink to a specific end device. NP links are established when the uplink to the NPV core switch comes up; the links are terminated when the uplink goes down. Once the uplink is established, the NPV switch performs an internal FLOGI to the NPV core switch, and then (if the FLOGI is successful) registers itself with the NPV core switch's name server.

When an NP port comes up, the NPV device first logs itself in to the NPV core switch and sends a FLOGI request that includes the following parameters:

•The fWWN (fabric port WWN) of the NP port used as the pWWN in the internal login.

•The VSAN-based sWWN (switch WWN) of the NPV device used as nWWN (node WWN) in the internal FLOGI.

After completing its FLOGI request, the NPV device registers itself with the fabric name server using the following additional parameters:

•Switch name and interface name (for example, fc1/4) of the NP port is embedded in the symbolic port name in the name server registration of the NPV device itself.

•The IP address of the NPV device is registered as the IP address in the name server registration of the NPV device.

Note The BB_SCN of internal FLOGIs on NP ports is always set to zero. The BB_SCN is supported at the F-port of the NPV device.

lthough fWWN-based zoning is supported for NPV devices, it is not recommended because:

•Zoning is not enforced at the NPV device (rather, it is enforced on the NPV core switch).

•Multiple devices behind an NPV device log in via the same F port on the core (hence, they use same fWWN and cannot be separated into different zones).

•The same device might log in using different fWWNs on the core switch (depending on the NPV link it uses) and may need to be zoned using different fWWNs.

When you enable NPV, your system configuration is erased and the system is rebooted with NPV mode enabled

On the 91x4 platform, before you upgrade to 3.2(2b) or downgrade from 3.2(2b), shut the F-ports connected to NPIV capable hosts, and then disable the NPIV feature. After the upgrade or downgrade is complete, enable the NPIV feature and up the F-ports.

switch(config)# npiv enable
switch(config)# interface fc2/1
switch(config-if)# switchport mode F
switch(config-if)# no shutdown
Configure the NPIV core switch port as an F port.
Changes Admin status to bring up the interfaces
switch(config)# npv enable
Enables NPV mode on a NPV device (module, Cisco MDS 9124 or Cisco MDS 9134 Fabric Switch). The module or switch is rebooted, and when it comes back up, is in NPV mode.
Note A write-erase is performed during the reboot.
switch(config)# interface fc1/1

switch(config-if)# switchport mode NP
switch(config-if)# no shutdown
On the NPV device, select the interfaces that will be connected to the aggregator switch and configure them as NP ports.

By grouping devices into different NPV sessions based on VSANs, it is possible to support multiple VSANs at the NPV-enabled switch. The correct uplink must be selected based on the VSAN(s) that the uplink can carry.


Issues I have seen with NPIV

- HP Virtual Connect does not load balance between the ports, the frames
might get lost, if there are multiple connections.

- Few Bugs on Cisco side,
CSCsk96105

Symptom: If you upgrade to Cisco SAN-OS Release 3.2(2c) from a lower version, or downgrade from Cisco SAN-OS Release 3.2(2c) to a lower version on an MDS 9124 switch, MDS 9134 switch, Cisco Fabric Switch for HP c-Class BladeSystem, or a Cisco Fabric Switch for IBM BladeCenter, zoning may not work as configured for the F ports connected to NPIV-capable hosts.

Workaround: This issue is resolved.
CSCsk00953

Symptom: HP Blade Servers that are connected through an HP Virtual Connect (VC) FC module to a Cisco Fabric Switch for HP c-Class BladeSystem using NPIV lose access to LUNs when load balancing on the VC module is switched from 16:1 to 8:1. When the load balancing ratio is 16:1, all servers connect through interface ext1. When the ratio is 8:1, servers 1 and 3 connect through ext1, servers 2 and 4 connect through ext2, and so on. Servers on ext2 are not affected by the switchover. In addition, packets might get dropped when the switchover occurs.

When more than 255 hosts logged into that vsan, there may be issue, if all the ports Qlogic ports.

As guessed each pwwn(of qlogic hba) belongs to auto-area-oui list and it consumes entire area. After 255 hosts are brought up in vsan 2 all 255 areas of domain 0x35 in vsan 2 are used and so the fcid allocation fails.

From fcdomain P2.log.txt:
grep "ENTIRE AREA" tt | grep " 2 " | wc -l
254

Workaround as suggested in Jerome's case:
MDS9216I-86-SJ# show fcid company-id-from-wwn 50:06:0b:00:00:c2:62:10
Extracted oui: 0x0060B0
MDS9216I-86-SJ# config t
MDS9216I-86-SJ(config)# no fcid-allocation area company-id 0x0060B0
shut all the hosts or host one by one to clear that area id.
purge fcdomain fcid vsan 2

Pending topics to add

Case notes:
NPIV
SME
Port stuck in initializing mode
hardware issues - PLOGI getting Lost
IVR and interop
Install /upgrade issues.
Zone Activation issues, Zone merge issues.
Fabric manager and Performance Manager
Ficon Configs.
Mismatch of Interop and IVR virtual domain add
SSM issues.
IVR Service Groups
Best Practices for Zone.

IPSEC

Basic FCIP setup:
(14+2 card)
interface GigabitEthernet9/2
no shutdown
ip address 10.10.10.2 255.255.255.0
fcip profile 1
ip address 10.10.10.2
interface fcip1
no shutdown
no channel-group auto
use-profile 1
peer-info ipaddr 10.10.10.1

On 9216i:
interface fcip1
no shutdown
use-profile 1
peer-info ipaddr 10.10.10.2
fcip profile 1
ip address 10.10.10.1
interface GigabitEthernet1/16 (1/2 in newer code)
ip address 10.10.10.1 255.255.255.0
no shutdown

iZone merge failed

Looked zone on vsan 1 in 9509A:
MDS9509-B1# show zoneset active v 1
zoneset name zs1 vsan 1
zone name zone1 vsan 1
attribute qos priority high
pwwn 10:10:10:10:10:10:10:10

zone name test2 vsan 1
interface fc1/3 swwn 20:00:00:05:30:00:24:1e

zone name chip vsan 1
interface fc1/2 swwn 20:00:00:05:30:00:24:1e
interface fc1/5 swwn 20:00:00:05:30:00:24:1e

So enable qos on 9216i and reshut/no shut fcip1 and
zones merged fine.

---------

StepII security:

9509B:
MDS9509-B1(config)# crypto ike enable
MDS9509-B1(config)# crypto ike domain ipsec
MDS9509-B1(config-ike-ipsec)# initiator version 1 address 10.0.0.1
MDS9509-B1(config-ike-ipsec)# key cisco address 10.0.0.1
MDS9509-B1(config)# crypto ike domain ipsec
MDS9509-B1(config)# crypto ipsec enable
MDS9509-B1(config)# ip access-list acl1 permit ip 10.10.10.0 0.0.0.255 10.10.10.0 0.0.0.255
MDS9509-B1(config)# crypto transform-set domain ipsec aes-xcbc esp-aes 128 esp-aes-xcbc-mac
MDS9509-B1(config)# crypto transform-set domain ipsec 3des-md5 esp-3des esp-md5-hmac
MDS9509-B1(config)# crypto map domain ipsec cm1 1
MDS9509-B1(config-(crypto-map-ip))# set peer 10.10.10.1
MDS9509-B1(config-(crypto-map-ip))# match address acl1
MDS9509-B1(config-(crypto-map-ip))# set transform-set aes-xcbc 3des-md5
MDS9509-B1(config-(crypto-map-ip))# exit
MDS9509-B1(config)# interface gigabitethernet 9/2
MDS9509-B1(config-if)# crypto map domain ipsec cm1


Did same on 9216i except that peer 10.10.10.2 applied to interface gi 1/16 ( newer
SANOS code treats same interface gige 1/2)
-------------------------------

show commands:
MDS9216i# show crypto sad domain ipsec
interface: GigabitEthernet1/16
Crypto map tag: cm1, local addr. 10.10.10.1
protected network:
local ident (addr/mask): (10.10.10.0/255.255.255.0)
remote ident (addr/mask): (10.10.10.0/255.255.255.0)
current_peer: 10.10.10.2
local crypto endpt.: 10.10.10.1, remote crypto endpt.: 10.10.10.2
mode: tunnel, crypto algo: esp-aes-128-cbc, auth algo: esp-aes-xcbc-mac
current outbound spi: 0x4827c082 (1210564738), index: 16
lifetimes in seconds:: 3600
lifetimes in bytes:: 4718592000
current inbound spi: 0x90c7011 (151810065), index: 16
lifetimes in seconds:: 3600
lifetimes in bytes:: 4718592000

MDS9509-B1# show crypto sad domain ipsec interface gigabitethernet 9/2
interface: GigabitEthernet9/2
Crypto map tag: cm1, local addr. 10.10.10.2
protected network:
local ident (addr/mask): (10.10.10.0/255.255.255.0)
remote ident (addr/mask): (10.10.10.0/255.255.255.0)
current_peer: 10.10.10.1
local crypto endpt.: 10.10.10.2, remote crypto endpt.: 10.10.10.1
mode: tunnel, crypto algo: esp-aes 128, auth algo: esp-aes-xcbc-mac
current outbound spi: 0x90c7011 (151810065), index: 128
lifetimes in seconds:: 3600
lifetimes in bytes:: 4718592000
current inbound spi: 0x4827c082 (1210564738), index: 129
lifetimes in seconds:: 3600
lifetimes in bytes:: 4718592000
---------
MDS9216i(config-ike-ipsec)# key cisco address ?
Peer IP address

MDS9216i(config-ike-ipsec)# key cisco address 10.10.10.2
MDS9216i(config-ike-ipsec)# policy 10
MDS9216i(config-ike-ipsec-policy)# exit
-------
MDS9509-B1(config)# crypto ike domain ipsec
MDS9509-B1(config-ike-ipsec)# key cisco address 10.10.10.1
MDS9509-B1(config-ike-ipsec)# policy 10
MDS9509-B1(config-ike-ipsec-policy)# exit
MDS9216i# show crypto ike domain ipsec initiator
initiator address 10.10.10.2 mode 0
MDS9509-B1# show crypto ike domain ipsec initiator
initiator version 1 address 10.0.0.1
initiator version 1 address 10.10.10.1
MDS9216i# show crypto ike domain ipsec sa
Tunn Local Addr Remote Addr Encr Hash Auth Method Lifetime
-------------------------------------------------------------------------------
41 10.10.10.1[500] 10.10.10.2[500] 3des sha preshared key 3600
Tunn Local Addr Remote Addr Encr Hash Auth Method Lifetime
-------------------------------------------------------------------------------
9 10.10.10.2[500] 10.10.10.1[500] 3des sha1 preshared key 3600
MDS9509-B1#
MDS9509-B1# crypto ike domain ipsec rekey sa 9
CMI request failed (Rekeying not supported for an IKEv1 tunnel)

MDS9216i# crypto ike domain ipsec rekey sa 41
CMI request failed (not supported)
----------

Compression/WA/TA
MDS9509-B1(config)# interface fcip 1
MDS9509-B1(config-if)# write-accelerator
MDS9509-B1(config-if)# write-accelerator tape-accelerator
MDS9509-B1(config-if)# ip-compression ?
mode1 Fast compression for high bandwidth links
mode2 Moderate compression for medium bandwidth links
mode3 High compression for low bandwidth links
MDS9509-B1(config-if)# ip-compression mode1
MDS9509-B1(config-if)# shut
MDS9509-B1(config-if)# no shut
MDS9509-B1(config-if)# exit
MDS9509-B1# show int fcip1
fcip1 is down (Link failure or not-connected)
rite acceleration mode is on
Tape acceleration mode is on
Tape Accelerator flow control buffer size is automatic
IP Compression is enabled and set for mode1

And other switch:
show interface fcip1
fcip1 is down (Link failure or not-connected)
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:0c:a9:00
Admin port mode is auto, trunk mode is on
-----------

So disable write/tape acce/ip compression

MDS9509-B1# show interface fcip1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 22:15:00:05:30:00:24:5e
Peer port WWN is 20:14:00:0d:ec:0c:a9:00
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,20,100-102,105,200,500)
Trunk vsans (operational) (1)
Trunk vsans (up) ()
Trunk vsans (isolated) (20,100-102,105,200,500)
Trunk vsans (initializing) (1)
Using Profile id 1 (interface GigabitEthernet9/2)
Peer Information
Peer Internet address is 10.10.10.1 and port is 3225
FCIP tunnel is protected by IPSec
i----------------
vsan is 1
Using Profile id 1 (interface GigabitEthernet1/16)
Peer Information
Peer Internet address is 10.10.10.2 and port is 3225
Write acceleration mode is on
Tape acceleration mode is on
Tape Accelerator flow control buffer is 256 KBytes
IP Compression is enabled and set for higher through put
IP Compression is enabled and set for higher compression ratio
9216i older version of code, does not have mode1/mode 2
MDS9216i(config-if)# ip-compression ?
high-comp-ratio Higher ratio slower compression
high-throughput Lower ratio faster compression
------
upgrading to same version
--

after the upgrade 9216i gige interface became gigethernet 1/2 instead
of gig 1/16. Eventhough 10.10.10.1 was not in running config, if I try to
give gig 1/2 10.10.10.1 , it failed saying that config was already given.

So I did write erase and recopied config to running config. copy r s.

then applied crypto and ip to gig 1/2.

------interface GigabitEthernet1/2
no shutdown
ip address 10.10.10.1 255.255.255.0
crypto map domain ipsec cm1

i-----


MDS9216i# show int fcip 1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:0c:a9:00
Peer port WWN is 22:15:00:05:30:00:24:5e
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1)
Trunk vsans (operational) (1)
Trunk vsans (up) (1)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet1/2)
Peer Information
Peer Internet address is 10.10.10.2 and port is 3225
FCIP tunnel is protected by IPSec
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is automatic
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
kickstart: version 2.0(1) [build 2.0(0.200)]
system: version 2.0(1) [build 2.0(0.200)]
----

MDS9509-B1# show int fcip 1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 22:15:00:05:30:00:24:5e
Peer port WWN is 20:14:00:0d:ec:0c:a9:00
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,20,100-102,105,200,500)
Trunk vsans (operational) (1)
Trunk vsans (up) (1)
Trunk vsans (isolated) (20,100-102,105,200,500)
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet9/2)
Peer Information
Peer Internet address is 10.10.10.1 and port is 3225
FCIP tunnel is protected by IPSec
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is automatic
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
------------

Look at FFCIP tunnel is protected by IPSec

-----

MDS9509-B1(config)# interface fcip 1
MDS9509-B1(config-if)# ip-compression mode1
MDS9216i# show int fcip 1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:0c:a9:00
Peer port WWN is 22:15:00:05:30:00:24:5e
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1)
Trunk vsans (operational) (1)
Trunk vsans (up) ()
Trunk vsans (isolated) ()
Trunk vsans (initializing) (1)
Using Profile id 1 (interface GigabitEthernet1/2)
Peer Information
Peer Internet address is 10.10.10.2 and port is 3225
FCIP tunnel is protected by IPSec
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is automatic
IP Compression is enabled and set for mode1
MDS9509-B1# show interface fcip 1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 22:15:00:05:30:00:24:5e
Peer port WWN is 20:14:00:0d:ec:0c:a9:00
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,20,100-102,105,200,500)
Trunk vsans (operational) (1)
Trunk vsans (up) (1)
Trunk vsans (isolated) (20,100-102,105,200,500)
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet9/2)
Peer Information
Peer Internet address is 10.10.10.1 and port is 3225
FCIP tunnel is protected by IPSec
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is automatic
IP Compression is enabled and set for mode1

----------------------

FCIP secure and compression on:
9216i:
ip access-list acl1 permit ip 10.10.10.0 0.0.0.255 10.10.10.0 0.0.0.255
fcip enable
crypto ike enable
crypto ike domain ipsec
policy 10
key cisco address 10.10.10.2
fcip profile 1
ip address 10.10.10.1

crypto ipsec enable
crypto transform-set domain ipsec 3des-md5 esp-3des esp-md5-hmac
crypto transform-set domain ipsec aes-xcbc esp-aes 128 esp-aes-xcbc-mac

crypto map domain ipsec cm1 1
set peer 10.10.10.2
match address acl1
set transform-set aes-xcbc 3des-md5

interface fcip1
no shutdown
use-profile 1
peer-info ipaddr 10.10.10.2
ip-compression 1
interface GigabitEthernet1/2
no shutdown
ip address 10.10.10.1 255.255.255.0
crypto map domain ipsec cm1

------------

9509A:
ip access-list acl1 permit ip 10.10.10.0 0.0.0.255 10.10.10.0 0.0.0.255
fcip enable
crypto ike enable
crypto ike domain ipsec
policy 1
policy 10
key cisco address 10.0.0.1
key cisco address 10.10.10.1
initiator version 1 address 10.0.0.1
initiator version 1 address 10.10.10.1
fcip profile 1
ip address 10.10.10.2

crypto ipsec enable
crypto transform-set domain ipsec 3des-md5 esp-3des esp-md5-hmac
crypto transform-set domain ipsec aes-xcbc esp-aes 128 esp-aes-xcbc-mac

crypto map domain ipsec cm1 1
set peer 10.10.10.1
match address acl1
set transform-set aes-xcbc 3des-md5

interface fcip1
no shutdown
no channel-group auto
use-profile 1
peer-info ipaddr 10.10.10.1
ip-compression 1


interface GigabitEthernet9/2
no shutdown
ip address 10.10.10.2 255.255.255.0
crypto map domain ipsec cm1


==================================

San extension Tunner

To check the thro'put via fcip pipe.
MDS9509-A1(config)# san-ext-tuner enable
MDS9509-A1# san-ext-tuner
MDS9509-A1(san-ext)#
MDS9509-A1(san-ext)# nwwn 10:00:00:0d:3f:2c:11:22
MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1# show flogi database
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/5 100 0x660300 21:00:00:e0:8b:08:f6:18 20:00:00:e0:8b:08:f6:18
iscsi4/1 100 0x660003 10:00:00:0d:3f:2c:11:21 10:00:00:0d:3f:2c:11:22

Total number of flogi = 2.
-----
without iscsi , I got this error
MDS9509-B1(san-ext)# nport pwwn 21:08:00:d0:b2:00:82:c0 vsan 100 interface gigabitethernet 2/1
Error: flogi for the virtual nport failed(0x40be0009)
i------

SAN EXT is not persistent between the reboots.

iscsi enable
interface iscsi 2/1 enable
MDS9509-B1(config)# san-ext-tuner enable
MDS9509-B1(config)# exit
MDS9509-B1# san-ext-tuner MDS9509-B1(config)# interface iscsi 2/1
MDS9509-B1(config-if)# no shut
MDS9509-B1(config-if)# exit
MDS9509-B1(san-ext)# nwwn 20:00:00:d0:b2:00:82:d0
MDS9509-B1(san-ext)# nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 interface gigabitethernet 2/1
MDS9509-B1# show flogi database
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/5 100 0x4e0001 21:00:00:d0:b2:00:82:c0 20:00:00:d0:b2:00:82:c0
[xio-hab0]
iscsi2/1 100 0x4e0004 20:08:00:d0:b2:00:82:d0 20:00:00:d0:b2:00:82:d0

Total number of flogi = 2.

-----
testing

Zone:

MDS9509-B1# show zoneset active vsan 100
zoneset name ZoneSet1 vsan 100
zone name Zone1 vsan 100
attribute qos priority high
pwwn 21:00:00:e0:8b:0b:fc:0d [dell6450]
* fcid 0x4e0001 [pwwn 21:00:00:d0:b2:00:82:c0] [xio-hab0]
* fcid 0x660300 [pwwn 21:00:00:e0:8b:08:f6:18]

zone name SanExtVirtualHosts vsan 100
* fcid 0x660003 [pwwn 10:00:00:0d:3f:2c:11:21]
* fcid 0x4e0004 [pwwn 20:08:00:d0:b2:00:82:d0]
MDS9509-B1# show fcns database

VSAN 100:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x4e0001 N 21:00:00:d0:b2:00:82:c0 scsi-fcp:both
[xio-hab0]
0x4e0004 N 20:08:00:d0:b2:00:82:d0 scsi-fcp
0x660003 N 10:00:00:0d:3f:2c:11:21 scsi-fcp
0x660300 N 21:00:00:e0:8b:08:f6:18 (Qlogic) ipfc scsi-fcp:init

Total number of entries = 4

MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1(san-ext-nport)# write command-id 100 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1# show san-ext-tuner nports
----------------------------------------------------------------------------
Interface NODE NAME PORT NAME VSAN
----------------------------------------------------------------------------
GigabitEthernet4/1 10:00:00:0d:3f:2c:11:22 10:00:00:0d:3f:2c:11:21 100
MDS9509-B1# show san-ext-tuner nports
----------------------------------------------------------------------------
Interface NODE NAME PORT NAME VSAN
----------------------------------------------------------------------------
GigabitEthernet2/1 20:00:00:d0:b2:00:82:d0 20:08:00:d0:b2:00:82:d0 100
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 230267
Reads : 66%
Writes : 33%
Egress throughput : 96.30 MBs/sec (Max - 205.18 MBs/sec)
Ingress throughput : 176.91 MBs/sec (Max - 177.43 MBs/sec)
Average response time : Read - 651 us, Write - 1297 us
Minimum response time : Read - 480 us, Write - 335 us
Maximum response time : Read - 990 us, Write - 1764 us
Errors : 0
MDS9509-B1# show san-ext-tuner interface gigabitethernet 2/1 nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 counters
Statistics for nport
Node name 20:00:00:d0:b2:00:82:d0 Port name 20:08:00:d0:b2:00:82:d0
I/Os per sec : 229946
Reads : 66%
Writes : 33%
Egress throughput : 176.65 MBs/sec (Max - 177.78 MBs/sec)
Ingress throughput : 96.17 MBs/sec (Max - 205.18 MBs/sec)
Average response time : Read - 1 us, Write - 656 us
Minimum response time : Read - 1 us, Write - 113 us
Maximum response time : Read - 7 us, Write - 1012 us
Errors : 0
MDS9509-A1(san-ext-nport)# write command-id 103 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 104 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 206454
Reads : 39%
Writes : 60%
Egress throughput : 142.51 MBs/sec (Max - 205.18 MBs/sec)
Ingress throughput : 105.52 MBs/sec (Max - 177.43 MBs/sec)
Average response time : Read - 1212 us, Write - 2417 us
Minimum response time : Read - 480 us, Write - 335 us
Maximum response time : Read - 1395 us, Write - 2561 us
Errors : 0


----

removed write/tape and ip compression .....shut/no shut the gige, fcip cameup
but sanext config went away , so recreated it again.

MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1(san-ext-nport)# write command-id 103 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 104 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 100 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# read command-id 101 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# read command-id 102 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# exit
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 205242
Reads : 57%
Writes : 42%
Egress throughput : 105.85 MBs/sec (Max - 204.80 MBs/sec)
Ingress throughput : 138.77 MBs/sec (Max - 139.05 MBs/sec)
Average response time : Read - 1708 us, Write - 3399 us
Minimum response time : Read - 893 us, Write - 354 us
Maximum response time : Read - 2199 us, Write - 4178 us
Errors : 0
----

Let me enable the write/tape and ip compression
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 203654
Reads : 57%
Writes : 42%
Egress throughput : 105.05 MBs/sec (Max - 105.30 MBs/sec)
Ingress throughput : 137.69 MBs/sec (Max - 213.75 MBs/sec)
Average response time : Read - 1721 us, Write - 3425 us
Minimum response time : Read - 259 us, Write - 2017 us
Maximum response time : Read - 2304 us, Write - 4240 us
Errors : 0

MDS9509-B1# show san-ext-tuner interface gigabitethernet 2/1 nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 counters
Statistics for nport
Node name 20:00:00:d0:b2:00:82:d0 Port name 20:08:00:d0:b2:00:82:d0
I/Os per sec : 203896
Reads : 57%
Writes : 42%
Egress throughput : 137.87 MBs/sec (Max - 213.85 MBs/sec)
Ingress throughput : 105.15 MBs/sec (Max - 105.42 MBs/sec)
Average response time : Read - 1 us, Write - 1720 us
Minimum response time : Read - 1 us, Write - 993 us
Maximum response time : Read - 14 us, Write - 2475 us
Errors : 0
-------------------------------
no compression
MDS9509-B1# config t
Enter configuration commands, one per line. End with CNTL/Z.
MDS9509-B1(config)# interface fcip 2
MDS9509-B1(config-if)# no write-accelerator
MDS9509-B1(config-if)# no write-accelerator tape-accelerator
MDS9509-B1(config-if)# no ip-compression
MDS9509-A1(config)# interface fcip 2
MDS9509-A1(config-if)# no write-accelerator
MDS9509-A1(config-if)# no write-accelerator tape-accelerator
MDS9509-A1(config-if)# no ip-compression
MDS9509-B1(config)# interface gigabitethernet 2/1
MDS9509-B1(config-if)# shut
MDS9509-B1(config-if)# no shut
MDS9509-A1(config-if)# interface gigabitethernet 4/1
MDS9509-A1(config-if)# shut
MDS9509-A1(config-if)# no shutdown
MDS9509-B1# show interface fcip2
fcip2 is trunking
Hardware is GigabitEthernet
Port WWN is 20:42:00:05:30:00:24:5e
Peer port WWN is 20:c2:00:05:30:00:24:1e
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,100)
Trunk vsans (operational) (1,100)
Trunk vsans (up) (1,100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet2/1)
Peer Information
Peer Internet address is 10.1.1.1 and port is 3225
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is 256 KBytes
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
Time Stamp is disabled
MDS9509-A1# show interface fcip2
fcip2 is trunking
Hardware is GigabitEthernet
Port WWN is 20:c2:00:05:30:00:24:1e
Peer port WWN is 20:42:00:05:30:00:24:5e
Admin port mode is auto, trunk mode is on
Port mode is TE
vsan is 1
Trunk vsans (allowed active) (1,100)
Trunk vsans (operational) (1,100)
Trunk vsans (up) (1,100)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet4/1)
Peer Information
Peer Internet address is 10.1.1.2 and port is 3225
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is 256 KBytes
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
Time Stamp is disabled
------
MDS9509-B1# san-ext-tuner
MDS9509-B1(san-ext)# nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 interface gigabitethernet 2/1
MDS9509-B1(san-ext-nport)# exit
MDS9509-A1(san-ext)# nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 interface gigabitethernet 4/1
MDS9509-A1(san-ext-nport)# read command-id 102 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# read command-id 101 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 100 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 104 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# write command-id 103 target 20:08:00:d0:b2:00:82:d0 transfer-size 1024 outstanding-ios 100 continuous
MDS9509-A1(san-ext-nport)# exit
MDS9509-A1# show san-ext-tuner interface gigabitethernet 4/1 nport pwwn 10:00:00:0d:3f:2c:11:21 vsan 100 counters
Statistics for nport
Node name 10:00:00:0d:3f:2c:11:22 Port name 10:00:00:0d:3f:2c:11:21
I/Os per sec : 210123
Reads : 57%
Writes : 42%
Egress throughput : 108.37 MBs/sec (Max - 108.46 MBs/sec)
Ingress throughput : 142.07 MBs/sec (Max - 213.73 MBs/sec)
Average response time : Read - 1668 us, Write - 3320 us
Minimum response time : Read - 197 us, Write - 1725 us
Maximum response time : Read - 2214 us, Write - 4109 us
Errors : 0
MDS9509-B1# show san-ext-tuner interface gigabitethernet 2/1 nport pwwn 20:08:00:d0:b2:00:82:d0 vsan 100 counters
Statistics for nport
Node name 20:00:00:d0:b2:00:82:d0 Port name 20:08:00:d0:b2:00:82:d0
I/Os per sec : 210175
Reads : 57%
Writes : 42%
Egress throughput : 142.08 MBs/sec (Max - 213.73 MBs/sec)
Ingress throughput : 108.42 MBs/sec (Max - 108.63 MBs/sec)
Average response time : Read - 1 us, Write - 1668 us
Minimum response time : Read - 1 us, Write - 815 us
Maximum response time : Read - 13 us, Write - 2205 us
Errors : 0

TACACS Config:

Config on MDS is very simple

tacacs+ enable
tacacs-server timeout 4
tacacs-server host 171.69.89.198 key 7 fewhg
aaa group server tacacs+ secteam
server 171.69.89.198

aaa authentication login default group secteam


MDS9216i# show user-acc sanremote
user:sanremote
expires on Wed Oct 12 23:59:59 2005
roles:network-admin
account created through REMOTE authentication
Local login not possible













Iscsi Via CHAP authentication:

VRRP/ipfc and mgmt on MDS

When you have to access remote MDS via fcip link or fc link, ie., access
FM via that IPFc, here is a design for you!

SAmple Config:


DS9120-A1# show vrrp
Interface VR Status
-------------------------------------------------------
mgmt0 1 backup
MDS9120-A1# show run int mgmt 0

interface mgmt0
ip address 172.16.33.82 255.255.255.128
switchport speed 100
vrrp 1
address 172.16.33.82
address 172.16.33.120 secondary
no shutdown


MDS9216i# show run int mgmt 0
version 2.1(1)

interface mgmt0
ip address 172.16.33.86 255.255.255.128
vrrp 1
address 172.16.33.86
address 172.16.33.120 secondary
no shutdown

MDS9216i# show vrrp
Interface VR Status
-------------------------------------------------------
mgmt0 1 master

the problem is uou have to vrrp address same as that mgmt 0.. It has
been confusing for me.. how does the remote switch know the ip address
of other switche's vrrp enabled address ?
Problem Statement:

a. Customer wants to use ISL link to FM/DM related management in a
Two Switch
Scenario, when one mgmt interface is messed up.

b. also the wants minimum traffic
to ISL and confirmed that no host or storage will talk to the storage or
host on the
remote switch. (localization).

Design:

a. both 9216 will have console access for better recovery.
-configure vrrp on the mgmt interface. , this is to resolve
problem with two static entries for same network. As
static route entries load balances , if there are two paths,
and it does not know when when one path fails.
So we configure only one static entry with vrrp interface 172.16.33.120and
vrrp will take care of routing thro' active path).
( for eg, configuration of 9506 with mgmt of 172.16.33.79
MDS9506-B1-sup1(config)# interface mgmt 0
MDS9506-B1-sup1(config-if)# vrrp 1
MDS9506-B1-sup1(config-if-vrrp)# address 172.16.33.79
MDS9506-B1-sup1(config-if-vrrp)# address 172.16.33.120 secondary
MDS9506-B1-sup1(config-if-vrrp)# no shutdown
MDS9506-B1-sup1# show vrrp

Interface VR Status
-------------------------------------------------------

mgmt0 1 master
and do the same thing for other switch with mgmt ip address 172.16.33.77
MDS9509-B1-sup1(config)# interface mgmt 0
MDS9509-B1-sup1(config-if)# vrrp 1
MDS9509-B1-sup1(config-if-vrrp)# address 172.16.33.77
MDS9509-B1-sup1(config-if-vrrp)# address 172.16.33.120 secondary
MDS9509-B1-sup1(config-if-vrrp)# no shutdown
MDS9509-B1-sup1# show vrrp

Interface VR Status
-------------------------------------------------------

mgmt0 1 backup

So even if mgmt 0 172.16.33.79 goes down, the vrrp 172.16.33.120 will route
thro' 172.16.33.77, so we need only static route path on host or the
router. )
- create ISL (TE port ) between 9216 in vsan 1
- configure ip for each 9216's vsan 1. (config t ; interface vsan 1)
- create vsan 10 with interfaces on one switch and empty vsan 20 on
the same switch.
- create empty vsan 10 on second switch and vsan 20 with interfaces on the
second switch.
- create second default routers with a different metric using vsan 1's ip
addresses.
- configure zoning from a single switch/ you can do either full zoneset to
propagate aliases
as well as non-active zonesets from that switch.
- you can do copy merge or leave it as it is on the second switch.
- if first 9216's mgmt fails, you can login to 9216 cli and connect to other
switche's vsan ip
and correct the problem. or thro' console.
- or configure static route to vsan's network to go via active mgmt (one
of the mgmt should be
up) on the mgmt wkstation and run fabric manager .


Also note that , even without VSAN 1 interface being configured for network,
you can
do zone editing etc thro' normal ISL because it uses FC-CT from one
switch.
Let me know if this suffice your requirements.

FCIP commands

show ips stats buffer interface gig 1/1 --- look free clusters
show ips stats tcp interface gig 1/1 det --- look for SACK/retrans
show ips stats dma inter gig 1/1 --- look for timestamp errors
show int gig 1/1 ----- mtu
show int fcip 1 ----- retransmits and other config cwm/retrans times

attach mod 1
show port internal port-control
show port internal link-events
exit
show tech internal link-events

show ips status module 1
show ips stats hw-comp int gig 1/1
show ips stats ip int gig 1/1 ----> reassembly/packets/dropped
show ips stats mac int gig 1/1
-----CRC erros -hw related stuff
extended ping with DF (y)
-
show ips internal eth-trace-logs gigabitethernet 1/1 (on NWMDS02 and
APMDS01)
- show ips internal eth-trace-logs gigabitethernet 1/2 (on NWMDS01 and
APMDS01)
-----------------------------

Internal Commands: don't give to customer

system core tftp://ip/
ips core dump full

Problem: show cores shows sibyte crashing and not able to
run any fcip commands.

show ips status --- said port1/1 failed (gig1/1 down)
show fcip-lock -- had a lock
show ips internal fcip-trace-log


show break-lock (feww times and show fcip-lock did not have any
locks)
show port internal event fcip showed SW-failed.

ips reset module 1 port 1 ( in ips it will reset in pairs, etherchannel
reason - same memory). ---- use with CARE.. very disruptive.

we could not get fcip link up - link failure error, so
and then killed and restarted ips manager,

show system internal mts buffer (attac)
debug ips fcip error port 1 (attach)
attach mod 1 port 1

show system int mts buffers sap 60 (Csceg 82721)
/open window - start capture
attach mod 1
debug ips fcip write-acc-err port 1
//open another window and start capture
attach mod 2
debug ips fcip write-acc-err port 2

And do the same on the other switch.

show int fcip X counters
show fcip target-tape-session
show fcip sum
show fcip host-tape-session 30
show fcip target-map
show fcip host-map

term len 0
show port internal even errors
show port internal even interface fcip 3
show port internal info global
show port internal info interface fcip 3
show hard internal errors all
show hard internal sup-fc0 errstats
show ips internal even epp interface fcip 3
show ips stats dma interface gig 4/1
show ips stats tcp interface gig 4/1 details
show ips stats buffer interface gig 4/1
show tech internal modul 4
from both switches in addition to
show tech details
show tech fcip

FCIP!

Before I get more details

Note this:

- retransmit failure, is because IP network not able to handle the fc traffic
( change CWM burstsize, reduce it tcp cwm burst 10) to so that when there is
fc traffic, it is not that bursty, reducing cwm might cause less traffic
to be sent in bursts when there is increase in fctraffic)

- if you see a lot B2b credit starvation on FC ports, (for eg, ports
which use fcip link, for eg, fc port where storage array is connected,
which replicates to remote storage array), increase send-buffer-size
in fcip profile.

- compression between various different cards might be different
ips-8, 18+4,etc.

- write accelaration/tape accelaration and ivr, please look for transit
vsan and might break fcip WA/TA because of equal cost paths available
via IVR., anyway fcip TA certainly has issues with multiple equal cost paths.

# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
b#wm Enable congestion window monitoring
keepalive-timeout Set keep alive timeout in sec
max-bandwidth-kbps Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps Configure maximum available path bandwidth in Mbps
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisecond
pmtu-enable Enable PMTU Discovery
sack-enable Enable SACK option for TCP
send-buffer-size Send buffer size in KBytes

The CWM parameter : The default value is 10K and should be left untouched under normal conditions. CWM is a way of controlling burstiness after long idle times or loss of Acks. CWM stands for congestion window monitoring .

The keepalive-timeout is the TCP keepalive timeout value and is set to 60 sec. by default . The configurable values range between 1 and 7200 sec.

The max- and min-bandwidth parameter programs the TCP Maximum Window Size (scaling factor) and engages an internal "shaper" functionality . These values should be carefully chosen and requires understanding of intermediate network's end-to-end topology .The default values are to be changed according to the aforementioned requirements.The Round-trip-time can be derived once you have your FCIP tunnel up and running as follows :

bison# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
bison#

Always add an additional margin of a few Microseconds to this value as a minium.

The max-retransmissions counter is set to 4 by default - in a healthy network environment this value should be left unchanged.

The max-retransmission timer is set to 200msec - If you experience extreme high retransmission counters this value might be increased -but in general this would not be required unless the RTT is above the 200msec value .

The PMTU (path mtu discovery) is enabled by default - best practice is to know which is the maximum MTU size supported by all interfaces along the logical path between both peers . Refer to RFC1191 for more details .

The SACK feature (selective acknowledgment) is not enabled by default - it could be considered when you have a lot of retransmissions going on between the two peers - SACK will allow selective retransmissions of your window - which is beneficial if larger maximum window sizes are configured and retransmissions are experienced frequently . In our sample config we have enabled it - when you do that make sure it is enabled at either side of the link .

The send-buffer-size is the amount of buffers in addition to the TCP window we allow to be transmitted out before we start to flow control the FC sources.The default value is set to 0 .









Configuration from MDS9216
c# sh run

Building Configuration ...
fcip profile 200
ip address 200.200.200.1
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10

fcip profile 201
ip address 200.200.200.5
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10

!.....the TCP parameters are identical to what we had configured on the peering FCIP interfaces , only in very specific cases we should consider different values , e.g. if the return-path(s) are running across a different part of the
interface fcip1
channel-group 2 force
no shutdown
use-profile 200
peer-info ipaddr 100.100.100.1


interface fcip2
channel-group 2 force
no shutdown
use-profile 201
peer-info ipaddr 100.100.100.5

!..both fcip1 and fcip2 are bound to the same Channel-group 2 - also note that we have no strict relationship between profile-id and fcip interface numbering here as this is not a requirement. However , from a management and troubleshooting perspective a "strict" relationship of both values is recommended...

FCIP Generic ( outputs needed to troubleshoot)

· show interface gig - Displays status of the relevant gig interface bound to the FCIP profile

· show ips stats tcp int gig details- Displays TCP stats and active connections for the relevant gig interface

. show ips stats dma-bridge int gig x/y - displays timestamp errors
. show ips stats buffer int gig x/y --- any buffer relted issue
- slow fcip connections if buffer is less than 70K
- show int fcip counters -- and show int fcip and show ips stats hw-comp
- hw compression /compression ratio , WA stats, TA stats

· show ips arp int gig - Dispalys all arp entries for the relevant gig interface , next hop or peer should be present in this list

· show ips ip route int gig - displays the specific routes going across the relevant gig interface

· show interface fcip - Displays the fcip interface status and all details related to this fcip tunnel

· show profile fcip - Displays ip address the profile is bound to and all configured TCP parameters

· show interface port - Displays the specified Port-channel number's information

· show int fcip counters - verify here if there are any frames going through the FCIP tunnel

· show fcdomain vsan - Lists all domain related details - verify here if the fabric is formed cross the fcip tunnel(s)

· show fcns da vsan - Displays all pwwn , FC4-Types and FCID's of the relevant vsan - verify here that all expected entries are distributed across the fcip tunnel(s)

· show

http://www-tac.cisco.com/Teams/SAN/Bru/IPS8_elaborate.htm

show ips - tcp stats/fcip
show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
9506# show ips internal fcip-trace-log

ips measure-rtt ip-address



---------------

show int fcip X counters ( any WA issues like ABTS)
show ips stats tcp/dma

If FC traffic is bursty, you may want to increase sendbuffer size
max 8M ( 3.0 it is 16K), so that FC will dump the frames on to
this buffer, then fcip can process the frames as per bw availability.

This will reduce back filling fc causing lack of b2b credits

If RTT time and TCP window ( how many bytes within RTT) might cause
timestamp errors, if the send buffer is not cleared within fcdrop
latency ( 500 ms)

If RTT is 40 ms and TCP window ( show ips stats) and show int fcip
will give tcp window., is 1256 K (1.2)and if send buffer is 8Mb,

then 8MB send buffer will be cleared with approx 40ms x 8/1.2 =320 ms

but if RTT is 80 ms, then it might take about 640 ms, that means some
frames might get dropped (dma-bridge timestamp errors)

CWM changes the burstiness of TCP side, default 50K means 50K frames
sent without ACK at a burst, this might clean up fcip traffic or
send buffer, but might cause TCP retrans if network can't handle it.

- small send buffer will cause lack of b2b credit and congestion
on FC side
- bigger send buffer will cause time stamperrors if the buffer
is not cleared within fc drop latency
- small cwm might cause time stamp errors because fc frames are
queued (etherenet send queue) and might stay longer in fc/switch
- higher cwm might cause bursty tcp traffic and might lead to
higher retrans.
- higher b2b credit might cause filling up of send buffer quickly.

so good luck tunning fcip params.!

Look for any hardware errors on fcip blade as well.


Requested Send buffer - configured params ( tcp send-buffer)
Allocated send buffer - configured + tcp window
8000+ 1257 = 9257 (for eg.)

fcip profile ---- max/min bw is post compression.
so show int fcip can have more thro'put than max bw configured in fcip
profile
show int gig 's thro'put is deteermined by max/min bw configured on fcip
profile.

RSPAN!

Caveats:
- trunking interface needed.
- ip routing needs to be enabled
on switches
- fc-tunnel needs to be enabled.
- vsan interface in same subnet in all three switches.
( participating switches)

Switch 1: (intermediate switch).
fc-tunnel enable
interface vsan100
ip address 10.1.1.75 255.255.255.0
no shutdown

ip routing
-----

SWitch II : ( rspan desitnation)
fc-tunnel enable
fc-tunnel tunnel-id-map 100 interface fc1/12
interface fc1/12
switchport mode SD
switchport speed 2000
connect DS_PAA here

interface vsan100
ip address 10.1.1.81 255.255.255.0
no shutdown
ip routing

Switch III: Rspan Source
fc-tunnel enable
interface fc-tunnel 100
destination 10.1.1.81
source 10.1.1.82
explicit-path rspan
no shutdown

fc-tunnel explicit-path rspan
next-address 10.1.1.75 strict

interface vsan100
ip address 10.1.1.82 255.255.255.0
no shutdown

ip routing

interface fc1/8^M
switchport mode ST
switchport speed 2000
rspan-tunnel interface fc-tunnel 100
no shut
----

span session 1^M
destination interface fc-tunnel 100
source interface fc1/10 rx
source interface fc1/10 tx

----

Testing:

show int fc-tunnel 100 ( it should be up, if
it wating for RESV, it means either path
is not reachabled, fspf cost is messed up
( if up and down cost for specific path is
different on vsan 100, or ST is not up or
SD is not configured ).

fc-tunnel 100 is up^M
Dest IP Addr: 10.1.1.81 Tunnel ID: 100^M
Source IP Addr: 10.1.1.82 LSP ID: 1^M
Explicit Path Name: rspan^M
Outgoing interface: port-channel 2^M
Outgoing Label(s) to Insert: 10008:0:1:ff'h ^M
Record Routes:^M
10.1.1.75 ^M
10.1.1.81 ^M
^M

iscsi

Proxy Iscsi Initiator

* pcit


* One time Config
* Config for each initiator
* CSM mapping
* show iscsi commands
* isid
* ethereal trace
* FC trace of two iscsi sessions with proxy initiator and Xiotech target
* FC trace for two iscsi sessions with different MTU
* proxy iscsi initiator in multiple vsans

Benefits:

- simple zoning ( no need to configure all the hosts wwns)
- simple lun mapping at the storage.
- it is in certain way like SN5428 , where we configure storage lun mapping for the internal
HBAs and there is no individual wwns for each initiator.
MDS Config:

One time Config:
MDS9509-B1-sup1(config)# iscsi interface-vsan-member-enable

by default proxy wwn goes to vsan 1 to change that, we need to do these steps.
MDS9509-B1-sup1(config)# vsan 40 interface iscsi 4/1
( where my targets are).

interface iscsi 4/1
switchport proxy-initiator ( if you want you can configure wwns manual too!)

show interface iscsi 4/1
Proxy Initiator Mode : enabled
nWWN is 20:15:00:05:30:00:24:60 (system-assigned)
pWWN is 20:16:00:05:30:00:24:60 (system-assigned)

Add this pwwn to Zoning/ and configure storage Lun mapping

for CSM:
MDS9509-B1-sup1(svc)# show cluster tacCluster host proxy-iscsi
Host proxy-iscsi:
Number of port is 2
Port WWN is 20:16:00:05:30:00:24:60
LUN 0 : vdisk piscsi1
LUN 1 : vdisk piscsi2
LUN 2 : vdisk piscsi3
LUN 3 : vdisk piscsi4
LUN 4 : vdisk piscsi5
LUN 5 : vdisk piscsi6
LUN 6 : vdisk piscsi7
LUN 7 : vdisk piscsi8
-----------

Configuration for each host (this config on MDS host - configuring two virtual targets
(multipathing) for ip 172.69.122.104)

You can not do lunzoning/zoning based on ip address with proxy initiator.(even after setting
switch initiator id ip-address on iscsi port.( verify CSCed82704 ))
All the subsequent initiators will have access to same luns/storages as the first initiator.
In the MDS 1.3.3 config guide, under the sections "Configuring iSCSI proxy initiators"
(Configuring iSCSI/Configuring IP Storage), it is clearly mentioned that when in proxy
initiator mode, you cannot use iSCSI attributes in the FC access control mechanisms.
You have to use iSCSI based access control to accomplish the same.


(still needs to verify above).

ip route 171.69.122.104 255.255.255.255 interface gig 4/1

iscsi virtual-target name csm
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit
iscsi virtual-target name csm2
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0000 iscsi-lun 0x0000
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0001 iscsi-lun 0x0001
advertise interface GigabitEthernet4/1
initiator ip address 171.69.122.104 permit <<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator

Linux host config on MDS:
ip route 172.69.122.104 255.255.255.255 interface GigabitEthernet4/1
iscsi virtual-target name csm-linux
pWWN 20:09:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator

iscsi virtual-target name csm-linux2
pWWN 20:04:00:05:30:00:24:60 fc-lun 0x0003 iscsi-lun 0x0000
advertise interface GigabitEthernet4/1
initiator ip address 171.69.104.104 permit<<<<<<<<<<<<<<<<, this is only option of initiator based access
control in proxy iscsi initiator



----------

on the iscsi PC host , I could see two luns for each targets.

VSAN 40:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x790000 N 20:04:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790001 N 20:09:00:05:30:00:24:60 (Cisco) scsi-fcp:target svc
0x790005 N 20:16:00:05:30:00:24:60 (Cisco) scsi-fcp:init isc..w (only one initiator)
0x790100 N 21:00:00:e0:8b:0b:fc:0d (QLogic) scsi-fcp:init
0x790200 N 21:00:00:e0:8b:08:f6:18 (QLogic) ipfc scsi-fcp:init
0x790300 N 21:01:00:e0:8b:28:f6:18 (QLogic) ipfc scsi-fcp:init

---
MDS9509-B1-sup1# show iscsi session
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
Session #1
Target csm
VSAN 40, ISID 400001370018, Status active, no reservation

Session #2
Target csm2
VSAN 40, ISID 400001370019, Status active, no reservation

Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1
Target csm-linux
VSAN 40, ISID 801234567800, Status active, no reservation

Session #2
Target csm-linux2
VSAN 40, ISID 801234567801, Status active, no reservation

MDS9509-B1-sup1# show iscsi initiator
iSCSI Node name is iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.104
iSCSI alias name:
Node WWN is 20:02:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005

iSCSI Node name is dhcp-173-228
Initiator ip addr (s): 171.69.104.104
iSCSI alias name:
Node WWN is 20:00:00:05:30:00:24:60 (dynamic)
Member of vsans: 1
Number of Virtual n_ports: 1
Virtual Port WWN is 20:16:00:05:30:00:24:60 (shared)
Virtual Node WWN is 20:15:00:05:30:00:24:60 (shared)
Interface iSCSI 4/1, Portal group tag: 0x180
VSAN ID 40, FCID 0x790005


On the storage , let us see which session is logged (proxy or the actual iscsi host)- it is proxy.
MDS9509-B1-sup1# show svc session svc 2/1 (we don't see 20:02 ....60 wwn of actual iscsi host
being logged on to the storage)
svc2/1:
Target N-port WWN is 20:04:00:05:30:00:24:60, vsan is 40, FCID is 0x790000
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:01:00:05:30:00:24:60, vsan is 30, FCID is 0x780000
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:05:00:05:30:00:24:60, vsan is 50, FCID is 0xd40000
pWWN 20:14:00:05:30:00:24:60, nWWN 20:0f:00:05:30:00:24:60, FCID 0xd40001

MDS9509-B1-sup1# show svc session svc 2/2
svc2/2:
Target N-port WWN is 20:09:00:05:30:00:24:60, vsan is 40, FCID is 0x790001
pWWN 21:00:00:e0:8b:08:f6:18, nWWN 20:00:00:e0:8b:08:f6:18, FCID 0x790200
pWWN 21:01:00:e0:8b:28:f6:18, nWWN 20:01:00:e0:8b:28:f6:18, FCID 0x790300
pWWN 21:00:00:e0:8b:0b:fc:0d, nWWN 20:00:00:e0:8b:0b:86:0e, FCID 0x790100
pWWN 20:16:00:05:30:00:24:60, nWWN 20:15:00:05:30:00:24:60, FCID 0x790005
Initiator N-port WWN is 20:08:00:05:30:00:24:60, vsan is 30, FCID is 0x780002
pWWN 50:06:04:82:c3:a1:2f:52, nWWN 50:06:04:82:c3:a1:2f:52, FCID 0x780001
Mgmt N-port WWN is 20:14:00:05:30:00:24:60, vsan is 50, FCID is 0xd40001
pWWN 20:05:00:05:30:00:24:60, nWWN 20:0e:00:05:30:00:24:60, FCID 0xd40000

------
debug ips iscsi flow
(with only pc iscsi connection)
Debugs:
MDS9509-B1-sup1# Dec 15 12:18:51 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:18:51 ips: Created initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Initiator(8) got nwwn 2002000530002460
Dec 15 12:18:51 ips: Initiator(8) got vsan list
Dec 15 12:18:51 ips: no:1 vsan_id 1
Dec 15 12:18:51 ips: Created an fc_port(7) pgt 384 iscsi-if-index 0x0b180000 intf 0x02180000 ip-addr: 172.16.34.10 for initiator(8)
Dec 15 12:18:51 ips: Created session(39) target name isid 400001370016 for initiator(8)
Dec 15 12:18:51 ips: fc_port(7) has a pwwn 0, mode: 1
Dec 15 12:18:51 ips: Put iscsi4/1 in vsan 40 status: 0
Dec 15 12:18:51 ips: Fc_port(7) pwwn 2016000530002460 member of 1 vsans registered 0
Dec 15 12:18:51 ips: fc_port(7) sent 1 flogi requests
Dec 15 12:18:51 ips: Flogi response: fc_port(7) fcid 00790005 in vsan 40
Dec 15 12:18:51 ips: fc_port(7) pwwn 2016000530002460 sent 1 NS reg requests
Dec 15 12:18:51 ips: NS reg resp: fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 fcid 00790005 vsan 40
Dec 15 12:18:51 ips: Discovery session.. no need to check target
Dec 15 12:18:51 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370016]
Dec 15 12:18:51 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:18:51 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:18:51 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:18:51 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:18:51 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:18:51 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:18:51 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:18:51 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:18:51 ips: Removing session(39) tgt-name: isid: 400001370016 failure code: 1
Dec 15 12:19:12 ips: Node 2016000530002460, vsan 40 is not discovered as init or target
Dec 15 12:19:12 ips: Initiator(8) iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com processing tgt_online 2016000530002460 vsan 40

Refresh on MS initiator

Dec 15 12:19:54 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target
Dec 15 12:19:54 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:19:54 ips: Created session(40) target name isid 400001370017 for initiator(8)
Dec 15 12:19:54 ips: Discovery session.. no need to check target
Dec 15 12:19:54 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[] isid:[400001370017]
Dec 15 12:19:54 ips: Get targets for init node iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com if_index 0x2180000 vrrp 0
Dec 15 12:19:54 ips: Querying NS for targets for fc-port nwwn 2015000530002460 pwwn 2016000530002460
Dec 15 12:19:54 ips: Querying NS for undiscovered node for fc-port nwwn 2002000530002460 pwwn 2016000530002460, wait_count 1
Dec 15 12:19:54 ips: NS Tgts response for iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num entries 2 wait-count 1
Dec 15 12:19:54 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:19:54 ips: Get targets response for init iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com num-targets 2
Dec 15 12:19:54 ips: Session Destroy node-name: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com tgt-name:
Dec 15 12:19:54 ips: Fc_port(7) nwwn 2015000530002460 pwwn 2016000530002460 cleaning session
Dec 15 12:19:54 ips: Removing session(40) tgt-name: isid: 400001370017 failure code:
1
Logon CSM:
Dec 15 12:20:36 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm
Dec 15 12:20:36 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:20:36 ips: Created session(41) target name csm isid 400001370018 for initiator(8)
Dec 15 12:20:36 ips: Target csm a virtual target checking access
Dec 15 12:20:36 ips: Node csm is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:20:36 ips: fc_port(7) Querying NS for target pwwn:[2004000530002460] sec pwwn:[0] wait 1
Dec 15 12:20:36 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790000
Dec 15 12:20:36 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm] isid:[400001370018]

Logon CSM2
Dec 15 12:21:18 ips: Session Create init: iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com, ip addr: 171.69.122.104, target csm2
Dec 15 12:21:18 ips: Fc-port(7) pwwn 2016000530002460 pgt 384 iscsi-if-index 0b180000 intf 02180000
Dec 15 12:21:18 ips: Created session(42) target name csm2 isid 400001370019 for initiator(8)
Dec 15 12:21:18 ips: Target csm2 a virtual target checking access
Dec 15 12:21:18 ips: Node csm2 is allowed to be advertised to if_index 0x2180000, initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Dec 15 12:21:18 ips: fc_port(7) Querying NS for target pwwn:[2009000530002460] sec pwwn:[0] wait 1
Dec 15 12:21:18 ips: Got NS tgt response fc_port(7) sid 00790005 vsan 40 did 00790001
Dec 15 12:21:18 ips: Sending Session Create Response for init_name:[iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com] target_name:[csm2] isid:[400001370019]
--------

Etherreal Trace from Linux:
Linux host 171.69.104.104 to proxy iscsi initiator using unh_iscsi with two virtual-targets to two CSM nodes
(one each) and these virtual targets have been mapped to one iscsi lun 0 (fc-lun 3).

configured unh iscsi conf : initiator dhcp-173-228 ,
target =csm-linux and csm-linux2 and the ipaddress
of target 172.16.34.10.

cat /proc/scsi/scsi showed two disks which are same.
[root@dhcp-173-228 root]# cat /proc/scsi/scsi
(only iscsi devices displayed)
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: 2062 Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04


etherreal trace

MDS9509-B1-sup1# show iscsi session detail
Initiator iqn.1991-05.com.microsoft:jejoseph-w2k15.cisco.com
Initiator ip addr (s): 171.69.122.48
Session #1 (index 4)
Target xiotech
VSAN 40, ISID 400001370004, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 20104, MaxCmdSN 20119, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 2
Stats:
PDU: Command: 933, Response: 933
Bytes: TX: 20158288, RX: 19995648
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.122.48
CID 1, State: Full-Feature
StatSN 937, ExpStatSN 0
MaxRecvDSLength 65536, our_MaxRecvDSLength 1024
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: Yes

Initiator dhcp-173-228
Initiator ip addr (s): 171.69.104.104
Session #1 (index 2)
Target xiotech-linux
VSAN 40, ISID 801234567800, TSIH 384, Status active, no reservation
Type Normal, ExpCmdSN 24429, MaxCmdSN 24443, Barrier 0
MaxBurstSize 0, MaxConn 1, DataPDUInOrder Yes
DataSeqInOrder Yes, InitialR2T Yes, ImmediateData No
Registered LUN 0, Mapped LUN 1
Stats:
PDU: Command: 2207, Response: 2206
Bytes: TX: 128568, RX: 202783744
Number of connection: 1
Connection #1
Local IP address: 172.16.34.10, Peer IP address: 171.69.104.104
CID 0, State: Full-Feature
StatSN 2209, ExpStatSN 0
MaxRecvDSLength 1392, our_MaxRecvDSLength 1392
CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
Version Min: 0, Max: 0
FC target: Up, Reorder PDU: No, Marker send: No (int 0)
Received MaxRecvDSLen key: No

Just for recap: here are the virtual targets defined.
target: xiotech
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 2
iSCSI LUN: 0x0000, FC LUN: 0x0000
iSCSI LUN: 0x0001, FC LUN: 0x0001
No. of initiators permitted: 1
initiator 171.69.122.48/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled

target: xiotech-linux
* Port WWN 21:06:00:d0:b2:00:82:c0
Configured node
No. of LU mapping: 1
iSCSI LUN: 0x0000, FC LUN: 0x0002
No. of initiators permitted: 1
initiator 171.69.104.104/32 is permitted
all initiator permit is disabled
trespass support is disabled
revert to primary support is disabled

MDS9509-B1-sup1# show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
Connection Stats
0 active openings, 109 accepts
0 failed attempts, 0 reset received, 109 established
Segment stats
4564268 received, 1961372 sent, 1133 retransmitted
43 bad segments received, 0 reset sent

TCP Active Connections
Local Address Remote Address State Send-Q Recv-Q
172.16.34.10:3260 171.69.122.48:1593 ESTABLISH 0 0
172.16.34.10:3260 171.69.104.104:32779 ESTABLISH 0 0
0.0.0.0:3260 0.0.0.0:0 LISTEN 0 0

Traces are in this directory

Here is the trace snapshots of proxy_linux1_win2_logoff.

a. no iscsi sessions were logged on before the taking the trace.
b. started /etc/init.d/unh_iscsi start ( Linux has fc-lun 2 mapped to iscsi-lun 0)
- you see prli from the proxy initiator
- lun inquiry proxied for Linux.
prli_linux_inquiry

3. mount /dev/sdb1 /xiotech1 and deleted some files in /xiotech1
4. Using microsoft initiator, I connect to virtual target xiotech , u will see microsoft's inquiry, no new plogi or prli session
initiated. (probably if PDU of this session is lower, then we might reinitiate , so that PMTU Is reneogiated.

windows_inquiry

and finally I removed windows session and then the linux session, so you will prlo as last iscsi session is cleared up.

prlo

ip payload size is 1460. (+ 20 byte TCP Options from ethereal trace )
MSS in PC is 1460 bytes
iscsi payload is 1440 ( iscsi header is 48 bytes)
FC data size is 1392

Example II:

win iscsi initiator logs in first with default MTU size (MSS 1460)
linux iscsi initiator logs in second with mtu size of 800.
We expect LOGO And PRLO to happen because proxy initiator relogs to target with lower Receive data field Size.

here is the picture with PLOGI when the win2k initiator comes in. Note the Class 3 receive data field size.
win2k_prli

after a little bit, here comes the linux session with lower MTU ( 800)
(setting up Linux mtu)
at> ifconfig eth0 mtu 800
at> ifconfig eth0 down
at> ifconfig eth0 up
at> route add default gw 171.69.104.1

second


iSCSI initiator 20.1.2.12 will be in VSAN 40, 41, 50 and 51. Not in VSAN 30. All initiators without "iscsi initiator" command or without vsan command will be in VSAN 30.

iscsi initiator ip-address 20.1.2.12
vsan 40
vsan 41
vsan 50
vsan 51

interface iscsi3/3
switchport initiator id ip-address
switchport proxy-initiator nWWN 11:11:11:11:11:11:11:00 pWWN 11:11:11:11:11:11:11:11

vsan database
vsan 30 interface iscsi 3/3

IVR concepts and some troubleshooting!

IVR TEST SETUP ( i will put some pictures)

show run of the switches:
9216
9509
Steps:
9216 Config Steps
9509 Config Steps

When you two different IVR fabric, even if you link one
VSAN between them (even if it is non-ivr), the ivr
zonesets will merge.

Also please make sure you use ivr distribute option
to keep the configs same in all ivr edge switches.

IVR NAT
I. connected ISL between 9216 and 9509 and had both
Xiotech and Dell PC in same VSAN and checked the Qlogic
SANSurfer to check if I am seeing the Xiotech Disks.
( set Xiotech to new servers can see all the luns
and Zoning to permit (note that it may be disruptive
when creating ivr zone)

II. moved interface fc1/8 on 9509 (Xiotech) to vsan 26
moved interface fc1/2 on 9216 (PC) to vsan 24
created empty vsan 23 on both switches.
default zone permit on vsan 26 and 24. (this is test,
so no problem if activating ivr zone is disruptive).

III. created ivr topology database and activated it.

IV. created ivr zoneset and activated it.
V. SAN BLADE on PC
VI. IVR Topology: show ivr internal top
ZoneStatus on the switches
VII. Debug IVR trace
VIII. show ivr (fsm ) commands are good to debug.

Links: Config Example in CCO
Uses of IVR:
IVR between brocade and MDS ( how to prevent ISL failure due to read-only zones ) -Dallas McCloon's Webpage.
IVR between SN5428 and MDS ( read-only zones and ivr virtual-fcdomain-add ) - Paul's Webpage.

IVR troubleshooting:

- please check if IVR topology is activated show ivr vsan-topology
- show ivr zoneset active ( * on all the ports)
- show zoneset active ( check ivr zone names are there and * active) ( show run).
if not do show ivr zone status ( check for failures) - mostly remove default permit to
valid zone/zonesets.
show ivr internal event-history error-log

------
Case # 600921106
Let us assume you have four switches connected via vsan 1000

switch 1 and 2 ---> common vsans 200,220,240,1000 : Fabric 1
switch 3 and 4 ---> common vsans 300,320,340,1000 : Fabric 2
switch1,2,3,4 - common vsan 1000

if you want to configure IVR to share resources among vsans (220,240 in Fabric I
and then you want to share 300,320 for vsan 2)

Ivr topology would be (same on all 4 switches)
Transist vsan will be 1000. ( without this, even though
fullzoneset may have appropriate zones/zonesets via FM, without
activating zoneset locally , you will not have active zoneset
with ivr zoneset, with transist vsan when you activate IVR ZS from
one switch).

autonomous fabric 1 switch-wwn switch1 vsan-ranges 220,240,1000
( max vsans you could enter
will be 5, use gui if need more or use vsan 220-240,1000)
autonomous fabric 1 switch-wwn switch2 vsan-ranges 220,240,1000
autonomous fabric 1 switch-wwn switch3 vsan-ranges 300,320,1000
autonomous fabric 1 switch-wwn switch4 vsan-ranges 300,320,1000

(if you type no autonomous fabric 1 xxxx , you will see it show run
until you activate the topology).

( GUI you should see 16 entries in active topology. if you have
4 switches)

and then activate topology.

Ivr zones and zonesets will be same across all 4 switches (FM will take care of
it)

ivr_zone1 - pwwnhost vsan 220, pwwnstorage vsan 240
ivr_zone2- pwwnhost vsan 300, pwwnstorage vsan 320
ivr_Zoneset- ivr_zone1+ivr_zone2
and activate the zoneset

If ivr zoneset activation (show ivr zoneset status ), stuck in activating or
as per show log logfile (waiting for lowest wwn), then it may be a bug in
1.3.4a CSCeh02256. At that point, only way to get out is to deactivate
ivr zoneset (disruptive if ivr zones are used) and then activate it.

It might be due to not having correct IVR topology, correct the ivr
topology and activate it again.

However to avoid this bug ( best is to have correct topology), but
you can use this work around:

From the list I have ( show ivr and show wwn switch will
give switch's wwn)

ivr vsan-topology database
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:0f:4f:80
vsan-ranges 220,230,240,270,290,1000
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:0f:2e:00
vsan-ranges 220,230,240,270,290,1000
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:0f:2d:c0
vsan-ranges 320,330,340,370,390,1000
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:ec:0f:50:40 vsan-ranges
320,330,340,370,390,1000

You can verify it one more time.
Lowest wwn is 20:00:00:0d:ec:0f:2d:c0 TACOMA_BD4
20:00:00:0d:ec:0f:2e:00 TUNDRA_AD1
20:00:00:0d:ec:0f:4f:80 TACOMA_AD3
Highest wwn is 20:00:00:0d:ec:0f:50:40 TUNDRA_BD2


Step I:
You can push the ivr zones and zoneset from Fabric manager.
(from IVR zone dialog)

Step II onwards should be done from lowest wwn to highest wwn in that order.

StepII:(verification of the ivr zones to see if they got pushed
from FM correctly)

From CLI of lowest wwn, verify if the zoneset in the config is correct by
show ivr zoneset

or show ivr zones

Step III: (activation)
and then activate it using
config t
ivr zoneset activate name
exit

Step IV ( commands to verify if activation completed in this switch)

show ivr zoneset active
show ivr zoneset status (look for activation sucess on all vsans).

If it is success you can move on to higher wwn switches.


_________________________________________________________________________________

Some more commands:

show ivr internal capa vsa
show ivr internal zone-per-vsan vsan
show ivr internal device

clear ivr zone
clear ivr session

Waiting for lowest wwn
in ivr zoneset activation:

show ivr zoneset status
show fcdomain domain vsan X ( whichever vsan it is waiting)
look at the lowest wwn and check that switch.
---------------------------

if you want to ivr vsan 70 in switch 1 to 240 in switch 2,
you need not have vsan 70 in switch2 as well as 240 in switch 1.

-----

ivr withdraw domain
ivr refresh

similar to reactivating ivr zoneset.
-------
transient vsans will have all the devices.
----

ivr zoneset activation failing with fabric unstable

look at show ivr internal global, see domain id false and
recreate that VSAN that complains about domain id.
show ivr int vdri summary
-----

if you two ivr switches and one of them does not have
virtual domains added to all ivr switches
then when activating ivr zoneset , you will see errors
that certain fcid already there.

if that is the case
show ivr virtual-fcdomain

and add
ivr virtual-domain vsan 5 - enables rdi.
--------

ivr withdraw domain and ivr refresh is useful for
swwn00:00:00 error.

show vsan usage


show ivr int zone-fsm - tells extra info
from show ivr zoneset status

show ivr tells ivr enabled switches.

if a vsan is missing from show ivr zoneset status ,

and shows up in global. then suspend and no suspend
the vsan
--------
Problem Description

a. swwn 00:00 in vsan 5
(due to RDI issue when upgrading to 2.1.1b without following
the right steps)
b. fswb stuck in waiting for lowest wwn due to lower wwn
switch running 2.1.1 and higher wwn switch running 2.0.x code.
c. Qlogic - zone activation failure in vsan 5 due to problem a.

a. we saw extra zone vsan_1_test_zone in vsan 5 ivr config, so
we tried to add it and activate the ivr zoneset, but fswb was
still stuck at waiting for lowest wwn
b. so we disabled ivr, enabled and added vsan topology
and copied ivr zone config to bootflash:dave.txt and
copy bootflash:dave.txt run
c. then activated the ivr zoneset and there were issues
with certain fcids not added because already there,
so to resolve that we did step b. again
d. before activating the zoneset we added
ivr virtual vsan 5 on fswb where it was missing.
e. then we activated zoneset. and show ivr zoneset
status showed all vsans are in active mode.
f. we then tested couple of ivr zoneset changes from
FM and it activated fine.

Problem 2:
1. we did ivr withdraw domain X vsan 5 on domain
with swwn 00:00:00
2. and then did ivr refresh, that domain had correct swwn but someother domain failed.
instead doing it on each domain,
3. we removed vsan 5 from fswa vsan-topology
and added back in.
4. then all the domains had correct swwn (show fcdom dom v 5)
5. show ivr zoneset status did not show vsan 5 , even though
internal vdri summary showed vsan 5 in RDI mode (because
of ivr virtual domain add)
6. we initiated Build Fabric (BF) by fcdomain domain restart v 5
but did not help.
7. we did disruptive suspend and no suspend vsan 5, it fixed the
issue.
8. we looked at qlogic and it was able to see the ns (show ns all) and we were able to activate the zoneset
on vsan 5 without any issue ( fixed problem c.)
We had some issues because proposed changes showed
a lot of zones but we added only one zone, which was fixed
by zone copy active full and then going to FM again

New Problem:

1. when looking at show ivr zoneset status on fswb it
had vsan 5 stuck again in lowest wwn issue.
2. we activated the ivr zoneset from FM and it fixed
the issue.
3. So Dave, please show ivr zoneset status after
you activate normal zoneset in any ivr enabled vsans.
if it is stuck, please add a test ivr zone and activate
ivr zoneset from Fabric manager. it should not happen
now, it happened because of too many changes yesterday.
-------------------------

IVR NAT:

When IVR NAT is enabled, two vsans that need to talk to each other can have
same domain id, PLOGI, show fcns database will show one fcid for local vsan
and other fcid for other vsan (persistent fcid and domain id for ivr nat is other
issue).
`show flogi database`
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/3 100 0x0a00ef 50:06:01:62:30:60:22:a8 50:06:01:60:b0:60:22:a8
fc1/12 200 0x0a0000 10:00:00:00:c9:4b:4f:de 20:00:00:00:c9:4b:4f:de

Note that vsan 100 and 200 domain id is same 10.

VSAN 100:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0a00ef N 50:06:01:62:30:60:22:a8 (Clariion) scsi-fcp
0xc2510c N 10:00:00:00:c9:4b:4f:de (Emulex) ipfc scsi-fcp

Total number of entries = 2

VSAN 200:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0a0000 N 10:00:00:00:c9:4b:4f:de (Emulex) ipfc scsi-fcp
0x858419 N 50:06:01:62:30:60:22:a8 (Clariion) scsi-fcp

ivr enable
ivr distribute
ivr nat
ivr vsan-topology database
autonomous-fabric-id 1 switch-wwn 20:00:00:0d:bc:76:76:80 vsan-ranges 100,200
ivr vsan-topology auto

ivr zone name z_udmstest_4fde_cxtest_spa2
member pwwn 50:06:01:62:30:60:22:a8 vsan 100
member pwwn 10:00:00:00:c9:4b:4f:de vsan 200
ivr zoneset name zs_122405_jmm
member z_udmstest_4fde_cxtest_spa2
ivr zoneset activate name zs_122405_jmm force
ivr commit
-----


If the host and storage needs to talk to each other,
host will get 0x858419 as the fcid which is NATed,
plogi into that address.

vsan 200 --- 0a0000 (host)---plogi --- 0x85419(storage)--(ivr nat) --cont nxt line--

(ivr nat)---0xc2510c(host)---plogi ----0a00ef

ACC will trvel same way ., we hold ACC(plogi) for 2 seconds , to fix some
issue. PLOGI and PLOGI acc can wait for RATOV ( 10 s).

IVR TRACES- NAT from Univ of M
note the time difference between plogi and plogi ACC.

Also IVR NATed FCIDs may be different.
--------

Lab setup
MDS9216I-86# show flogi database
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
fc1/4 1 0x640201 20:00:00:05:ad:22:8e:3c 20:00:00:05:ad:02:8e:3c
fc1/4 1 0x640202 20:01:00:05:ad:22:8e:3c 20:01:00:05:ad:02:8e:3c
fc1/4 1 0x640204 20:02:00:05:ad:22:8e:3c 20:02:00:05:ad:02:8e:3c
fc1/6 2 0x640000 50:06:04:82:c3:a1:2f:52 50:06:04:82:c3:a1:2f:52
fc1/7 1 0x640500 21:00:00:e0:8b:0b:fc:0d 20:00:00:e0:8b:0b:fc:0d

Total number of flogi = 4.
MDS9216I-86# show fcns database

VSAN 1:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x640201 N 20:00:00:05:ad:22:8e:3c scsi-fcp
0x640202 N 20:01:00:05:ad:22:8e:3c scsi-fcp
0x640204 N 20:02:00:05:ad:22:8e:3c scsi-fcp
0x640500 N 21:00:00:e0:8b:0b:fc:0d (Qlogic) scsi-fcp:init

Total number of entries = 3

VSAN 2:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x640000 N 50:06:04:82:c3:a1:2f:52 (EMC) scsi-fcp:target 250

---
Config persistent:
ivr fcdomain database autonomous-fabric-num 1 vsan 1
native-autonomous-fabric-num 1 native-vsan 2 domain 223
pwwn 50:06:04:82:c3:a1:2f:52 fcid 0xdf407a
ivr fcdomain database autonomous-fabric-num 1 vsan 2
native-autonomous-fabric-num 1 native-vsan 1 domain 83
pwwn 20:01:00:05:ad:22:8e:3c fcid 0x533f4f
pwwn 20:02:00:05:ad:22:8e:3c fcid 0x533f5f
pwwn 21:00:00:e0:8b:0b:fc:0d fcid 0x530000

ivr vsan-topology auto
ivr nat
ivr distribute
ivr zone name IVR_Zone1
member pwwn 20:02:00:05:ad:22:8e:3c vsan 1
member pwwn 50:06:04:82:c3:a1:2f:52 vsan 2
member pwwn 20:01:00:05:ad:22:8e:3c vsan 1
ivr zoneset name IVR_ZoneSet1
member IVR_Zone1

----

activate the zoneset and commit
zoneset name IVR_ZoneSet1
zone name IVR_Zone1
* pwwn 20:02:00:05:ad:22:8e:3c vsan 1 autonomous-fabric-id 1
* pwwn 50:06:04:82:c3:a1:2f:52 vsan 2 autonomous-fabric-id 1
* pwwn 20:01:00:05:ad:22:8e:3c vsan 1 autonomous-fabric-id 1
MDS9216I-86# show fcns database

VSAN 1:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x640201 N 20:00:00:05:ad:22:8e:3c scsi-fcp
0x640202 N 20:01:00:05:ad:22:8e:3c scsi-fcp
0x640204 N 20:02:00:05:ad:22:8e:3c scsi-fcp
0xdf407a N 50:06:04:82:c3:a1:2f:52 (EMC) scsi-fcp:target 250
0x640500 N 21:00:00:e0:8b:0b:fc:0d (Qlogic) scsi-fcp:init
Total number of entries = 4

VSAN 2:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x533f4f N 20:01:00:05:ad:22:8e:3c scsi-fcp
0x533f5f N 20:02:00:05:ad:22:8e:3c scsi-fcp
0x640000 N 50:06:04:82:c3:a1:2f:52 (EMC) scsi-fcp:target 250
0x530000 N 21:00:00:e0:8b:0b:fc:0d (Qlogic) scsi-fcp:init
----

Here is how persistent setup needs to be done.
----
MDS9216I-86# show ivr fcdomain database
----------------------------------------------------
AFID Vsan Native-AFID Native-Vsan Virtual-domain
----------------------------------------------------
1 1 1 2 0xdf(223)
1 2 1 1 0x53(83)

Number of Virtual-domain entries: 2

----------------------------------------------------
AFID Vsan Pwwn Virtual-fcid
----------------------------------------------------
1 1 50:06:04:82:c3:a1:2f:52 0xdf407a
1 2 20:01:00:05:ad:22:8e:3c 0x533f4f
1 2 20:02:00:05:ad:22:8e:3c 0x533f5f
1 2 21:00:00:e0:8b:0b:fc:0d 0x530000

Number of Virtual-fcid entries: 3
MDS9216I-86# show ivr int pnat vdom-info
IVR2 PNAT: Virtual domain info for 1:1:223
--------------------------------------------------
is_owner=true, owner_dom=100, local_dom=100
ID: VDOM-1:1:223
Peer domain list: 100
Response pending list:
IVR2 PNAT: Virtual domain info for 1:2:83
--------------------------------------------------
is_owner=true, owner_dom=100, local_dom=100
ID: VDOM-1:2:83
Peer domain list: 100
Response pending list:

-----
fcnalyser trace will show plogi and prli.

MDS9216I-86(config)# fcanalyzer local br limit-captured-frames 0

8.134606 64.05.00 -> df.40.7a 0x43c0 0xffff FC ELS PLOGI <<<<<<<
2 second delay to pass it on to next VSAN (added in 2.1.2b)
10.139095 53.00.00 -> 64.00.00 0x43c0 0xffff FC ELS PLOGI
10.139793 ff.ff.fc -> 64.00.00 0x8027 0x5e9 dNS ACC (GNN_ID)
10.148506 64.00.00 -> 53.00.00 0x43c0 0x0 FC ELS ACC (PLOGI)
10.148953 df.40.7a -> 64.05.00 0x43c0 0x0 FC ELS ACC (PLOGI
10.149091 64.05.00 -> df.40.7a 0x43c0 0xffff FC ELS PRLI
10.149479 53.00.00 -> 64.00.00 0x43c0 0xffff FC ELS PRLI
10.151872 64.00.00 -> ff.ff.fc 0x8028 0xffff dNS GPN_ID
10.152399 ff.ff.fc -> 64.00.00 0x8028 0x5ea dNS ACC (GPN_ID)
10.163429 64.00.00 -> 53.00.00 0x43c0 0xffff FC ELS ACC (PRLI)
10.163791 df.40.7a -> 64.05.00 0x43c0 0xffff FC ELS ACC (PRLI)
10.171180 64.00.00 -> ff.ff.fc 0x8029 0xffff dNS GSNN_NN

show ivr int pnat debug-history
21:44:12:is_sync_done() called - (1, 223)->TRUE
21:44:12:Received ELS_PLOGI from 0xfffc64 to 0xdf407a
21:44:12:Forwarding to (2, 0xfffc64, 0x640000)
21:44:12:Received ELS_ACC from 0x640000 to 0xfffc64
21:44:12:Forwarding to (1, 0xdf407a, 0xfffc64)
21:44:12:Received ELS_PRLI from 0xfffc64 to 0xdf407a
21:44:12:Forwarding to (2, 0xfffc64, 0x640000)
21:44:12:Received ELS_ACC from 0x640000 to 0xfffc64
21:44:12:Forwarding to (1, 0xdf407a, 0xfffc64)
21:44:12:Received ELS_PRLO from 0xfffc64 to 0xdf407a
21:44:12:Forwarding to (2, 0xfffc64, 0x640000)
21:44:12:Received ELS_ACC from 0x640000 to 0xfffc64
21:44:12:Forwarding to (1, 0xdf407a, 0xfffc64)
21:44:12:Received ELS_LOGO from 0xfffc64 to 0xdf407a
21:44:12:Forwarding to (2, 0xfffc64, 0x640000)
21:44:12:Received ELS_ACC from 0x640000 to 0xfffc64
21:44:12:Forwarding to (1, 0xdf407a, 0xfffc64)
00:12:12:Received ELS_PLOGI from 0xfffc64 to 0x530000
00:12:12:Forwarding to (1, 0xfffc64, 0x640500)
00:12:12:Received ELS_ACC from 0x640500 to 0xfffc64
00:12:12:Forwarding to (2, 0x530000, 0xfffc64)
00:12:12:Received ELS_PRLI from 0xfffc64 to 0x530000
00:12:12:Forwarding to (1, 0xfffc64, 0x640500)
00:12:12:Received ELS_ACC from 0x640500 to 0xfffc64
00:12:12:Forwarding to (2, 0x530000, 0xfffc64)
00:12:12:Received ELS_LOGO from 0xfffc64 to 0x530000
00:12:12:Forwarding to (1, 0xfffc64, 0x640500)
00:12:12:Received ELS_ACC from 0x640500 to 0xfffc64
00:12:12:Forwarding to (2, 0x530000, 0xfffc64)
00:12:15:Received ELS_ACC from 0x640000 to 0x530000
00:12:15:Routing (2, 0x640000, 0x530000) ->(1, 0xdf407a, 0x640500)
00:12:15:Forwarding to (1, 0xdf407a, 0x640500)
00:12:15:Received ELS_PRLI from 0x640500 to 0xdf407a <<<<<<<<<<<<<<<<<<<<<<
00:12:15:Routing (1, 0x640500, 0xdf407a) ->(2, 0x530000, 0x640000)
00:12:15:Forwarding to (2, 0x530000, 0x640000)
00:12:15:Received ELS_ACC from 0x640000 to 0x530000
00:12:15:Routing (2, 0x640000, 0x530000) ->(1, 0xdf407a, 0x640500)
00:12:15:Forwarding to (1, 0xdf407a, 0x640500)


---------------

Troubleshooting:

if fcns database does not have fcid of the storage/hosts in appropriate vsan
and ivr nat enabled.

how ivr internal fcid-rewrite-listshow ivr int event-history fcid-rewrite-fsm vsan 20 did 0x65440show ivr int event-history pv-fsm pwwn 21:00:00:e0:8b:1e:22:82 vsan 20 show ivr int event-history pv-fsm pwwn 21:00:00:e0:8b:1e:32:82 vsan 20 show ivr internal event-history errshow ivr int event-history pv-fsm err sh ivr internal area-port-allocation pwwn 21:00:00:e0:8b:1e:22:82 sh ivr internal area-port-allocation pwwn 21:00:00:e0:8b:1e:32:82 sh ivr internal area-port-allocation pwwn vsan 20Show ivr internal pvm

---

If you have issues with host not talking to storage, eventhough
show ivr zoneset active
show zoneset active
does show those devices are zoned together and active.

look at show fcns database vsan X where VSAN X is host vsan and check in storage edge vsan as well and look for fc4 type, node type. If it is not exported correctly
even though it shows correctly in native VSAN,
then it may be few bugs
but herre are few commands (non-disruptive) that you can try.
x9# ivr dev pwwn fcns register vsan
x9# ivr dev pwwn fcns register vsan
or
Here was my reply to the customer

Symptom:

0xde0010 - 50:---------------:24 on x9 vsan 21.

Action Plan 1:

shut/no shut on the device port 50:-----------------:24 and
50:-------------------:24
Action Plan 2:

ivr withdraw domain 0xde vsan 21
ivr withdraw domain 0x32 vsan 21
ivr refresh and see if it fixes the issue.

Action Plan 3:
ivr pv pwwn vsan 21 ns-query
ivr pv pwwn 50:------------:24 vsan 21 post 28


or Action Plan 4: (customer tried this and it worked)


x9# ivr dev pwwn 50:----------------:24 fcns register vsan 21
x9# ivr dev pwwn 50:------------:24 fcns register vsan 21

Bug might be
CSCsk49761

VSAN 21: then try this and see if fc4 type is populated correctly.
show fcns database | include 50:----------- :24