Note this:
- retransmit failure, is because IP network not able to handle the fc traffic
( change CWM burstsize, reduce it tcp cwm burst 10) to so that when there is
fc traffic, it is not that bursty, reducing cwm might cause less traffic
to be sent in bursts when there is increase in fctraffic)
- if you see a lot B2b credit starvation on FC ports, (for eg, ports
which use fcip link, for eg, fc port where storage array is connected,
which replicates to remote storage array), increase send-buffer-size
in fcip profile.
- compression between various different cards might be different
ips-8, 18+4,etc.
- write accelaration/tape accelaration and ivr, please look for transit
vsan and might break fcip WA/TA because of equal cost paths available
via IVR., anyway fcip TA certainly has issues with multiple equal cost paths.
# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
b#wm Enable congestion window monitoring
keepalive-timeout Set keep alive timeout in sec
max-bandwidth-kbps Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps Configure maximum available path bandwidth in Mbps
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisecond
pmtu-enable Enable PMTU Discovery
sack-enable Enable SACK option for TCP
send-buffer-size Send buffer size in KBytes
The CWM parameter : The default value is 10K and should be left untouched under normal conditions. CWM is a way of controlling burstiness after long idle times or loss of Acks. CWM stands for congestion window monitoring .
The keepalive-timeout is the TCP keepalive timeout value and is set to 60 sec. by default . The configurable values range between 1 and 7200 sec.
The max- and min-bandwidth parameter programs the TCP Maximum Window Size (scaling factor) and engages an internal "shaper" functionality . These values should be carefully chosen and requires understanding of intermediate network's end-to-end topology .The default values are to be changed according to the aforementioned requirements.The Round-trip-time can be derived once you have your FCIP tunnel up and running as follows :
bison# ips measure 200.200.200.1 interface gigabitethernet 4/1
Round trip time is 53 micro seconds (0.05 milliseconds )
bison#
Always add an additional margin of a few Microseconds to this value as a minium.
The max-retransmissions counter is set to 4 by default - in a healthy network environment this value should be left unchanged.
The max-retransmission timer is set to 200msec - If you experience extreme high retransmission counters this value might be increased -but in general this would not be required unless the RTT is above the 200msec value .
The PMTU (path mtu discovery) is enabled by default - best practice is to know which is the maximum MTU size supported by all interfaces along the logical path between both peers . Refer to RFC1191 for more details .
The SACK feature (selective acknowledgment) is not enabled by default - it could be considered when you have a lot of retransmissions going on between the two peers - SACK will allow selective retransmissions of your window - which is beneficial if larger maximum window sizes are configured and retransmissions are experienced frequently . In our sample config we have enabled it - when you do that make sure it is enabled at either side of the link .
The send-buffer-size is the amount of buffers in addition to the TCP window we allow to be transmitted out before we start to flow control the FC sources.The default value is set to 0 .
Configuration from MDS9216
c# sh run
Building Configuration ...
fcip profile 200
ip address 200.200.200.1
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10
fcip profile 201
ip address 200.200.200.5
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 100 round-trip-time-ms 10
!.....the TCP parameters are identical to what we had configured on the peering FCIP interfaces , only in very specific cases we should consider different values , e.g. if the return-path(s) are running across a different part of the
interface fcip1
channel-group 2 force
no shutdown
use-profile 200
peer-info ipaddr 100.100.100.1
interface fcip2
channel-group 2 force
no shutdown
use-profile 201
peer-info ipaddr 100.100.100.5
!..both fcip1 and fcip2 are bound to the same Channel-group 2 - also note that we have no strict relationship between profile-id and fcip interface numbering here as this is not a requirement. However , from a management and troubleshooting perspective a "strict" relationship of both values is recommended...
FCIP Generic ( outputs needed to troubleshoot)
· show interface gig
· show ips stats tcp int gig
. show ips stats dma-bridge int gig x/y - displays timestamp errors
. show ips stats buffer int gig x/y --- any buffer relted issue
- slow fcip connections if buffer is less than 70K
- show int fcip counters -- and show int fcip and show ips stats hw-comp
- hw compression /compression ratio , WA stats, TA stats
· show ips arp int gig
· show ips ip route int gig
· show interface fcip
· show profile fcip
· show interface port
· show int fcip
· show fcdomain vsan
· show fcns da vsan
· show
http://www-tac.cisco.com/Teams/SAN/Bru/IPS8_elaborate.htm
show ips - tcp stats/fcip
show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
9506# show ips internal fcip-trace-log
ips measure-rtt ip-address
---------------
show int fcip X counters ( any WA issues like ABTS)
show ips stats tcp/dma
If FC traffic is bursty, you may want to increase sendbuffer size
max 8M ( 3.0 it is 16K), so that FC will dump the frames on to
this buffer, then fcip can process the frames as per bw availability.
This will reduce back filling fc causing lack of b2b credits
If RTT time and TCP window ( how many bytes within RTT) might cause
timestamp errors, if the send buffer is not cleared within fcdrop
latency ( 500 ms)
If RTT is 40 ms and TCP window ( show ips stats) and show int fcip
will give tcp window., is 1256 K (1.2)and if send buffer is 8Mb,
then 8MB send buffer will be cleared with approx 40ms x 8/1.2 =320 ms
but if RTT is 80 ms, then it might take about 640 ms, that means some
frames might get dropped (dma-bridge timestamp errors)
CWM changes the burstiness of TCP side, default 50K means 50K frames
sent without ACK at a burst, this might clean up fcip traffic or
send buffer, but might cause TCP retrans if network can't handle it.
- small send buffer will cause lack of b2b credit and congestion
on FC side
- bigger send buffer will cause time stamperrors if the buffer
is not cleared within fc drop latency
- small cwm might cause time stamp errors because fc frames are
queued (etherenet send queue) and might stay longer in fc/switch
- higher cwm might cause bursty tcp traffic and might lead to
higher retrans.
- higher b2b credit might cause filling up of send buffer quickly.
so good luck tunning fcip params.!
Look for any hardware errors on fcip blade as well.
Requested Send buffer - configured params ( tcp send-buffer)
Allocated send buffer - configured + tcp window
8000+ 1257 = 9257 (for eg.)
fcip profile ---- max/min bw is post compression.
so show int fcip can have more thro'put than max bw configured in fcip
profile
show int gig 's thro'put is deteermined by max/min bw configured on fcip
profile.
1 comment:
This is the only info I could find outside of Cisco docs on FCIP implementation on Cisco hardware. I am going to be going through an implementation shortly. One thing I have read in the cisco docs is that we should be able to port channel the GigE interfaces for link redundancy, but I can't ever add the GigE interfaces to a port channel. I can add FCIP interfaces to a port channel. I suppose that could overcome link failure, I need to test this though. Do you have any thoughts on that?
I'm surprised I can't find more people doing this stuff. I appreciate your blog though, it's one of the only sources of info outside of Cisco I come to when I need to do something new.
Post a Comment