Operating DWDM SFP+ with Fabric OS 7.x.x on Brocade 6505 / 6510 / 6520

Thomas Weible 14. Jun 2013

Typically you can run 8G Fibre Channel SFP+ up to 25km for single mode fibre in Brocade Fabric OS (e.g. serving the new Brocades 6505, 6510 or 6520).
Sometimes it is handy to have DWDM or CWDM or an absolute must, but for some reason you are not able to get hold of these via Brocade. We managed to get our DWDM SFP+ ( ) and CWDM SFP+ ( ) up and running in the Brocade gear. When you plug them in, a warning prints itself on the console:

SW6505:admin> errdump -r
2013/06/12-10:05:48, [FW-1047], 15, FID 128, WARNING, SW6505, Sfp Current for port 1, 
is above high boundary(High=50, Low=0). Current value is 82 mA.

This looks like the SFP+ is faulty, but this isn’t true. The Fabric OS has hardcoded thresholds for the SFPs. The SFP current should not exceed 50mA, otherwise a warning will be generated. Now, looking at the SFP+´s own thresholds, a high current warning will be prompted above 110mA. In other words, something must be there to set this low current threshold. If you dig deeper into the Fabric OS you will find that the so-called “Fabric Watch” takes care of certain thresholds, including SFP relevant once. The Fabric Watch is quite powerful, e.g. you can set defined actions (e.g. sending an email) directly onto the switch. The nice thing about this is that you can tweak those thresholds in such a way that this warning message does not appear in your log. By default, the Fabric Watch is enabled ,if you have purchased the appropriate licence:

SW6505:admin> licenseshow
...
    Extended Fabric license
    Fabric Watch license

SW6505:admin> fwalarmsfiltershow
FW: Alarms are enabled

SW6505:admin> thmonitor --show
Brcd SFP Threshold Monitoring is enabled

To see the actual thresholds for the SFPs, you can run a (I reduced it to “Current” but you can also gather the Fabric OS SFP relevant thresholds for “TX level”, “RX level”, “Voltage” or “Temperature”):

SW6505:admin> thconfig --show sfp -area Current
Class: SFP 
        Area    : CURRENT
        ThLevel : Def
        ActLevel: Def
        High    : 
                Custom: 
                        TimeBase: None
                        Value   : 50
                        Trigger : Above Action: Raslog
                        Trigger : Below Action: None
                Default: 
                        TimeBase: None
                        Value   : 50
                        Trigger : Above Action: Raslog 
                        Trigger : Below Action: None 
        Low: 
                Custom: 
                        TimeBase: None
                        Value   : 0
                        Trigger : Below Action: Raslog
                Default: 
                        TimeBase: None
                        Value   : 0
                        Trigger : Below Action: Raslog 
        Buffer: 
                Custom: 
                        Value   : 1
                Default: 
                        Value   : 1

You can see that there are a “Default” and a “Custom” level. By default ,they are set to the same range, but the “Custom” area you can modify. And that’s what we do now. We change the high current threshold to 110mA and set the “Custom” level as the one which the Fabric Watch should consider:

SW6505:admin> thconfig --set sfp -area current -highth -value 110
SW6505:admin> thconfig --apply sfp -area Current -thresh_level cust

after executing the command above, the output will look like this:

SW6505:admin> thconfig --show sfp -area current
Class: SFP 
        Area    : CURRENT
        ThLevel : Cust (Applied)
        ActLevel: Def
        High    : 
                Custom: 
                        TimeBase: None
                        Value   : 110
                        Trigger : Above Action: Raslog
                        Trigger : Below Action: None
                Default: 
                        TimeBase: None
                        .....

and the warning messages will disappear. You can double-check it with (my DWDM SFP+ are plugged in at Port 0 and 1):

SW6505:admin> fwportdetailshow 
Port Detail Report                          Report time: 06/12/2013 11:36:46 AM
Switch Name:    SW6505
IP address:     xxx.xxx.xxx.xxx
Port Exception report [by All]
                             --------Port-Errors------------ -----SFP-Errors----
Port# Type  State   Dur(H:M) LFA LSY LSI PER INW CRC PSC LR BLP STM SRX STX SCU SVO
--------------------------------------------------------------------------------
000     LB  ONLINE    000:05   -   -   -   -   -   -   -   -   -   -   -   -   -
001     LB  ONLINE    000:05   -   -   -   -   -   -   -   -   -   -   -   -   -
002      U  OFFLINE   039:23   -   -   -   -   -   -   -   -   -   -   -   -   -
003      U  OFFLINE   039:23   -   -   -   -   -   -   -   -   -   -   -   -   -

Finding the correct nomenclature for “thmonitor” command requires little bit of time. If you read the entire output of the command, you will have succeeded. For more information there is some interesting documentation on the Fabric Watch from Brocade.