--- name: "ibdmsh" section: "1" platform: "linux" package: "ibutils" version: "1.5.7+0.2.gbd7e502-3" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/ibdmsh" --- # ibdmsh(1) InfiniBand data model - an extentended TCL shell ## NAME ibdmsh - InfiniBand data model - an extentended TCL shell ## DESCRIPTION **ibdmsh** is a TCL shell extended with interface for the IB data model. To use this shell you will write TCL code that directly access the IB data model objects and functions. The following sub sections provide detailed definition for those objects and API. ## IBDM Constants ### Node Types The following constants are used by the node object type field \[ Constant : int \] `$IB_UNKNOWN_NODE_TYPE` = IB\_UNKNOWN\_NODE\_TYPE \[ Constant : int \] `$IB_SW_NODE` = IB\_SW\_NODE \[ Constant : int \] `$IB_CA_NODE` = IB\_CA\_NODE ### Log Verbosity Flags The following constants are used as argument bits for the global variable `$FabricUtilsVerboseLevel` \[ Constant : int \] `$FABU_LOG_NONE` = 0x0 \[ Constant : int \] `$FABU_LOG_ERROR` = 0x1 \[ Constant : int \] `$FABU_LOG_INFO` = 0x2 \[ Constant : int \] `$FABU_LOG_VERBOSE` = 0x4 ## IBDM Globals Log level: set to FABU\_LOG\* values \[ Global : int \] `$FabricUtilsVerboseLevel` ## IBDM Objects This section describes the various object types exposed by IBDM. IBDM exposes some of its internal objects. The objects identifiers returned by the various function calls are formatted according to the following rules: Fabric: fabric:\ System: system:\:\ SysPort: sysport:\:\:\ Node: node:\:\ Port: port:\:\/\ IBDM Objects are standard Swig-Tcl objects. As such they have two flavors for their usage: Variables, Objects. **Variables/Pointers:**\ For each object attribute a "get" and "set" methods are provided.\ The format of the methods is: \\_\\_\.\ The "set" method is only available for read/write attributes. ``` Example: set nodes [ibdm_get_nodes] set node [lindex $nodes 0] IBNode_numPorts_get $node ``` **Objects:**\ Given an object pointer one can convert it to a Tcl "Object"\ using the following command:\ \ \ -this \ ``` Once declared the can be used in conjunction to with the standard "configure" and "cget" commands. Example (following the previous one): IBFabric VaTech -this $fabric VaTech cget -NodeByName To delete an object symbol (and enable its mapping to another pointer) use: rename "" for example: rename VaTech "" ``` ### class IBPort The following are the different fields and methods of the IB Port class which describes an IB device (chip) physical port. - \[ Member data: returns IBPort \* \] -p\_remotePort Port connected on the other side of link - \[ Member data: returns IBSysPort \* \] -p\_sysPort The system port (if any) connected to - \[ Member data: returns IBNode \* \] -p\_node The node the port is part of. - \[ Member data: returns int \] -num Physical ports are identified by number. - \[ Member data: returns unsigned int \] -base\_lid The base lid assigned to the port. - \[ Member data: returns IBLinkWidth \* \] -width The link width of the port - \[ Member data: returns IBLinkSpeed \* \] -speed The link speed of the port - \[ Member data: returns unsigned int \] -counter1 A generic value to be used by various algorithms - \[ Constructor: returns IBPort \* \] IBPort name p\_nodePtr number IBPort constructor - \[ Member : returns new\_uint64\_t \] guid\_get Obtain the guid of the port - \[ Member : returns void \] guid\_set guid Modify the guid of the port - \[ Member : returns new\_string \] getName Get the port name: A device port connected to system port (front panel) returns the front panel port name. - \[ Member : returns void \] connect p\_otherPort ?width? ?speed? Connect the port to another node port with optional width and speed parameters - \[ Member : returns int \] disconnect Disconnect the port. Return 0 if successful ### class IBNode The IB Node class represents a single IB Device (chip) - \[ Member data: returns string \* \] -name Name of the node (instance name of the chip) - \[ Member data: returns IBNodeType \] -type Either a `$IB_SW_NODE` or `$IB_CA_NODE` - \[ Member data: returns uint16\_t \* \] -devId The device ID of the node - \[ Member data: returns uint16\_t \* \] -revId The device revision Id. - \[ Member data: returns uint16\_t \* \] -vendId The device Vendor ID. - \[ Member data: returns string \* \] -attributes Comma-sep string of arbitrary attributes k=v - \[ Member data: returns uint8\_t \] -rank The rank of the node (0 is a root) - \[ Member data: returns IBSystem \* \] -p\_system What system we belong to - \[ Member data: returns IBFabric \* \] -p\_fabric What fabric we belong to. - \[ Member data: returns unsigned int \] -numPorts Number of physical ports - \[ Member data: returns vec\_pport \* \] -Ports Vector of all the ports - \[ Member data: returns vec\_vec\_byte \* \] -MinHopsTable A table of the number of hops required to get from each port to each target LID - \[ Member data: returns vec\_byte \* \] -LFT The LFT of this node (for switches only) which is a long vector of target ports - index is the LID - \[ Member : returns new\_uint64\_t \] guid\_get Obtain the node GUID - \[ Member : returns void \] guid\_set guid Set the node GUID - \[ Constructor: returns IBNode \* \] IBNode name n p\_fab p\_sys t np Constractor - \[ Member : returns IBPort \* \] makePort num Create a new port by its number (if required) return the port pointer - \[ Member : returns IBPort \* \] getPort num Get a port by number num = 1..N: - \[ Member : returns void \] setHops p\_port lid hops Set the min hop for the given port (\* is all) lid pair - \[ Member : returns int \] getHops p\_port lid Get the min number of hops defined for the given port or all - \[ Member : returns IBPort \* \] getFirstMinHopPort lid Scan the node ports and find the first port with min hop to the lid - \[ Member : returns void \] setLFTPortForLid lid portNum Set the Linear Forwarding Table for the given LID - \[ Member : returns int \] getLFTPortForLid lid Get the LFT for a given LID - \[ Member : returns void \] repHopTable Dump out the min hop table of the node ### class IBSysPort The IBSysPort class represents an IB plug on the system front or back panels - \[ Member data: returns string \* \] -name The front panel name (silk) of the port - \[ Member data: returns IBSysPort \* \] -p\_remoteSysPort If connected the other side sys port - \[ Member data: returns IBSystem \* \] -p\_system System it benongs to - \[ Member data: returns IBPort \* \] -p\_nodePort The node port it connects to. - \[ Constructor: returns IBSysPort \* \] IBSysPort name n p\_sys Constructor - \[ Member : returns void \] connect p\_otherSysPort ?width? ?speed? Connect two SysPorts - \[ Member : returns int \] disconnect Disconnect the SysPort (and ports). Return 0 if successful ### class IBSystem The IBSystem class represents an entire chassis - \[ Member data: returns string \* \] -name The "host" name of the system - \[ Member data: returns string \* \] -type What is the type i.e. Cougar, Buffalo, MTS2400, etc. A corresponding IBNL file should exist - defining this system type - \[ Member data: returns IBFabric \* \] -p\_fabric Fabric the system belongs to - \[ Member data: returns map\_str\_pnode \* \] -NodeByName Provide the node pointer by its name - \[ Member data: returns map\_str\_psysport \* \] -PortByName A map provising pointer to the SysPort by name - \[ Constructor: returns IBSystem \* \] IBSystem name n p\_fab t Constractor - \[ Member : returns new\_uint64\_t \] guid\_get Obtain the system image GUID - \[ Member : returns void \] guid\_set guid Set the system image GUID - \[ Member : returns IBSysPort \* \] makeSysPort pName Make sure we got the port defined (so define it if not) - \[ Member : returns IBPort \* \] getSysPortNodePortByName sysPortName Get the node port for the given sys port by name - \[ Member : returns IBSysPort \* \] getSysPort name Get a Sys Port by name ### class IBFabric Represents an entire IB subnet made of systems - \[ Member data: returns map\_str\_pnode \* \] -NodeByName Provide a list of node name and pointer pairs - \[ Member data: returns map\_str\_psys \* \] -SystemByName Provide a list of system name and pointer pairs - \[ Member data: returns vec\_pport \* \] -PortByLid Provides a list of system port name and pointer pairs - \[ Member data: returns map\_guid\_pnode \* \] -NodeByGuid Provides a list of node guid and node pointer pairs - \[ Member data: returns map\_guid\_psys \* \] -SystemByGuid Provides a list of system image guid and system pointer pairs - \[ Member data: returns map\_guid\_pport \* \] -PortByGuid Provides a list of port guid and port pointer pairs - \[ Member data: returns unsigned int \] -minLid Track min lid used. - \[ Member data: returns unsigned int \] -maxLid Track max lid used. - \[ Member data: returns unsigned int \] -lmc LMC value used - \[ Member : returns IBNode \* \] makeNode n p\_sys type numPorts Get the node by its name (create one of does not exist) - \[ Member : returns IBNode \* \] getNode name Get the node by its name - \[ Member : returns list\_pnode \* \] getNodesByType type Return the list of node pointers matching the required type - \[ Member : returns IBSystem \* \] makeGenericSystem name Create a new generic system - basically an empty container for nodes... - \[ Member : returns IBSystem \* \] makeSystem name type Create a new system - the type must have a registered factory. - \[ Member : returns IBSystem \* \] getSystem name Get system by name - \[ Member : returns IBSystem \* \] getSystemByGuid guid get the system by its guid - \[ Member : returns IBNode \* \] getNodeByGuid guid get the node by its guid - \[ Member : returns IBPort \* \] getPortByGuid guid get the port by its guid - \[ Member : returns void \] addCable t1 n1 p1 t2 n2 p2 ?width? ?speed? Adds a cable given two sets of node type, node name and port number. Optionally use a given width and speed for the connection - \[ Member : returns int \] parseCables fn Parse the cables file and build the fabric - \[ Member : returns int \] parseTopology fn Parse Topology File and build the fabric - \[ Member : returns int \] addLink type1 numPorts1 sysGuid1 nodeGuid1 portGuid1 vend1 devId1 rev1 desc1 lid1 portNum1 type2 numPorts2 sysGuid2 nodeGuid2 portGuid2 vend2 devId2 rev2 desc2 lid2 portNum2 ?width? ?speed? Add a link into the fabric - this will create system and nodes as required. - \[ Member : returns int \] parseSubnetLinks fn Parse the OpenSM subnet.lst file and build the fabric from it. - \[ Member : returns int \] parseFdbFile fn Parse OpenSM FDB dump file and fill in the switches LFT tables - \[ Member : returns int \] parseMCFdbFile fn Parse an OpenSM MCFDBs file and set the MFT table accordingly - \[ Member : returns int \] parsePSLFile fn Parse Path to SL mapping file. Each line with: src\_node\_guid DLID SL Used by credit loop check - \[ Member : returns int \] parseSLVLFile fn Parse SLVL tables file. Each line holds: sw\_node\_guid in\_port out\_port 0x(sl0)(sl1) 0x(sl2)(sl3)... - \[ Member : returns void \] setLidPort lid p\_port Set a lid port - \[ Member : returns IBPort \* \] getPortByLid lid Get a port by lid - \[ returns IBFabric \* \] new\_IBFabric Construct a new fabric - \[ returns void \] delete\_IBFabric p\_fabric Destruct a fabric ## IBDM Functions This section provide the details about the functions IBDM exposes. The order follows the expected order in a regular IBDM flow. They all return 0 on succes. ### Subnet Utilities The file holds a set of utilities to be run on the subnet to mimic OpenSM initialization and analyze the results: - \[ returns int \] ibdmAssignLids p\_smNodePort ?lmc? Assign LIDs with an optional LMC (multiple LID per port) - \[ returns int \] ibdmCalcMinHopTables p\_fabric Calculate and populate the MinHopTables required for running OpenSM style routing. - \[ returns int \] ibdmCalcUpDnMinHopTbls p\_fabric rootNodesNameRex Calculate and populate the MinHopTables following Up/Down rule. - \[ returns int \] ibdmOsmRoute p\_fabric Route the fabric with OpenSM style routing - \[ returns int \] ibdmEnhancedRoute p\_fabric Route the fabric with OpenSM style routing enhanced with better support for LMC \> 0 - \[ returns int \] ibdmFatTreeRoute p\_fabric rootNodes Route the fabric using algorithm that fits only full fat trees - \[ returns int \] ibdmFatTreeAnalysis p\_fabric Analyze the fabric to see if it is a fat tree and route it if it is - \[ returns int \] ibdmVerifyCAtoCARoutes p\_fabric Make sure all (H)CAs are connected to each other based on the LFT settings - \[ returns int \] ibdmVerifyAllPaths p\_fabric Make sure all Switches and (H)CAs are connected to each other based on the LFT settings - \[ returns int \] ibdmAnalyzeLoops p\_fabric A rigorous check for credit loops. This algorithm does a full and accurate check but its reporting of credit loop paths is hard to interpret. If you know the roots of the tree (or the tree is symmetrical) it is preferable to use the ibdmReportNonUpDownCa2CaPaths - \[ returns list\_pnode \] ibdmFindSymmetricalTreeRoots p\_fabric Analyze the tree topology and find the roots of the tree based on its symmetry - \[ returns list\_pnode \] ibdmFindRootNodesByMinHop p\_fabric Based on the MinHopTable find the roots of the tree. A 5% assymetry is allowed - \[ returns int \] ibdmRankFabricByRoots p\_fabric rootNodes Given the list of root nodes (names) rank the nodes (root = 0) - \[ returns int \] ibdmReportNonUpDownCa2CaPaths p\_fabric rootNodes Analyze the routes to make sure Up/Down rule is maintained - \[ returns int \] ibdmCheckMulticastGroups p\_fabric Verify connectivity of multicast routing - \[ returns int \] ibdmCheckFabricMCGrpsForCreditLoopPotential p\_fabric rootNodes Analyze multicast routing to make sure it obeys Up/Down rule - \[ returns int \] ibdmLinkCoverageAnalysis p\_fabric rootNodes Prepare a schedule for transmission from a set of sources to destinations such that in each stage there are no links that are over subscribed and after all stages all the links of the fabric were excersized ### Tracing Utilities These functions allows tracing paths reporting back the visited nodes - \[ returns int \] ibdmTraceDRPathRoute p\_smNodePort drPathPortNums Trace a directed route path from the given port - \[ returns int \] ibdmTraceRouteByMinHops p\_fabric slid dlid Trace a path along the MinHop from the source to destination LIDs - \[ returns int \] ibdmTraceRouteByLFT p\_fabric slid dlid hops p\_nodesList Trace a path following the LFT updating the hops and node list variables ### Topology Matching Utilities The following utilities matches two fabrics providing both mismatch messages and a unified fabric - \[ returns int \] ibdmMatchFabrics p\_spec\_fabric p\_discovered\_fabric anchorNodeName anchorPortNum anchorPortGuid Match a topology based fabric with a discovered one starting at the enchor port - \[ returns int \] ibdmBuildMergedFabric p\_spec\_fabric p\_discovered\_fabric p\_merged\_fabric After matching using ibdmMatchFabrics populate the "merged fabric" with information merged for the matched nodes ### Congestion Analysis Utilities Provide ability to track and report link oversubscription - \[ returns int \] ibdmCongInit p\_fabric Initialize the tracking system - \[ returns int \] ibdmCongCleanup p\_fabric Cleanup the counters and deallocate - \[ returns int \] ibdmCongClear p\_fabric Cleanup counters - \[ returns int \] ibdmCongTrace p\_fabric srcLid dstLid Trace a path between the LIDs updating link usage info - \[ returns int \] ibdmCongReport p\_fabric Report the resulting congestion information - \[ returns int \] ibdmCongDump p\_fabric Dump all congestion tracking info ## AUTHOR Eitan Zahavi, Mellanox Technologies LTD, eitan@mellanox.co.il