IBM* PCI Hot-Plug Controller Driver v1.05b for Netware v5.1/v6.0 and Novell NetWare v5.1/v6.0 Installation Instructions. Readme.txt Version 1.05b CONTENTS --------- 1.0 Hot-Plug PCI Installation Overview for Novell NetWare v5.1/v6.0 1.1 Overview 1.2 Limitations 1.3 Dependencies: 2.0 Change History 3.0 Installation and Setup Instructions for Netware v5.1/v6.0 4.0 How to perform Hot-Plug PCI Operations 4.1 Required steps in Hot-Plug Operations 4.1.1 Removing an adapter 4.1.2 Replacing an adapter 4.1.3 Adding a new adapter to an empty slot 4.2 How to determine if your driver supports Hot-Plug PCI Operations 4.3 Warnings about incorrect usage of Hot Plug PCI. 5.0 Troubleshooting hot plug PCI operations 5.1 Basic Troubleshooting 5.1.1 Attention Indicator LED 5.1.2 IBMSBD.NLM messages 5.1.3 Adapter Driver error messages 5.2 Advanced Troubleshooting 5.2.1 How to determine if the problem is with the adapter driver or the hot-plug driver. 5.2.2 Use of switches and log file for IBMSBD driver 5.2.3 Interrupt handling 5.2.4 NCMCON issues 6.0 Disclaimer 1.0 Hot-Plug PCI Installation Overview for Novell NetWare v5.1/v6.0 1.1 Overview The Hot Plug PCI support for NetWare v5.1/v6.0 provided by IBM is compliant with many different specifications including full compliance with the NetWare System Bus Driver specification v1.01, PCI v2.1, and the PCI Hot-Plug PCI v1.1 specification. The IBM Hot Plug PCI system support consists of an interlock switch and a set of two LED's for each hot plug PCI slot. One LED will remain on when power is ON to the given slot. The other LED will indicate that attention is required to that slot. Slots that do not have these devices are NOT hot plug slots. All hot plug operations must be controlled through a console provided by Novell. IN NO CASE SHOULD HOT PLUG OPERATIONS BE DONE WITHOUT FIRST REMOVING POWER FROM THE SLOT THROUGH THE NETWARE CONFIGURATION MANAGER CONSOLE (NCMCON). IF A SLOT DOES NOT HAVE A LATCH,DO NOT REMOVE THE ADAPTER. DOING SO CAN CAUSE SERIOUS DAMAGE TO THE SYSTEM AND ADAPTER. The order of events in any hot-plug operation are: - Load the IBM PCI Hotplug controller driver to provide hot plug PCI support. - Go to the NetWare Configuration Manager Console (NCMCON) - Choose an add or remove operation for the given slot. - When the adapter power is OFF, open the latch and remove, replace or add an adapter and any necessary cabling. - Close the latch and return to the NCMCON console. NOTE: The power to a slot cannot be turned ON until the latch is closed for the given slot. The position of the latch (opened or closed) is not important if no adapter is in the given slot. - In the case of adding a new adapter or replacing an old one, the NCMCON console will prompt you to turn on the newly added adapter. - If selected, the adapter power is turned ON and the adapter is configured. At this point, if you are using a LAN driver, HWDETECT.NLM will run and load the appropriate driver for you. If you are using any other type of adapter, you will have to go to the system console and manually load the appropriate driver. NOTE: In the case of a SCSI controller, you may need to issue the console command 'SCAN FOR NEW DEVICES' prior to the drive being recognized by NetWare. 1.2 Limitations - Cannot load driver multiple times within same boot session. If the driver is loaded, adapters are hot-added, then the driver is unloaded and re-loaded, erroneous behavior will occur. - Multi-Function PCI devices are supported as long as the multi-function capability is not provided with or through a PCI-to-PCI bridge. - Video adapters are not supported in the hot plug PCI environment due to I/O space limitations and restrictions. - Devices that are not PCI v2.1 compliant or that do not implement the PCI presence detection pins are not supported. - Device drivers that are not LAN ODI v3.31 compliant or SCSI NWPA v3.00b compliant are NOT supported. - Non-PCI devices are not hot pluggable. - HWDETECT.NLM, a Novell provided module, may not find the correct driver if more than one driver is capable of loading for the given adapter. - Because of errors encountered in testing, HWDETECT.NLM is not provided by IBM. If you want the capability of being able to auto-detect the correct driver for newly added adapters, download the latest HWDETECT.NLM from the Novell web site. The URL to find the latest version of HWDETECT.NLM from Novell's web site is: http://support.novell.com/search/ff_index.htm 1.3 Dependencies: All files that are required by the Hotplug controller driver are provided in the standard Novell Netware v5.1/v6.0 package. 2.0 Change History -------------------- ********************** * V1.05b Changes * * January 2002 * ********************** Added delay to handle ServeRAId cards. ********************** * V1.05 Changes * * November 2001 * ********************** Made changes in the error handling for powering on a slot. Needed to pass Novell certification. ********************** * V1.04b Changes * * November 2001 * ********************** Fixed bug involving stale data in bus resgister. This bug prevented the addition of some adapter cards displaying a bus speed mismatch. ********************** * v1.04 Changes * * November 2001 * ********************** Added dynamic bus speed change capability. Placed debug information inside of DEBUG conditionals ********************** * v1.03 Changes * * October 2001 * ********************** Added readme file. Corrected bus speed displayed on user interface. ********************** * v1.00 Changes * * October 2001 * ********************** Original Release 3.0 Installation and Setup Instructions for Netware v5.1/v6.0 ----------------------------------------- To install the IBM PCI Hotplug controller driver, perform the following operation: 1. Start the Netware server and go to the Netware console. 2. Copy the PCI Hotplug controller driver (IBMXSBD.NLM) to hard drive (C:\) 3. Type Load C:\IBMXSBD.NLM NOTE: When the NetWare Configuration Manger Console (NCMCON) is opened, it will show the available Hotplug PCI slots, their current status, and any adapters that are currently in the Hotplug PCI slots. NOTE: If there is a failure of any type, please power down the machine, reboot, and Type 'Load C:\IBMXSBD.NLM -DBGALL > C:\IBMXSBD.TXT' and provide the IBMXSBD.TXT file for debugging purposes. 4.0 How to perform Hot-Plug PCI Operations 4.1 Required steps in Hot-Plug Operations NOTE: Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system will most likely hang, requiring the system to be rebooted. 4.1.1 Removing an adapter 1. Verify that the IBMXSBD.NLM and NCMCON.NLM modules are loaded. 2. Go to the NetWare Configuration Manager Console (NCMCON) screen. 3. Select the slot where the adapter you want to remove is located and Press Enter. 4. Select 'Remove Adapter' and press Enter. 5. If a driver is loaded for the adapter, you will get a prompt asking if you want to unload the driver. You must have the driver unloaded for the power to the slot to be removed. 6. If you receive a message stating that the driver could not be unloaded, go to the system console and manually remove the driver. Return to NCMCON after the driver has been unloaded and repeat steps 3 through 5. 7. Verify that the status for the slot reads 'Powered off'. 8. Verify that the power-on light status for the adapter is OFF. If the light is ON, go back to the console and verify that the given slot is powered OFF. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 9. Open the latch and remove the adapter from the slot. 4.1.2 Replacing an adapter 1. Verify that the IBMXSBD.NLM and NCMCON.NLM modules are loaded. 2. Go to the NetWare Configuration Manager Console (NCMCON) screen. 3. Select the slot where the adapter you want to remove is located and Press Enter. 4. Select 'Remove Adapter' and press Enter. 5. If a driver is loaded for the adapter, you will get a prompt asking if you want to unload the driver. You must have the driver unloaded for the power to the slot to be removed. 6. If you receive a message stating that the driver could not be unloaded, go to the system console and manually remove the driver. Return to NCMCON after the driver has been unloaded and repeat steps 3 through 5. NOTE: Beware that unloading the driver in most cases unloaded it for ALL instances of the adapter in the system. See Section 5.2 for any special commands that may be required to remove an individual instance of the driver. In the case where a driver does not have a special command to unload it and NCMCON is not able to unload the driver, a new device driver supporting 'instance unload' must be obtained from the adapter vendor. 'Instance unload' is part of the Novell NetWare ODI v3.31 LAN Specification and the Novell NetWare NWPA v3.00b Disk Specification. 7. Verify that the status for the slot reads 'Powered off'. 8. Verify that the power-on light status for the adapter is OFF. If the light is ON, go back to the console and verify that the given slot is powered OFF. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 9. Open the latch and remove the adapter from the slot. 10. Place the new adapter in the slot and close the latch. 11. Go back to the NCMCON screen. A message should be displayed asking if power should be turned on to the given slot. 12. After selecting to turn on the power to a given slot, you will be prompted to verify that any necessary cables have been attached to the adapter card prior to power being applied. 13. Press Enter at the cable reminder screen. The adapter power will be turned on, the adapter configured, and a message displayed as to the success of adding the adapter. NOTE: If an error occurs during power ON and configuration of the adapter, the power to the slot will be turned back OFF, and an appropriate message displayed on the NCMCON screen and the system console screen. If an adapter does not power up, go to Section 5.0 Troubleshooting Hot Plug PCI Operations. NOTE: In the case of a network adapter, be sure to BIND the appropriate protocols to the driver after the driver is loaded. NOTE: In the case of a SCSI controller, you may need to issue the console command 'SCAN FOR NEW DEVICES' prior to the drive being recognized by NetWare. NOTE: Drivers for newly added adapters do not automatically reload. Please see the instructions for the specifically supported adapters for information on how to reload drivers. 14. The status of the adapter in the NCMCON screen will show READY if a driver is not loaded for the given slot, and ACTIVE when a driver is loaded. 4.1.3 Adding a new adapter to an empty slot NOTE: Adding a new adapter to a slot with an adapter already in the slot is the same as replacing an adapter. The steps to replacing an adapter are given in Section 4.1.2 above. 1. Verify that the IBMXSBD.NLM and NCMCON.NLM modules are loaded. 2. Go to the NetWare Configuration Manager Console (NCMCON) screen. 3. Verify that the status for the slot reads 'Powered off'. 4. Verify that the power-on light status for the adapter is OFF. If the light is ON, go back to the console and verify that the given slot is powered OFF. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 5. Using NCMCON, choose the slot that the adapter will be added, select that slot by pressing ENTER, and choose ADD ADAPTER. 6. Open the latch and place the new adapter in the slot and close the latch. 7. Go back to the NCMCON screen. A message should be displayed asking if power should be turned on to the given slot. 8. After selecting to turn on the power to a given slot, you will be prompted to verify that any necessary cables have been attached to the adapter card prior to power being applied. 9. Press Enter at the cable reminder screen. The adapter power will be turned on, the adapter configured, and a message displayed as to the success of adding the adapter. NOTE: If an error occurs during power ON and configuration of the adapter, the power to the slot will be turned back OFF, and an appropriate message displayed on the NCMCON screen and the system console screen. If an adapter does not power up, go to Section 5.0 Troubleshooting Hot Plug PCI Operations. NOTE: In the case of a network adapter, be sure to BIND the appropriate protocols to the driver after the driver is loaded. NOTE: In the case of a SCSI controller, you may need to issue the console command 'SCAN FOR NEW DEVICES' prior to the drive being recognized by NetWare. NOTE: Drivers for newly added adapters do not automatically reload. Please see the instructions for the specifically supported adapters for information on how to reload drivers. 10. The status of the adapter in the NCMCON screen will show READY if a driver is not loaded for the given slot, and ACTIVE when a driver is loaded. 4.2 How to determine if your driver supports Hot-Plug PCI Operations Operations NOTE: This section shows how to handle devices that are not on the IBM approved list of hot plug PCI devices. Some devices may work correctly, but have not been tested and approved by IBM. NOTE: There are no changes to the physical adapter required to support hot plug PCI operations. Many other adapter vendors will be providing drivers that meet the required Novell specifications for hot plug PCI in the future. However, there are times where you may have a driver and will not know whether the driver is hot plug compatible. The following is a list of methods to determine if a given driver supports hot plug PCI operations. 1. Almost any driver will allow a singular instance in the hot plug environment. 2. The device can be powered OFF through the NCMCON screen with a driver loaded or has a command-line option to remove a single instance of the driver. NOTE: A driver that forces all instances of itself to be unloaded is not considered a hot plug PCI driver. 3. Can the driver detect newly added adapters after a hot plug operation? NOTE: Many adapters will not detect new slot numbers because they do a static scan for cards on their first load. If a new card is added, the driver will report that no slots are available. Unloading the adapter (and sometimes the underlying support module, TOKENTSM.NLM, ETHERTSM.NLM, etc) and reloading them will allow the new slots to be seen in most instances. However, this will require unloading the driver for all adapters in the system. 4. Does the driver complain about PCI configuration problems? NOTE: Some older drivers do not allow for PCI configuration resources to be reallocated or changed. These drivers will report that one or more of the PCI resource requirements (IRQs, I/O ports, Memory, or Prefetchable Memory) are not correct/unavailable. 4.3 Warnings about incorrect usage of Hot Plug PCI. Below are several warnings about hot plug PCI operations: 1. Do not remove an adapter from a non-Hotplug PCI slot. 2. Grounding equipment, including wrist straps, should be used when working inside any system unit to protect against electrostatic discharge (ESD) that could damage system components and adapter cards. 3. Make sure that any adapter card hot added to the system is fully seated before attempting to turn on power. 4. Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system could hang, requiring the system to be rebooted. 5. Do NOT remove the plastic cover over the system components. Removal of the plastic cover exposes system components that may be shorted by adapter brackets during hot plug PCI operations. 6. Processors, memory and integrated systems devices are not hot swappable. 5.0 Troubleshooting hot plug PCI operations ------------------------------------------- 5.1 Basic Troubleshooting There are several troubleshooting techniques that can be used to determine why a hot plug PCI operation failed. Two LED's are provided for each hot plug PCI capable slot. One LED blinks to indicate that attention is required. The second LED indicates power state. Messages are generated by the NCMCON screen, IBMXSBD.NLM driver, and the various adapter drivers. 5.1.1 Attention Indicator LED The Attention Indicator LED is controlled by NetWare. At present, the Attention Indicator LED is used only by device drivers to indicate that an adapter in a given slot needs attention. Not all device drivers support the attention indicator messaging. The Attention Indicator will only be cleared upon successful replacement of the adapter in the slot or by the device driver clearing the condition that led it to turn on the Attention Indicator LED in the first place. 5.1.2 IBMXSBD.NLM messages The IBMXSBD.NLM IBM PCI Hotplug controller Driver generates messages to indicate change of state in the hot plug PCI system. If the driver is not loaded, no system messages are generated. The messages that you may see generated by the IBMXSBD driver will all be proceeded by the IBMXSBD: or IBMXSBD Error: tags. The messages that may be seen are: "IBMXSBD: New Adapter added. Please use the Novell Configuration Manager console (NCMCON) to configure this new adapter." This indicates that a new adapter was added into a previously empty slot. The empty slot can occur as a result of a previous replace or add hot plug PCI operation. This is an information message directing you to use the console (NCMCON) to configure the newly added adapter. "IBMXSBD Error: Allocate NEB.AESTag() failed." This error indicates that the IBMXSBD driver was unable to allocate an asynchronous event system (AES) tag. Because events happen asynchronously in the hot plug PCI system, IBMXSBD must have an AES handle to function. Unloading other NLM's that have AES tags registered will free up AES resources so that the IBMXSBD driver can properly load. "IBMXSBD Error: Not enough memory to generate event for queue." This error indicates that memory could not be allocated to place a hot plug PCI event onto the internal resource queue. This normally occurs when the system runs out of available memory. To fix this problem, unload other NLM's or add additional memory to the system. "IBMXSBD Error: Add Adapter command failed because of empty slot. Slot is x" This messages occurs when NCMCON is directed to add an adapter to a slot that does not currently have an adapter card in it. This error may also occur if a PCI adapter does not meet the PCI v2.1 specification requirement that the adapter use presence pins. The IBMXSBD module uses the presence pins to determine when an adapter has been inserted and removed from the system. "IBMXSBD: The IBM Hot-Plug PCI controller is not present." This error occurs when an attempt is made to run the IBMXSBD module on a system that does not have a hot plug PCI controller. This error may also occur if the hot plug PCI controller is malfunctioning or not working. The -COMMAND -DEBUG switches can be used on the IBMXSBD driver to determine if the hot plug controller is working. See Section 5.2.2 for more information about debug switches for the IBMXSBD device driver. All other messages are provided through the Novell Configuration Manager Console (NCMCON) screen. For information about problems in NCMCON such as all slots show 'No' in the Hot Plug field or a message appears asking whether to continue waiting another 10 seconds, see Section 5.2.4. 5.1.3 Adapter Driver error messages Many non hot plug PCI device drivers will not load correctly with the ODI v3.31 or the NWPA v3.00b specifications. If you have such a device driver, first determine if you adapter vendor has a certified driver available. If no such driver exists, you will need to either use a different adapter or lose at least some of the hot plug functionality. Drivers that do not support the Novell specifications may exhibit such problems as: - Failure to see new adapters until unloaded and reloaded. - Failure to support single instance unload. Single instance unload is the ability to unload the driver for a single adapter even though the driver may be supported many adapters. - Failure to be able to register PCI resources such as interrupt, I/O ports, memory or prefetchable memory ranges. 5.2 Advanced Troubleshooting This section contains information that is more technically advanced than the Basic Troubleshooting section. Information contained in this section includes: - How to determine if a problem loading an adapter device driver is caused by the hot plug PCI driver, IBMXSBD.NLM, or the device driver itself - debug switches available with the IBMXSBD.NLM driver - how interrupts are assigned - NCMCON issues such as no hot plug slots showed, or a query about waiting another ten seconds 5.2.1 How to determine if the problem is with the adapter driver or the hot plug driver. NOTE: There are no changes to the physical adapter required to support hot plug PCI operations. Many other adapter vendors will be providing drivers that meet the required Novell specifications for hot plug PCI in the future. However, there are times where you may have a driver and will not know whether the driver is hot plug compatible. The following is a list of methods to determine if a given driver supports hot plug PCI operations. 1. Almost any driver will allow a singular instance in the hot plug environment. 2. The device can be powered OFF through the NCMCON screen with a driver loaded or has a command-line option to remove a single instance of the driver. NOTE: A driver that forces all instances of itself to be unloaded is not considered a hot plug PCI driver. 3. Can the driver detect newly added adapters after a hot plug operation? NOTE: Many adapters will not detect new slot numbers because they do a static scan for cards on their first load. If a new card is added, the driver will report that no slots are available. Unloading the adapter (and sometimes the underlying support module, TOKENTSM.NLM, ETHERTSM.NLM, etc) and reloading them will allow the new slots to be seen in most instances. However, this will require unloading the driver for all adapters in the system. 4. Does the driver complain about PCI configuration problems? NOTE: Some older drivers do not allow for PCI configuration resources to be reallocated or changed. These drivers will report that one or more of the PCI resource requirements (IRQs, I/O ports, Memory, or Prefetchable Memory) are not correct/unavailable. 5. If new device drivers, support modules, and installation of the latest Support Packs does not fix the problem, then proceed to Section 5.2.2 to debug the IBMXSBD.NLM module. 5.2.2 Use of switches and log file for IBMXSBD driver There are a number of command line switches that can be used to create an IBMXSBD.LOG file in the specified directory on the hard drive. This file can be looked at to verify that the IBMXSBD.NLM module is working properly. The following are the command line switches and a description of each: The following commands are case sensitive, but they they require the leading hyphen. If multiple commands are given, they should be separated by spaces. (ie. LOAD IBMXSBD -DETECT DEBUG -COMMAND) /?,-?, ? These flags are used to print out the following list of switches. -DETECT This parameter will allow the IBMXSBD.NLM driver to detect if the hot plug PCI system and controller are detected. The driver will print a message to the system console as to its finding and then terminate. This flag cannot be used with any other flag. -DBGALL This parameter must be listed for any debug actions to be logged to the IBMXSBD.LOG file. Various system information including version of NetWare, PCI BIOS discovery, Novell Event Bus (NEB) events, Novell HIN numbers, driver deregistration, memory release traces and the PCI Interrupt Routing Options table are documented. If a problem arises where you must create the IBMXSBD.LOG file, the file can be submitted as part of the call to the HelpCenter for further diagnostic help. 5.2.3 Interrupt handling Interrupts are handled in the following manner: - Only interrupts 9, 10, and 11 will be assigned to hot plug PCI devices. These options are not settable. If during BIOS setup, these three interrupts are reserved for ISA Legacy devices, all hot plug PCI operations will fail. - IRQ 15 will never be assigned to hot plug PCI devices unless an adapter already installed in the server of the same type has already been assigned to IRQ 15. - Interrupts are preserved from the MPS table located in the extended BIOS data area (EBDA) of the system. - For slots that do not have an adapter in them at boot, interrupts are assigned according to the following formula: - If another device that matches the vendorID and deviceID in the PCI Configuration header is found, then the newly added adapter will receive the same interrupt as the other device. - An unused interrupt between 9, 10, and 11 is assigned. - The least used interrupt between interrupts 9, 10, and 11 is assigned. NOTE: There may be instances where a LAN and SCSI controller may be assigned the same interrupt. If this occurs, the LAN adapter may fail to handle interrupts correctly when the SCSI controller accesses the DOS partition and lose connections. If this occurs, it will be necessary to down the server, reboot it, go into System Setup, and manually assign the interrupts so the devices do not share interrupts. 5.2.4 NCMCON issues - If NCMCON.NLM fails to report the correct bus speed at which any of the busses is operating, exit NCMCON and reload NCMCON. - If NCMCON.NLM fails to show any slots with 'Yes' in the Hot Plug field, the IBMXSBD.NLM module is not loaded. To fix the problem, exit NCMCON, load IBMXSBD at the system console, and reload NCMCON. - If you receive a message about Slot Status not being available with a query to wait another 10 seconds, NCM.NLM or one of its support modules, IOCONFIG.NLM or NEB.NLM is not loaded. To fix this problem, exit NCMCON, load the appropriate support modules, load NCM, then NCMCON. It is not necessary for NCM or NCMCON to be loaded for the IBMXSBD driver to load properly. - If a fully supported ODI v3.31 LAN device driver cannot be unloaded, verify that ODINEB.NLM is loaded. It can be loaded at the system console after NEB.NLM is loaded. - All SCSI controller drivers must be unloaded from the system console. The following command to remove a single adapter and its volumes from the system: REMOVE STORAGE ADAPTER Ax where x is the adapter number as determined by the scan order of the PCI buses in the system. All integrated devices will be assigned first, followed by adapters in PCI Bus 0, PCI Bus 1, etc. - When using failover pairs in conjunction with hot plug PCI, you should not setup failover pairs using the Netfinity Ethernet Adapter 2 and the Netfinity Fault Tolerant Ethernet Adapter as two independent pairs. Failover pairs should consist of one adapter type or the other, but not both simultaneously. 6.0 Disclaimer ---------------------------- 6.1 THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. 6.2 Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation.