HP OpenVMS Systems Documentation

This section describes LAN Communication Problems and how to address them.

F.2.1 Symptoms

Communication trouble in OpenVMS Cluster systems may be indicated by symptoms such as the following:

• Poor performance
• Console messages
  • "Virtual circuit closed" messages from PEA0 (PEDRIVER) on the console
  • "Connection loss" OPCOM messages on the console
  • CLUEXIT bugchecks
  • "Excessive packet losses on LAN Path" messages on the console
• Repeated loss of a virtual circuit or multiple virtual circuits over a short period of time (fewer than 10 minutes)

Before you initiate complex diagnostic procedures, do not overlook the obvious. Always make sure the hardware is configured and connected properly and that the network is started. Also, make sure system parameters are set correctly on all nodes in the OpenVMS Cluster.

F.2.2 Traffic Control

Keep in mind that an OpenVMS Cluster system generates substantially heavier traffic than other LAN protocols. In many cases, cluster behavior problems that appear to be related to the network might actually be related to software, hardware, or user errors. For example, a large amount of traffic does not necessarily indicate a problem with the OpenVMS Cluster network. The amount of traffic generated depends on how the users utilize the system and the way that the OpenVMS Cluster is configured with additional interconnects (such as DSSI and CI).

If the amount of traffic generated by the OpenVMS Cluster exceeds the expected or desired levels, then you might be able to reduce the level of traffic by:

• Shifting the user load between machines
• Adding LAN segments and reconfiguring the LAN connections across the OpenVMS Cluster system

Prior to OpenVMS Version 7.3, an SCS virtual circuit closure was the first indication that a LAN path had become unusable. In OpenVMS Version 7.3, whenever the last usable LAN path is losing packets at an excessive rate, PEDRIVER displays the following console message:

%PEA0, Excessive packet losses on LAN path from local-device-name 
 to device-name on REMOTE NODE node-name 

This message is displayed when PEDRIVER recently had to perform an excessively high rate of packet retransmissions on the LAN path consisting of the local device, the intervening network, and the device on the remote node. The message indicates that the LAN path has degraded and is approaching, or has reached, the point where reliable communications with the remote node are no longer possible. It is likely that the virtual circuit to the remote node will close if the losses continue. Furthermore, continued operation with high LAN packet losses can result in significant loss in performance because of the communication delays resulting from the packet loss detection timeouts and packet retransmission.

The corrective steps to take are:

1. Check the local and remote LAN device error counts to see whether a problem exists on the devices. Issue the following commands on each node:

   $ SHOW DEVICE local-device-name $ MC SCACP SCACP> SHOW LAN device-name $ MC LANCP LANCP> SHOW DEVICE device-name/COUNTERS

2. If device error counts on the local devices are within normal bounds, contact your network administrators to request that they diagnose the LAN path between the devices.

If the symptoms and preliminary diagnosis indicate that you might have a network problem, troubleshooting LAN communication failures should start with the step-by-step procedures described in Appendix C. Appendix C helps you diagnose and solve common Ethernet and FDDI LAN communication failures during the following stages of OpenVMS Cluster activity:

• When a computer or a satellite fails to boot
• When a computer fails to join the OpenVMS Cluster
• During run time when startup procedures fail to complete
• When a OpenVMS Cluster hangs

The procedures in Appendix C require that you verify a number of parameters during the diagnostic process. Because system parameter settings play a key role in effective OpenVMS Cluster communications, Section F.2.6 describes several system parameters that are especially important to the timing of LAN bridges, disk failover, and channel availability.

F.2.5 Tracing Intermittent Errors

Because PEDRIVER communication is based on channels, LAN network problems typically fall into these areas:

• Channel formation and maintenance
  Channels are formed when HELLO datagram messages are received from a remote system. A failure can occur when the HELLO datagram messages are not received or when the channel control message contains the wrong data.
• Retransmission
  A well-configured OpenVMS Cluster system should not perform excessive retransmissions between nodes. Retransmissions between any nodes that occur more frequently than once every few seconds deserve network investigation.

Diagnosing failures at this level becomes more complex because the errors are usually intermittent. Moreover, even though PEDRIVER is aware when a channel is unavailable and performs error recovery based on this information, it does not provide notification when a channel failure occurs; PEDRIVER provides notification only for virtual circuit failures.

However, the Local Area OpenVMS Cluster Network Failure Analysis Program (LAVC$FAILURE_ANALYSIS), available in SYS$EXAMPLES, can help you use PEDRIVER information about channel status. The LAVC$FAILURE_ANALYSIS program (documented in Appendix D) analyzes long-term channel outages, such as hard failures in LAN network components that occur during run time.

This program uses tables in which you describe your LAN hardware configuration. During a channel failure, PEDRIVER uses the hardware configuration represented in the table to isolate which component might be causing the failure. PEDRIVER reports the suspected component through an OPCOM display. You can then isolate the LAN component for repair or replacement.

Reference: Section F.8 addresses the kinds of problems you might find in the NISCA protocol and provides methods for diagnosing and solving them.

F.2.6 Checking System Parameters

Table F-4 describes several system parameters relevant to the recovery and failover time limits for LANs in an OpenVMS Cluster.

Note: The TIMVCFAIL system parameter, which optimizes the amount of time needed to detect a communication failure, is not recommended for use with LAN communications. This parameter is intended for CI and DSSI connections. PEDRIVER (which is for Ethernet and FDDI) usually surpasses the detection provided by TIMVCFAIL with the listen timeout of 8 to 9 seconds.

F.2.7 Channel Timeouts

Channel timeouts are detected by PEDRIVER as described in Table F-5.

This section describes how to use SDA to monitor LAN or IP communications.

F.3.1 Isolating Problem Areas

If your system shows symptoms of intermittent failures during run time, you need to determine whether there is a network problem or whether the symptoms are caused by some other activity in the system.

Generally, you can diagnose problems in the NISCA protocol or the network using the OpenVMS System Dump Analyzer utility (SDA). SDA is an effective tool for isolating problems on specific nodes running in the OpenVMS Cluster system.

Reference: The following sections describe the use of some SDA commands and qualifiers. You should also refer to the HP OpenVMS System Analysis Tools Manual or the OpenVMS VAX System Dump Analyzer Utility Manual for complete information about SDA for your system.

F.3.2 SDA Command SHOW PORT

The SDA command SHOW PORT provides relevant information that is useful in troubleshooting PEDRIVER and LAN adapters in particular. Begin by entering the SHOW PORT command, which causes SDA to define cluster symbols. Example F-1 illustrates how the SHOW PORT command provides a summary of OpenVMS Cluster data structures.

$ ANALYZE/SYSTEM
SDA> SHOW PORT
VAXcluster data structures 
-------------------------- 
 
                  --- PDT Summary Page --- 
 
 PDT Address          Type         Device          Driver Name 
 -----------          ----         -------         ----------- 
 
  80C3DBA0             pa          PAA0            PADRIVER 
  80C6F7A0             pe          PEA0            PEDRIVER

To examine information about the virtual circuit (VC) that carries messages between the local node (where you are running SDA) and another remote node, enter the SDA command SHOW PORT/VC=VC_remote-node-name. Example F-2 shows how to examine information about the virtual channel running between a local node and the remote node, NODE11.

SDA> SHOW PORT/VC=VC_NODE11
VAXcluster data structures 
-------------------------- 
                 --- Virtual Circuit (VC) 98625380 --- 
Remote System Name:  NODE11  (0:VAX)     Remote SCSSYSTEMID:  19583 
Local System ID:  217 (D9)              Status: 0005 open,path 
------ Transmit -------  ----- VC Closures -----  (7)--- Congestion Control ---- 
Msg Xmt(1)      46193196  SeqMsg TMO            0  Pipe Quota/Slo/Max(8) 31/ 7/31 
  Unsequence          3  CC DFQ Empty          0  Pipe Quota Reached(9)   213481 
  Sequence     41973703  Topology Change(5)     0  Xmt C/T(10)              0/1984 
  ReXmt(2)       128/106  NPAGEDYN Low(6)        0  RndTrp uS(11)        18540+7764 
  Lone ACK      4219362                           UnAcked Msgs                0 
Bytes Xmt     137312089                           CMD Queue Len/Max        0/21 
------- Receive -------  - Messages  Discarded -  ----- Channel Selection ----- 
Msg Rcv(3)      47612604  No Xmt Chan           0  Preferred Channel    9867F400 
  Unsequence          3  Rcv Short Msg         0  Delay Time           FAAD63E0 
  Sequence     37877271  Illegal Seq Msg       0  Buffer Size              1424 
  ReRcv(4)         13987  Bad Checksum          0  Channel Count              18 
  Lone ACK      9721030  TR DFQ Empty          0  Channel Selections      32138 
  Cache             314  TR MFQ Empty          0  Protocol                1.3.0 
  Ill ACK             0  CC MFQ Empty          0  Open(12) 8-FEB-1994 17:00:05.12 
Bytes Rcv    3821742649  Cache Miss            0  Cls(13) 17-NOV-1858 00:00:00.00 

The SHOW PORT/VC=VC_remote-node-name command displays a number of performance statistics about the virtual circuit for the target node. The display groups the statistics into general categories that summarize such things as packet transmissions to the remote node, packets received from the remote node, and congestion control behavior. The statistics most useful for problem isolation are called out in Example F-2 and described in Table F-6.

Note: The counters shown in Example F-2 are stored in fixed-size fields and are automatically reset to 0 when a field reaches its maximum value (or when the system is rebooted). Because fields have different maximum sizes and growth rates, the field counters are likely to reset at different times. Thus, for a system that has been running for a long time, some field values may seem illogical and appear to contradict others.