[an error occurred while processing this directive]
Software  >  OpenVMS Systems > Documentation > 83final > 6552

HP OpenVMS Systems Documentation
Content starts here

HP OpenVMS Availability Manager User's Guide

Previous Contents Index

Chapter 6
Performing Fixes on OpenVMS Nodes

Fixes allow you to resolve resource availability problems and improve system availability.

This chapter discusses the following topics:

  • Understanding fixes
  • Performing fixes

Caution
Performing certain fixes can have serious repercussions, including possible system failure. Therefore, only experienced system managers should perform fixes.

6.1 Understanding Fixes

When you suspect or detect a resource availability problem, in many cases you can use the Availability Manager to analyze the problem and to perform a fix to improve the situation.

Availability Manager fixes fall into these categories:

  • Node fixes
  • Process fixes
  • Cluster interconnect fixes

You can access fixes, by category, from the pages listed in Table 6-1.

Table 6-1 Accessing Availability Manager Fixes
Fix Category and Name Available from This Page
Node fixes:
Crash Node
Adjust Quorum
Node Summary
CPU
Memory Summary
I/O Process
SCA Port
SCA Circuit
LAN Virtual Circuit
LAN Path (Channel)
LAN Device
Process fixes:
General process fixes:
Delete Process
Exit Image
Suspend Process
Resume Process
Process Priority


Process memory fixes:

Purge Working Set (WS)
Adjust Working Set (WS)


Process limits fixes:

Direct I/O
Buffered I/O
AST
Open file
Lock
Timer
Subprocess
I/O Byte
Pagefile Quota
All of the process fixes are available from the following pages:
Memory Summary
I/O Process
CPU Process
Single Process
Cluster interconnect fixes: These fixes are available from the following lines of data on the Cluster Summary page (Figure 4-1):
- SCA Port:/ Adjust Priority Right-click a data item on the Local Port Data display line to display a menu. Then select Port Fix....
- SCA Circuit:/ Adjust Priority Right-click a data item on the Circuits Data display line to display a menu. Then select Circuit Fix....
LAN Virtual Circuit Summary:
Maximum Transmit Window Size
Maximum Receive Window Size
Checksumming
Compression
ECS Maximum Delay
Right-click a data item on the LAN Virtual Circuit Summary line to display a menu. Then select VC LAN Fix.... Alternatively, you can use the Fix menu on the LAN VC Details page.
LAN Path (Channel) Summary:
Adjust Priority
Hops
Right-click a data item on the LAN Path (Channel) Summary line to display a menu. Then select Fixes.... Alternatively, you can use the Fix menu on the Channel Details page.
LAN Device Details:
Adjust Priority
Set Maximum Buffer Size
Start LAN Device
Stop LAN Device
You can access these fixes in the following ways:
  • Right-click an item in the LAN Path (Channel) Summary category to display a menu. Then select LAN Device Details... to display pages containing Fix options.
  • Right-click an item in the LAN Device Summary page and then select LAN Device Fixes.....
  • Select Fixes... on the LAN Device Details page.

Table 6-2 summarizes various problems, recommended fixes, and the expected results of fixes.

Table 6-2 Summary of Problems and Matching Fixes
Problem Fix Result
Node resource hanging cluster Crash Node Node fails with operator-requested shutdown. See Section 6.2.2 for the crash dump footprint for this type of shutdown.
Cluster hung Adjust Quorum Quorum for cluster is adjusted.
Process looping, intruder Delete Process Process no longer exists.
Endless process loop in same PC range Exit Image Exits from current image.
Runaway process, unwelcome intruder Suspend Process Process is suspended from execution.
Process previously suspended Resume Process Process starts from point it was suspended.
Runaway process or process that is overconsuming Process Priority Base priority changes to selected setting.
Low node memory Purge Working Set (WS) Frees memory on node; page faulting might occur for process affected.
Working set too high or low Adjust Working Set (WS) Removes unused pages from working set; page faulting might occur.
Process quota has reached its limit and has entered RWAIT state Adjust Process Limits Process limit is increased, which in many cases frees the process to continue execution.
Process has exhausted its pagefile quota Adjust Pagefile Quota Pagefile quota limit of the process is adjusted.

Most process fixes correspond to an OpenVMS system service call, as shown in the following table:

Process Fix System Service Call
Delete Process $DELPRC
Exit Image $FORCEX
Suspend Process $SUSPND
Resume Process $RESUME
Process Priority $SETPRI
Purge Working Set (WS) $PURGWS
Adjust Working Set (WS) $ADJWSL
Adjust process limits of the following:
Direct I/O (DIO)
Buffered I/O (BIO)
Asynchronous system trap (AST)
Open file (FIL)
Lock queue (ENQ)
Timer queue entry (TQE)
Subprocess (PRC)
I/O byte (BYT)
None

Note
Each fix that uses a system service call requires that the process execute the system service. A hung process has the fix queued to it, and the fix does not execute until the process is operational again.

Be aware of the following facts before you perform a fix:

  • You must have write access to perform a fix. To perform LAN fixes, you must have control access.
  • You cannot undo many fixes. For example, after using the Crash Node fix, the node must be rebooted (either by the node if the node reboots automatically, or by a person performing a manual boot).
  • Do not apply the Exit Image, Delete Process, or Suspend Process fix to system processes. Doing so might require you to reboot the node.
  • Whenever you exit an image, you cannot return to that image.
  • You cannot delete processes that have exceeded their job or process quota.
  • The Availability Manager ignores fixes applied to the SWAPPER process.

How to Perform Fixes

Standard OpenVMS privileges restrict users' write access. When you run the Data Analyzer, you must have the CMKRNL privilege to send a write (fix) instruction to a node with a problem.

The following options are displayed at the bottom of all fix pages:

Option Description
OK Applies the fix and then exits the page. Any message associated with the fix is displayed in the Event pane.
Cancel Cancels the fix.
Apply Applies the fix and does not exit the page. Any message associated with the fix is displayed in the Return Status section of the page and in the Event pane.

The following sections explain how to perform node fixes and process fixes.

6.2 Performing Node Fixes

One node fix has the special ability among Availability Manager fixes to allow you to deliberately fail (or crash) a node. Another node fix allows you to adjust cluster quorum.

To perform a node fix, follow these steps:

  1. On the Node Summary, CPU, Memory, or I/O page, select the Fix menu.
  2. Select Fix Options.

6.2.1 Adjust Quorum

The default node fix displayed is the Adjust Quorum fix, which forces a node to recalculate the quorum value. This fix is the equivalent of the Interrupt Priority level C (IPC) mechanism used at system consoles for the same purpose. The fix forces the adjustment for the entire cluster so that each node in the cluster has the same new quorum value.

The Adjust Quorum fix is useful when the number of votes in a cluster falls below the quorum set for that cluster. This fix allows you to readjust the quorum so that it corresponds to the current number of votes in the cluster.

The Adjust Quorum page is shown in Figure 6-1.

Figure 6-1 Adjust Quorum

6.2.2 Crash Node

Caution
The Crash Node fix is an operator-requested bugcheck from the driver. It takes place as soon as you click OK in the Crash Node fix. After you perform this fix, the node cannot be restored to its previous state. After a crash, the node must be rebooted.

When you select the Crash Node option, the Availability Manager displays the Crash Node page, shown in Figure 6-2.

Figure 6-2 Crash Node

Note
Because the node cannot report a confirmation when a Crash Node fix is successful, the crash success message is displayed after the timeout period for the fix confirmation has expired.

Recognizing a System Failure Forced by the Availability Manager

Because a user with suitable privileges can force a node to fail from the Data Analyzer by using the Crash Node fix, system managers have requested a method for recognizing these particular failure footprints so that they can distinguish them from other failures. These failures all have identical footprints: they are operator-induced system failures in kernel mode at IPL 8. The top of the kernel stack is similar the following display: 


                SP => Quadword system address
                      Quadword data
                      1BE0DEAD.00000000
                      00000000.00000000
                      Quadword data            TRAP$CRASH
                      Quadword data            SYS$RMDRIVER + offset

6.3 Performing Process Fixes

To perform a process fix, follow these steps:

  1. On the Memory or I/O page, right-click a process name.
  2. Click Fix Options.
    The Availability Manager displays these Process tabs:
    Process General
    Process Memory
    Process Limits
  3. Click one of these tabs to bring it to the front.
  4. Click the down arrow to display the process fixes in this group, as shown in Figure 6-3, where the Process General tab has been chosen.

    Figure 6-3 Process General Options

  5. Select a process fix (for example, Process Priority, shown in Figure 6-3), to display a fix page.

Some of the fixes, such as Process Priority, require you to use a slider to change the default value. When you finish setting a new process priority, click Apply at the bottom of the page to apply that fix.

6.3.1 General Process Fixes

The following sections describe Availability Manager general process fixes. These fixes include instructions telling how to delete, suspend, and resume a process.

6.3.1.1 Delete Process

In most cases, a Delete Process fix deletes a process. However, if a process is waiting for disk I/O or is in a resource wait state (RWAST), this fix might not delete the process. In this situation, it is useless to repeat the fix. Instead, depending on the resource the process is waiting for, a Process Limit fix might free the process. As a last resort, reboot the node to delete the process.

Caution
Deleting a system process can cause the system to hang or become unstable.

When you select the Delete Process option, the Availability Manager displays the page shown in Figure 6-4.

Figure 6-4 Delete Process

After reading the explanation, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

6.3.1.2 Exit Image

Exiting an image on a node can stop an application that a user requires. Make sure you check the Single Process page before you exit an image to determine which image is running on the node.

Caution
Exiting an image on a system process could cause the system to hang or become unstable.

When you select the Exit Image option, the Availability Manager displays the page shown in Figure 6-5.

Figure 6-5 Exit Image Page

After reading the explanation in the page, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

6.3.1.3 Suspend Process

Suspending a process that is consuming excess CPU time can improve perceived CPU performance on the node by freeing the CPU for other processes to use. (Conversely, resuming a process that was using excess CPU time while running might reduce perceived CPU performance on the node.)

Caution
Do not suspend system processes, especially JOB_CONTROL, because this might make your system unusable. (For more information, see HP OpenVMS Programming Concepts Manual, Volume I.)

When you select the Suspend Process option, the Availability Manager displays the page shown in Figure 6-6.

Figure 6-6 Suspend Process

After reading the explanation, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

6.3.1.4 Resume Process

Resuming a process that was using excess CPU time while running might reduce perceived CPU performance on the node. (Conversely, suspending a process that is consuming excess CPU time can improve perceived CPU performance by freeing the CPU for other processes to use.)

When you select the Resume Process option, the Availability Manager displays the page shown in Figure 6-7.

Figure 6-7 Resume Process

After reading the explanation, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

6.3.1.5 Process Priority

If the priority of a compute-bound process is too high, the process can consume all the CPU cycles on the node, affecting performance dramatically. On the other hand, if the priority of a process is too low, the process might not obtain enough CPU cycles to do its job, also affecting performance.

When you select the Process Priority option, the Availability Manager displays the page shown in Figure 6-8.

Figure 6-8 Process Priority

To change the base priority for a process, drag the slider on the scale to the number you want. The current priority number is displayed in a small box above the slider. You can also click the line above or below the slider to adjust the number by 1.

When you are satisfied with the new base priority, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

6.3.2 Process Memory Fixes

The following sections describe the Availability Manager fixes you can use to correct process memory problems--- Purge Working Set and Adjust Working Set fixes.

6.3.2.1 Purge Working Set

This fix purges the working set to a minimal size. You can use this fix to reclaim a process's pages that are not in active use. If the process is in a wait state, the working set remains at a minimal size, and the purged pages become available for other uses. If the process becomes active, pages the process needs are page-faulted back into memory, and the unneeded pages are available for other uses.

Be careful not to repeat this fix too often: a process that continually reclaims needed pages can cause excessive page faulting, which can affect system performance.

When you select the Purge Working Set option, the Availability Manager displays the page shown in Figure 6-9.

Figure 6-9 Purge Working Set

After reading the explanation on the page, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

6.3.2.2 Adjust Working Set

Adjusting the working set of a process might prove to be useful in a variety of situations. Two of these situations are described in the following list.
  • If a process is page-faulting because of insufficient memory, you can reclaim unused memory from other processes by decreasing the working set of one or more of them.
  • If a process is page-faulting too frequently because its working set is too small, you can increase its working set.

Caution
If the automatic working set adjustment is enabled for the system, a fix to adjust the working set size disables the automatic adjustment for the process. For more information, see OpenVMS online help for SET WORKING_SET/ADJUST, which includes /NOADJUST.

When you select the Adjust Working Set fix, the Availability Manager displays the page shown in Figure 6-10.

Figure 6-10 Adjust Working Set

To perform this fix, use the slider to adjust the working set to the limit you want. You can also click the line above or below the slider to adjust the number by 1.

When you are satisfied with the new working set limit, click Apply at the bottom of the page to apply the fix. A message displayed on the page indicates that the fix has been successful.

Previous Next Contents Index