HP OpenVMS I/O User's Reference Manual

Optional Features for Improving I/O Performance

Fast Path (Alpha Only)

Fast I/O  

The Fast I/O services support 64-bit addresses for data transfersto and from disk and tape devices.

While Fast I/O services are available on OpenVMS VAX, theperformance advantage applies only to OpenVMS Alpha. OpenVMS VAXhas a run-time library (RTL) compatibility package that translatesthe Fast I/O service requests to $QIO system service requests, soone set of source code can be used on both VAX and Alpha systems.

FastI/O Benefits  

The performance benefits of Fast I/O result from streamlininghigh-volume I/O requests. The Fast I/O system service interfacesare optimized to avoid the overhead of general-purpose services.For example, I/O request packets (IRPs) are now permanently allocatedand used repeatedly for I/O rather than allocated and deallocatedanew for each I/O.

The greatest benefits stem from having user data buffers anduser I/O status structures permanently locked down and mapped usingsystem space. This allows Fast I/O to do the following:

• For direct I/O, avoid per-I/O bufferlockdown or unlocking.



• For buffered I/O, avoid allocation and deallocationof a separate system buffer, because the user buffer is always addressable.



• Complete Fast I/O operations at IPL 8, thereby avoidingthe interrupt chaining usually required by the more general-purpose$QIO system service. For each I/O, this eliminates the IPL 4 IOPOSTinterrupt and a kernel AST.

In total, Fast I/O services eliminate four spinlock acquisitionsper I/O (two for the MMG spinlock and two for the SCHED spinlock).The reduction in CPU cost per I/O is 20 percent for uniprocessorsystems and 10 percent for multiprocessor systems.

UsingBuffer Objects  

The lockdown of user-process data structures is accomplishedby buffer objects. A "buffer object" is process memorywhose physical pages have been locked in memory and double-mappedinto system space. After creating a buffer object, the process remainsfully pageable and swappable and the process retains normal virtualmemory access to its pages in the buffer object.

If the buffer object contains process data structures to bepassed to an OpenVMS system service, the OpenVMS system can usethe buffer object to avoid any probing, lockdown, and unlockingoverhead associated with these process data structures. Additionally,double-mapping into system space allows the OpenVMS system directaccess to the process memory from system context.

To date, only the $QIO system service and the Fast I/O serviceshave been changed to accept buffer objects. For example, a bufferobject allows a programmer to eliminate I/O memory management overhead.On each I/O, each page of a user data buffer is probed and thenlocked down on I/O initiation and unlocked on I/O completion. Insteadof incurring this overhead for each I/O, it can be done once atbuffer object creation time. Subsequent I/O operations involvingthe buffer object can completely avoid this memory management overhead.

Two system services can be used to create and delete bufferobjects, respectively, and can be called from any access mode. Tocreate a buffer object, the $CREATE_BUFOBJ system service is called.This service expects as inputs an existing process memory rangeand returns a buffer handle for the buffer object. The buffer handleis an opaque identifier used to identify the buffer object on futureI/O requests. The $DELETE_BUFOBJ system service is used to deletethe buffer object and accepts as input the buffer handle. Althoughimage rundown deletes all existing buffer objects, it is good formfor the application to clean up properly.

A 64-bit equivalent version of the $CREATE_BUFOBJ system service($CREATE_BUFOBJ_64) can be used to create buffer objects from thenew 64-bit P2 or S2 regions. The $DELETE_BUFOBJ system service canbe used to delete 32-bit or 64-bit buffer objects.

Buffer objects require system management. Because buffer objectstie up physical memory, extensive use of buffer objects requiressystem management planning. All the bytes of memory in the bufferobject are deducted from a systemwide system parameter called MAXBOBMEM(maximum buffer object memory). System managers must set this parametercorrectly for the application loads that run on their systems.

The MAXBOBMEM parameter defaults to 100 Alpha pages, but forapplications with large buffer pools it will likely be set muchlarger. To prevent user-mode code from tying up excessive physicalmemory, user-mode callers of $CREATE_BUFOBJ must have a new systemidentifier, VMS$BUFFER_OBJECT_USER, assigned. This new identifieris automatically created in an OpenVMS Version 7.0 upgrade if thefile SYS$SYSTEM:RIGHTSLIST.DAT is present. The system manager canassign this identifier with the DCL command SET ACL command to aprotected subsystem or application that creates buffer objects fromuser mode. It may also be appropriate to grant the identifier toa particular user with the Authorize utility command GRANT/IDENTIFIER(for example, to a programmer who is working on a development system).

There is currently a restriction on the type of process memorythat can be used for buffer objects. Global section memory cannotbe made into a buffer object.

DifferencesBetween Fast I/O Services and $QIO  

The precise definition of high-volume I/O operations optimizedby Fast I/O services is important. I/O that does not comply withthis definition either is not possible with the Fast I/O servicesor is not optimized. The characteristics of the high-volume I/Ooptimized by Fast I/O services can be seen by contrasting the operation ofFast I/O system services to the $QIO system service as follows:

• The $QIO system service I/O statusblock (IOSB) is replaced by an I/O status area (IOSA) that is larger andquadword aligned. The transfer byte count returned in IOSA is 64bits, and the field is aligned on a quadword boundary. Unlike theIOSB, which is optional, the IOSA is required.



• User data buffers must be aligned to a 512-byteboundary.



• All user process structures passed to the Fast I/Osystem services must reside in buffer objects. This includes theuser data buffer and the IOSA.



• Only transfers that are multiples of 512 bytes aresupported.



• Only the following function codes are supported:IO$_READVBLK, IO$_READLBLK, IO$_WRITEVBLK, and IO$_WRITELBLK.



• Only I/O to disk and tape devices is optimized forperformance.



• No event flags are used with Fast I/O services.If application code must use an event flag in relation to a specificI/O, then the Event No Flag EFN (EFN$C_ENF) can be used. This eventflag is a no-overhead EFN that can be used in situations when anEFN is required by a system service interface but has no meaningto an application.
  
  For example, Fast I/O services do not use EFNs, so the applicationcannot specify a valid EFN associated with the I/O to the $SYNCHsystem service with which to synchronize I/O completion. To resolvethis issue, the application can call the $SYNCH system service passingas arguments: EFN$C_ENF and the address of the appropriate IOSA.Specifying EFN$C_ENF signifies to $SYNCH that no EFN is involved inthe synchronization of the I/O. Once the IOSA has been written witha status and byte count, return from the $SYNCH call occurs. TheIOSA is now the central point of synchronization for a given FastI/O (and is the only way to determine whether the asynchronous I/Ois complete).



• To minimize arguments passing overhead to theseservices, the $QIO parameters P3 through P6 are replaced by a singleargument that is passed directly by the Fast I/O system servicesto device drivers. For disk-like devices, this argument is the mediaaddress (VBN or LBN) of the transfer. For drivers with complex parameters,this argument is the address of a descriptor or of a buffer specificto the device and function.



• Segmented transfers are supported by Fast I/O butare not fully optimized. There are two major causes of segmentedtransfers. The first is disk fragmenting. While this can be an issue,it is assumed that sites seeking maximum performance have eliminatedthe overhead of segmenting I/O due to fragmentation.
  
  A second cause of segmenting is issuing an I/O that exceedsthe port's maximum limit for a single transfer. Transfers beyondthe port maximum limit are segmented into several smaller transfers.Some ports limit transfers to 64KB. If the application limits itstransfers to less than 64KB, this type of segmentation should notbe a concern.

UsingFast I/O Services  

The three Fast I/O system services are:

• $IO_SETUP---Sets up an I/O



• $IO_PERFORM[W]---Performs an I/O request



• $IO_CLEANUP--Cleans up an I/O request

UsingFandles  

A key concept behind the operation of the Fast I/O servicesis the file handle or fandle. A fandle isan opaque token that represents a "setup" I/O.A fandle is needed for each I/O outstanding from a process.

All possible setup, probing, and validation of arguments isperformed off the mainline code path during application startupwith calls to the $IO_SETUP system service. The I/O function, theAST address, the buffer object for the data buffer, and the IOSAbuffer object are specified on input to $IO_SETUP service, and afandle representing this setup is returned to the application.

To perform an I/O, the $IO_PERFORM system service is called,specifying the fandle, the channel, the data buffer address, theIOSA address, the length of the transfer, and the media address(VBN or LBN) of the transfer.

If the asynchronous version of this system service, $IO_PERFORM,is used to issue the I/O, then the application can wait for I/Ocompletion using a $SYNCH specifying EFN$C_ENF and the appropriateIOSA. The synchronous form of the system service, $IO_PERFORMW,is used to issue an I/O and wait for it to complete. Optimum performancecomes when the application uses AST completion; that is, the application doesnot issue an explicit wait for I/O completion.

To clean up a fandle, the fandle can be passed to the $IO_CLEANUPsystem service.

ModifyingExisting Applications  

Modifying an application to use the Fast I/O services requiresa few source-code changes. For example:

1. A programmeradds code to create buffer objects for the IOSAs and data buffers.
2. The programmer changes the application to use theFast I/O services. Not all $QIOs need to be converted. Only high-volumeread/write I/O requests should be changed.
   
   A simple example is a "database writer" program,which writes modified pages back to the database. Suppose the writercan handle up to 16 simultaneous writes. At application startup,the programmer would add code to create 16 fandles by 16 $IO_SETUPsystem service calls.
3. In the main processing loop within the databasewriter program, the programmer replaces the $QIO calls with $IO_PERFORMcalls. Each $IO_PERFORM call uses one of the 16 available fandles.While the I/O is in progress, the selected fandle is unavailablefor use with other I/O requests. The database writer is probablyusing AST completion and recycling fandle, data buffer, and IOSAonce the completion AST arrives.
   
   If the database writer routine cannot return until all dirtybuffers are written (that is, it must wait for all I/O completions),then $IO_PERFORMW can be used. Alternatively $IO_PERFORM calls canbe followed by $SYNCH system service calls passing the EFN$C_ENFargument to await I/O completions.
   
   The database writer will run faster and scale better becauseI/O requests now use less CPU time.
4. When the application exits, an $IO_CLEANUP systemservice call is done for each fandle returned by a prior $IO_SETUPsystem service call. Then the buffer objects are deleted. Imagerundown performs fandle and buffer object cleanup on behalf of theapplication, but it is good form for the application to clean upproperly.

I/OStatus Area (IOSA)  

The central point of synchronization for a given Fast I/Ois its IOSA. The IOSA replaces the $QIO system service's IOSB argument.Larger than the IOSB argument, the byte count field in the IOSAis 64 bits and quadword aligned. Unlike the $QIO system service,Fast I/O services require the caller to supply an IOSA and requirethe IOSA to be part of a buffer object.

The IOSA context field can be used in place of the $QIO systemservice ASTPRM argument. The $QIO ASTPRM argument is typically usedto pass a pointer back to the application on the completion ASTto locate the user context needed for resuming a stalled user-thread;however, for the $IO_PERFORM system service, the ASTPRM on the completionAST is always the IOSA. Because there is no user-settable ASTPRM,an application can store a pointer to the user-thread context forthis I/O in the IOSA context field and retrieve the pointer fromthe IOSA in the completion AST. )

$IO_SETUP  

The $IO_SETUP system service performs the setup of an I/Oand returns a unique identifier for this setup I/O, called a fandle,to be used on future I/Os. The $IO_SETUP arguments used to createa given fandle remain fixed throughout the life of the fandle. Thishas implications for the number of fandles needed in an application.For example, a single fandle can be used only for reads or onlyfor writes. If an application module has up to 16 simultaneous readsor writes pending, then potentially 32 fandles are needed to avoidany $IO_SETUP calls during mainline processing.

The $IO_SETUP system service supports an expedite flag, whichis available to boost the priority of an I/O among the other I/Orequests that have been handed off to the controller. Unrestraineduse of this argument is useless, because if all I/O is expedited,nothing is expedited. Note that this flag requires the use of ALTPRI andPHY_IO privilege.

$IO_PERFORM[W]  

The $IO_PERFORM[W] system service accepts a fandle and fiveother variable I/O parameters for the high-performance I/O operation.The fandle remains in use to the application until the $IO_PERFORMW returnsor if $IO_PERFORM is used until a completion AST arrives.

The CHAN argument to the fandle contains the data channelreturned to the application by a previous file operation. This argumentallows the application the flexibility of using the same fandlefor different open files on successive I/Os; however, if the fandleis used repeatedly for the same file or channel, then an internal optimizationwith $IO_PERFORM is taken.

Note that $IO_PERFORM was designed to have no more than sixarguments to take advantage of the OpenVMS Calling Standard,which specifies that calls with up to six arguments can be passedentirely in registers.

$IO_CLEANUP  

A fandle can be cleaned up by passing the fandle to the $IO_CLEANUPsystem service.

FastI/O FDT Routine (ACP_STD$FASTIO_BLOCK)  

Because $IO_PERFORM supports only four function codes, thissystem service does not use the generalized function decision table(FDT) dispatching that is contained in the $QIO system service.Instead, $IO_PERFORM uses a single vector in the driver dispatchtable called DDT$PS_FAST_FDT for the four supported functions. TheDDT$PS_FAST_FDT field is a FDT routine vector that indicates whetherthe device driver called by $IO_PERFORM is set up to handle FastI/O operations. A nonzero value for this field indicates that thedevice driver supports Fast I/O operations and that the I/O canbe fully optimized.

If the DDT$PS_FAST_FDT field is zero, then the driver is notset up to handle Fast I/O operations. The $IO_PERFORM system servicetolerates such device drivers, but the I/O is only slightly optimizedin this circumstance.

The OpenVMS disk and tape drivers that ship as part of OpenVMSVersion 7.0 have added the following line to their driver dispatchtable (DDTAB) macro:

FAST_FDT=ACP_STD$FASTIO_BLOCK,- ; Fast-IO FDT routine

If you have a disk or tape device driver that can handle FastI/O operations, you can add this DDTAB macro line to your driver.If you cannot use the standard Fast I/O FDT routine, ACP_STD$FASTIO_BLOCK,you can develop your own based on the model presented in this routine.

AdditionalInformation  

Refer to the HP OpenVMS System Services Reference Manualfor additional information about thefollowing Fast I/O system services:

$CREATE_BUFOBJ

$DELETE_BUFOBJ

$CREATE_BUFOBJ_64

$IO_SETUP

$IO_PERFORM

$IO_CLEANUP

To see a sample program that demonstrates the use of bufferobjects and the Fast I/O system services, refer to the IO_PERFORM.Cprogram in the SYS$EXAMPLES directory.

Fast Path (Alpha Only)