|
HP OpenVMS Systemsask the wizard |
|
The Question is: How can we access and process the information stored in an OpenVMS RMS Journaling file using a 3GL program? We basically need to know the FILE FORMAT (spec) of OpenVMS RMS Journaling JOURNAL FILES. (OpenVMS 6.2+) We are attempting to create a data warehousing/management/transfer system using existing OpenVMS RMS keyed database files by nightly batch "mirroring" the data to another system/file/database format. (unix/relational) To make the process work as productively as possible, in regard to speed and accuracy, we are in need of transaction information regarding file modifications to the RMS files. Specifically, we need to know when a record has been updated, added and deleted. We are only in need of the "raw" modification method, as opposed to actual field or record changes. That is, that "a record" was added, deleted, or modified, which specific record (RID?), and, if possible, when the record transaction occurred -- date/time. We have researched this problem and have not found any Compaq (DEC) or third-party products that would help us solve our problem. Therefore, we have come up with a solution that we believe would be the best way to solve the problem, but we are in need of additional information. Our solution is to use RMS Journaling "journal file" information to detect when a record has been added, deleted, or modified. However, we need to know what the journaling file format is, so that we can write a (VAX BASIC/C) program to read and process the information, which will then be used to provide incremental data transfers (reducing size/time problems in a multiple large file system). Can you provide us with the JOURNALING FILE FORMAT SPECification and/or any other information that would help us in this regard? We could even use a report output generated from the journaling file, parsing the text output to extract the information we require, if something like that was available. Any hints, tips, suggestions and other information provided by any persons would be greatly appreciated, and may be sent via e-mail to Bernard Buckley at: buckleybe@butte.cc.ca.us Thank you. The Answer is :
You do not and should not directly access the contents of the RMS
journaling product's journal files. The contents, operation, and
format of RMS journal files are undocumented and subject to change
without notice.
If you wish to perform a rapid backup of your application, please look
to coordinate the your application file system activities -- via lock,
via the distributed transaction manager, or otherwise -- with a utility
or tool based on the BACKUP API or other archival tool or product that
is specific to the application. This approach typically involves
synchronous flushes of RMS caches via calls to sys$flush and synchronous
flushes of global memory via $updsec[w], and/or (at its simplest and
slowest) calls to temporarily close files via sys$close or similar,
perform the BACKUP, and then re-open the files.
Various products -- including Oracle Rdb -- utilize an approach that is
based on an application-integrated BACKUP.
OpenVMS uses an approach based on a lock for operations such as extending
files -- when the volume lock is set via ANALYZE/DISK/REPAIR or other
tool, various file activity incompatible with the analysis and repair
is temporarily halted, while various compatible activity can continue.
Once the lock is released, activity continues.
One option for quick(er) BACKUP operations involves quiescing the
application and flushing the buffers, and then splitting a volume
off a shadowset -- minicopy and related work in OpenVMS specifically
targets shortening the window of quiescence required for the BACKUP.
Also consider acquiring a commercial relational database product.
Also consider acquiring one of the ODBC/JDBC packages or another other
distributed or served network-based approach -- please see the OpenVMS
FAQ and various discussions here in the OpenVMS Ask The Wizard area of
ODBC/JDBC -- and directly access the contents of the existing RMS file
from the remote system(s). This remote access can be performed via a
commercial package, or via a an application-specific network server
process.
Note that the RMS Journal files currently do not maintain information on
record modification operations, these files only contain the after images
of the changed buckets as a result of the record modification.
For example: Bucket number 6 in an index file is 6 blocks big, and has
record with primary key A,B and C, for a total of 2300 bytes. Record B
is now updated. The AI file will get a copy of the first 2300 (out of
3072) bytes of bLOck 6 data containing old records A nd C and the new
data for record B. There will be no indication what-so-ever that record
B caused the change.
To establish the record context, you would have to intercept the AI bucket
as is was about to be appllied to a datafile backup and compare old and
new data. This is clearly non-trivial. Once you have achieved this, you
you would acquire the date and time stamps as reward in the journal file
record header data.
The most expedient mechanism the OpenVMS Wizard can provide that would
reconstitute the AI data back to record operations would require three
copies of the file:
1 - Live data
2 - Master backup
3 - copy of backup with AI file applied.
Now write a program to compare file 2 and file 3
Records found in 3 but not in 2 become a $PUT.
Records found in 2 but not in 3 become a $DELETE.
Records which are different become an $UPDATE.
To handle this properly, you would probably need to monitor KEY and RFA
values. You would likely loose temporal insight, although you could
choose to apply the AI file up until a certain timestamp.
The OpenVMS Wizard would strongly recommend encapsulating RMS record
operation calls within application routines (MY_PUT versus SYS$PUT), and
those jacket routines would call onwards to RMS for the real operation
and would log the changes in a sequential file. This is still not trivial,
but is feasible, supportable, and documented.
|