                        Creating & Managing Databases

                                     "This meane and unrefined stuffe of
                                      mine will make your glistering
                                      gold but more to shine"
                                             Anne Bradstreet, 1612-1672

Introduction

          The GOLDDB unit provides a complete set of routines for
     developing database applications such as a customer list or parts
     inventory. Developers of database toolkits often complicate the
     final product by offering several levels of access to database
     information. The simplest level doesn't allow enough flexibility
     while the deepest level becomes so complicated that a complete
     understanding of database management and indexing is required.
     Keeping the developer in mind, we have attempted to deliver real
     flexibility while making the GOLDDB unit as simple as possible.

File Support

          The GOLDDB unit supports the extremely popular dBase III+ file
     format (dBase is a trademark of Borland International). Only one
     index file per database is permitted, and the index file name uses
     the associated database's file name while appending our own .GDX
     (Gold Database Index) extension. For example, if the database was
     named PARTS.DBF, the index would be named PARTS.GDX. Behind the
     scenes, Gold uses a custom implementation of Btree indexing.

          GOLDDB also supports the standard .DBT memo file.

General Database Activities

     Following are the general routines that relate to the general database access.

     DBFExist(FN: PathStr): boolean;

     Determines whether or not a database file exists. Sometimes the user may not provide the .DBF extension, so DBFExist forces this extension before making its determination. To ensure that the application flows smoothly as files are opened, it is recommended that you check for their existence first.

     The following code sample demonstrates checking for the existence of the data file.

          begin { main }
             if DBFExist('DEMCUST') then
             begin
                Handle := DbOpenDataSet('DEMCUST.DBF');
                if (Handle = 0) then
                   PromptOK(' DATA ERROR ','Unable to open DEMCUST.DBF
                                         or one of its related files.')
                else
                   { Data activity takes place here }
                if Handle > 0 then
                   DbCloseDatabase(Handle);
             end;
             Clear(LightGrayOnBlack,' ');
          end.

          The above code illustrates the use of the database handle.
     Since Gold allows you to open multiple databases at any one time,
     you need some way of informing Gold which database you want to work
     with. When a database is opened, Gold returns a handle. The
     procedure DbSetActiveDatabase is used to set the focus to a
     specific database, by passing a specific handle. All database
     operations are directed to the database with focus. Notice that
     this same handle is used in the call to DbCloseDatabase. These
     primary database arrangement functions are defined as follows:

     DbOpenDataSet(DBFile: pathstr): integer;

          Opens, validates and assigns a handle to the specified
     database. If it exists, the associated index file is also opened
     and validated. The integer value that is returned becomes the
     handle that identifies the database just opened, i.e. the database
     with focus. A returned handle value of zero implies that the data
     file did not validate correctly and may be corrupt.

          GOLDDB maintains an internal list of pointers, each of which
     individually points to the associated attributes of the opened data
     files. The file handle is actually the value of the associated
     sequential entry within this list.

     DbSetActiveDataBase(Handle:integer);

          If more than one database is opened at a time, focus may be
     changed to another file by specifying its assigned handle. All
     subsequent activity associated with a database is focused on the
     new file handle until focus is changed again or the file is closed.

     DbCloseDataBase(Handle: integer);

          After all activity has been completed surrounding the data, a
     database must be closed to free up the memory that was allocated
     for its use. The handle identifies the database that will be
     closed.

     Listed below are some other common data management routines.

     DbSetFullStrings(On: boolean);

          Inside a dBase file, all data is stored as "space padded"
     strings. This procedure instructs Gold on how to return data from
     each field. Setting the parameter to FALSE will strip all the
     trailing spaces from the data before it is returned to the
     application. The default is TRUE, so that the complete length of
     the field is returned each time.

     DbIndexedField: integer;

          Returns the field used for the database index. A value of zero
     implies that the index file is not valid or does not exist.

     DbSeqSearch(var RecNo: longint; FieldNo: integer;
                                           SearchTxt: String): boolean;

          Instructs Gold to perform a sequential search fo the database,
     i.e. a non-indexed search which checks the field in natural or
     stored order. A database index is not used for this search, even if
     one exists. Passing a value of zero to RecNo signals the search to
     begin at record one, any other value begins at the specified
     record.

     DbCloseAllDatabases;

          If more than one database has been opened to perhaps compare
     data or design some relational report about the data, all opened
     databases may be closed by calling this one procedure. This is in
     lieu of running down the list of opened data files and closing them
     one at a time.

What's the Header?

          Each .DBF file contains certain information which defines the
     database characteristics along with the fields it contains. This
     data is stored at the beginning of the DBS file and is referred to
     as the header. The first 32 bytes of the header is information
     which specifically pertains to the .DBF file, e.g. version number,
     last data it was modified, etc. There is an additional 32 bytes
     added to the header for each field within the database.

          The GOLDDB unit maintains a record depicting the first 32
     bytes of the header and is defined as follows:

          HeaderInfo = record
             VersionNumber: byte;
             Update: array [1..3] of byte;
             NbrRec: longint;
             HdrLen: integer;
             RecLen: integer;
             Reserved: array [1..20] of char;
          end;

          You do not need to access this record directly. Use the
     following list of routines to retrieve this information.

     DbGetVersion: byte;

          Returns the version byte of the database file. This is
     generally one of two values, either a 3 ($03) or a 131 ($83). A
     value of 131 ($83) indicates that at least one memo field exists in
     the database. Some third party dBase-clone vendors have decided to
     use this version byte as their finger print, modifying it to their
     own value. When a database is opened Gold  inspects the version
     byte to ensure that GOLDDB is using a true dBase III+ compatible
     file.

     DbGetUpDate: dates;

          Another part of the header is a date which represents the last
     time a value was changed in the database. This function returns a
     longint value of type dates, that may be converted to a string
     value using JultoStr.

     DbTotalFields: integer;

     Represents the total number of fields defined in the database.

     DbGetNumRecs: longint;

          Each time a record is added to the data file, this value is
     incremented by one. It represents all records whether or not they
     have been marked as deleted.

     DbGetHdrLen: integer;

          The header length is made up of a multiple of 32 byte sections
     and is terminated with a single <CR> ($0D) character. The first 32
     byte section contains the file specific information while the
     remaining sections are field definitions. The header length is
     inclusive of the end-of-header character.

     DbGetRecLen: word;

          The first byte of each record is the status byte. The record
     length is defined as all the individual field lengths added
     together plus one byte (status byte).

     DbCurrRecNum: longint;

          The current record is the record being held in the work space,
     i.e. the location of the database cursor. GOLDDB unit maintains a
     small portion of memory which is the exact size of a record from
     the associated database.

Field Specifics

          Records in a DBF file are said to be of fixed length. This
     means that each record is exactly the same length, regardless of
     the data stored within. Each record is made up of a pre-defined set
     of fields. Each field has its own definition which resides in the
     database header. The GOLDDB unit maintains a record type which is
     defined as follows:

          FieldDesc = record
             FdName: array [1..11] of char;
             FdType: char;
             Reserved1: array [1..4] of char;
             FdLength: byte;
             FdDec: byte;
             Reserved2: array [1..14] of char;
          end;

          You do not need to access this record directly. Use the
     following list of routines to retrieve this information.

     DbGetFldName(FieldNo: integer): string;

          Each field has its own name. The name may be no longer than 10
     characters and must begin with an alpha character. The remaining
     characters may be alphanumeric or the underscore character.

     DbGetFldType(FieldNo: integer): char;

     GOLDDB supports only dBase III+ compatible field types as follows:

          C - Character
          N - Numeric (includes floating)
          L - Logical
          D - Date
          M - Memo

     DbGetFldLength(FieldNo: integer): integer;

          Each field type has its own limitations. The supported field
     types have the following field length definitions:

          C - 1 to 254 characters
          N - 1 to 19 characters
          L - 1 character ( T or F )
          D - 8 characters
          M - 10 characters

     DbGetFldDec(FieldNo: integer): integer;

          When a field is defined as a numeric type it may also have a
     number of decimal places. This value may not be greater than [(the
     defined field length) - 2] and may never be greater than 9.

Retrieving Field Information

          Behind the scenes, all data in a .DBF file is stored in
     character form. Each of the following routines performs an
     appropriate conversion and returns the data in its native form. Any
     time information is requested about a specific field the whole
     record is loaded into the internal work space. This is so that
     subsequent requests from the same record will not have to be read
     from the storage media.

          The following code fragment demonstrates the ease of
     retrieving data from a database file:

          procedure DatabaseToScreen(RecNo:longint);
          {}
          begin
             with UserRec do
             begin
                Entered := DbGetFldDate(RecNo,1);
                Client := DbGetFldString(RecNo,2);
                Addr1 := DbGetFldString(RecNo,3);
                Addr2 := DbGetFldString(RecNo,4);
                City := DbGetFldString(RecNo,5);
                State := DbGetFldString(RecNo,6);
                Zip := DbGetFldString(RecNo,7);
                Country := DbGetFldString(RecNo,8);
                Phone := DbGetFldString(RecNo,9);
                Units := DbGetFldLong(RecNo,10);
                DbGetFldMemo(RecNo,11,MemoFldVar);
             end;
          end; { DatabaseToScreen }

     DbGetFldString(RecNo: longint; FieldNo: integer): string;

          Since all data stored in a .DBF file is in character form,
     there is no validation performed for the calling field. This
     routine may be called using any field, but you may have to do your
     own type conversion for field types other than character. It is
     recommended that you use the following routines when calling field
     types other than character.

     DbGetFldInt(RecNo: longint; FieldNo: integer): integer;

          Returns an integer value of the string stored in the database
     field. A validation is performed first to ensure that the field
     being requested is numeric.

     DbGetFldLong(RecNo: longint; FieldNo: integer): longint;

          Returns a longint value of the string stored in the database
     field. A validation is performed first to ensure that the field
     being requested is numeric.

     DbGetFldReal(RecNo: longint; FieldNo: integer): extended;

          Returns a real value of the string stored in the database
     field. A validation is performed first to ensure that the field
     being requested is numeric.

     DbGetFldLogical(RecNo: longint; FieldNo: integer): boolean;

          Validates that the field is a logical field, and returns a
     TRUE or FALSE value. Note that Gold (line dBase) stores a "T" or
     "F" in the field.

     DbGetFldDate(RecNo: longint; FieldNo: integer): Dates;

          Validates that the field is a date field, AND returns the
     Julian date. Refer to Chapter 18 for a description of Gold data
     management routines which can convert Julian dates to strings, etc.

     DbGetFldMemo(RecNo: longint; FieldNo: integer;var

     MemoDetails:MemoCfg);

          Ensures that the field is a memo field and that a memo is
     actually stored for this record. If these requirements are met,
     this routine populates the MemoDetails variable with the data
     stored in the associated .DBT file.

     DbGetMemoRecNum(RecNo:longint;FieldNo:integer):longint;

          Provides a means to extract the numeric value stored in the
     memo field of the .DBF file. This number (of ten digits, or less)
     indicates the record number, in the memo file, of the first part of
     the memo text. A zero is returned when there is no memo data for
     the record specified.

     Note:

          Memo files are built from 512 byte blocks of data. The first
     block, block 0, is the header of the file and stores only the value
     of the next available block. The remaining blocks, 1 thru N,
     directly equate to the 10 digit value stored in the memo field.

     DbFldIsEmpty(RecNo: longint;FieldNo: integer):boolean;

          Returns TRUE if the field in question only contains spaces,
     i.e. is empty. (Remember that dBase fields are always padded with
     spaces to fill the field).

          A Gold error is raised if you try to get data of the wrong
     type, e.g. you try to call DbGetFldReal for a logical field. Be
     sure to use LastDbError to check the error status after calling a
     dbGetFldxxx function. Better still, verify the field using
     DbGetFldType (discussed earlier) prior to calling DbGetFldxxx
     function..

Updating Field Information

          A variety of functions (all named DbSetFldxxx) are used to add
     new records, or update existing records. For example, the function
     DbSetFldDate (shown in the code fragment below) sets a new value
     for a date field. It is important that you understand that these
     functions do not directly access the data on disk. Gold maintains
     all the information about the active record (i.e. the record
     containing the database cursor) in a special buffer called the work
     space. The set and get functions all access the workspace, not the
     physical on-disk record.

          Having used the set functions to update the workspace, you
     must call DbAddRecord or DbPutRecord to transfer the record from
     the workspace to disk.

          Memo fields are handled slightly differently. The memo routine
     updates a separate file and then the memo field (in the DBF file)
     is updated with the correct memo block number.

     The following code fragment illustrates this technique.

          procedure ScreenToDatabase;
          {}
          var MemRecNum: longint;
          begin
             with UserRec do
             begin
                DbSetFldDate(1,Entered);
                DbSetFldString(2,Client);
                DbSetFldString(3,Addr1);
                DbSetFldString(4,Addr2);
                DbSetFldString(5,City);
                DbSetFldString(6,State);
                DbSetFldString(7,Zip);
                DbSetFldString(8,Country);
                DbSetFldString(9,Phone);
                DbSetFldInt(10,Units);
                MemRecNum := DbSetFldMemo(11,Comments);
             end;
             if Adding then
             begin
                DbAddRecord;
                Adding := false;
             end
             else
                DbPutRecord;
          end; { ScreenToRecord }

          Listed below are the individual procedures used to modify
     individual fields in the work space:

     DbSetFldString(FieldNo: integer; StrVar: string);

          After validating the field type, this routine updates the work
     space with the passed string value.

     DbSetFldInt(FieldNo: integer; IntVar: longint);

          After validating the field type, this routine updates the work
     space with the passed numeric value.

     DbSetFldReal(FieldNo: integer; RealVar: Extended);

          After validating the field type, this routine updates the work
     space with the passed real value.

     DbSetFldLogical(FieldNo: integer; BoolVar: boolean);

          After validating the field type, this routine updates the work
     space with a T or an F based on the value passed in BoolVar.

     DbSetFldDate(FieldNo: integer; DateVar: longint);

          After validating the field type, this routine converts the
     passed Julian date to the format YYYYMMDD and updates the work
     space.

     DbSetFldMemo(FldNo: integer;var SL: SingleLL): longint;

          Updates the .DBT file with the information contained in the
     single linked list after which it sets the appropriate field with
     the newly updated block number of the .DBT file.

Record Specific Routines

          Some of the routines in GOLDDB deal specifically with the
     whole record. A record is a set of data inclusive of all the fields
     defined in the header, with one byte added for the record's status.

     DbRecordIsActive(RecNo: longint): boolean;

     This function returns false if the record has been deleted.

          This routine loads the entire record into the work space and
     reads the status byte. It will be a space character if the record
     is active or an asterisk (*) character if the record has been
     marked as deleted.

     DbAddRecord;

          This procedure should only be called after all data changes to
     a record have been completed. The data currently in the work space
     is appended to the end of the .DBF file.

     DbPutRecord;

          This procedure should only be called after all modifications
     to a record have been completed. The data currently in the work
     space is returned to the record from which it was retrieved.

     DbDeleteRecord(RecNo: longint);

          Updates the status byte of the specified record with an
     asterisk (*) character. It is now marked as deleted although the
     data is still accessible.

     DbUnDeleteRecord(RecNo: longint);

          Marks a record as undeleted or active. A space character is
     placed into the status byte.

Creating a New DBF File

          Creating a new data file is simple. It is helpful (if not
     essential!) to first determine what data will be stored in the
     database and which field types best accommodate this data. Call
     DbAddDbfField once for each field to be included in the database,
     then call DbBuildDataFile. These functions are defined as follows:

     DbAddDbfField(FldName: string; FldType: char; FldLen, FldDecPl: integer): integer;

          Adds the passed information to a temporary internal list which
     is used later to build the new database. Returns a zero value if
     the operation was successful.

          Behind the scenes, Gold verifies all the data to ensure it
     remains compatible with the dBase III+ format. See field specifics
     for details on field requirements.

     DbBuildDataFile( FN: Pathstr; NDXFld : byte): integer;

          After all fields have been added using DbAddDbfField, this
     routine must be called once to create the new data file. You may
     specify the field from which to build the index. If a zero is
     passed, no index file is built. If a memo field is included in the
     list of fields, a memo file with the same name will automatically
     be included. Returns a zero value if the file was created
     successfully.

     Here's an example:

          procedure CreateNewDataFile;
          {}
          var EValue: integer;
          begin
             EValue := 0;
             inc(EValue,DbAddDbfField('DATE','D',8,0));
             inc(EValue,DbAddDbfField('CLIENT','C',30,0));
             inc(EValue,DbAddDbfField('ADDR1','C',30,0));
             inc(EValue,DbAddDbfField('ADDR2','C',30,0));
             inc(EValue,DbAddDbfField('CITY','C',22,0));
             inc(EValue,DbAddDbfField('STATE','C',2,0));
             inc(EValue,DbAddDbfField('ZIP','C',10,0));
             inc(EValue,DbAddDbfField('COUNTRY','C',20,0));
             inc(EValue,DbAddDbfField('PHONE','C',10,0));
             inc(EValue,DbAddDbfField('UNITS','N',10,0));
             if EValue = 0 then
                inc(EValue,DbBuildDataFile(FN,1))
             if EValue <> 0 then
             begin
               PromptOK(' File Error ','Unable to create file!');
               Halt;
             end;
          end; { CreateNewDataFile }

          The code illustrates a convenient way to trap for errors
     without bogging the code in nested if statements. EValue will be
     incremented by zero each time a field is added successfully. If
     DbAddDbfField fails, a non-zero value will be returned. EValue is
     then checked before building the data file to ensure that all
     fields were added correctly.

Powerful Searching and Sorting

          The power of a database lies not in its ability to store
     information, but how quickly and in what order this information can
     be retrieved. This power is achieved by indexing, and the following
     index functions are available in Gold:

     NdxBuildNew(FieldNo: integer): integer;

          Builds a brand new index file. Remember that the index file
     retains the same name as the associated data file. This routine
     will allow the application to switch the field on which the index
     file is built.

     SetShowNdxProgress(Proc: ShowNdxProgressProc);

          Turns on a hook that will be called each time a key is added
     to the new index file. Used in conjunction with WriteProgressLong
     to display the progress of the index build. Proc must be declared
     at the root level and be declared as a far procedure. The current
     record number and total number of records are passed each time the
     hook is called.

     NdxGetRecNum(EntryNum: longInt) : longInt;

     Returns the record number of a specific entry in the index.

     NdxCount : longint;

          Returns the total number of entries in the index. This should
     be the same as the total number of active records in the data file.

     NdxGotoFirst: longint;

     Returns the record number of the first entry of the index.

     NdxGotoLast: longint;

     Returns the record number of the last entry in the index.

     NdxGotoNext: longint;

          Returns the record number of the next index entry from your
     present location.

     NdxGotoPrev: longint;

          Returns the record number of the previous index entry from
     your present location.

     DbFindFirst(FieldNo: integer; var findValue;
                                       PartialMatch: boolean): longint;

          Returns the record number of the first match successfully
     found. When PartialMatch is set to true, the user may search for
     incomplete information.

     DbFindNext: longInt;

          Returns the record number of the next match found with the
     same criteria used in DbFindFirst. Always call DbFindFirst prior to
     the first call to DbFindNext.

     NdxSetMaxPages(n: Word);

          Sets the number of pages that can be loaded into memory at
     once. The default is 25 pages. The default will suit the majority
     of applications, but you may increase or decrease this value as
     necessary. Increasing allows more of the data to remain in memory
     and reduces the number of disk reads, and at the same time reduces
     the amount of memory available for your program. Decreasing does
     just the opposite, reducing the amount of data in memory, but
     increasing the number of disk reads.

     NdxSetUpperCase(x: boolean);

          Pass TRUE to force the index to be built in uppercase. This is
     the default.

     NdxSetMaxStrLength(n: Byte);

          Sets the maximum length that will be accepted into an index
     key. The default is 30 characters. Increasing this value will also
     increase the size of the index. If you change the index string
     length, you must rebuild the index.

Maintenance Procedures

          A DBF file must remain synchronized with its associated index
     (.GDX) and memo (.DBT) files. Data can be currupted for a variety
     of reasons, e.g. disk failures, coding bugs (yours, not ours!). If
     the data gets corrupted, you can use one of the following routines
     to fix the problem:

     DbRebuildMemo(FName: PathStr): integer;

          Any time a memo is added or modified in any way (even if only
     one character is changed)  the entire memo is appended to the same
     memo file. The space where the memo existed is never modified. The
     database memo field is updated to reflect the new memo block
     number. This mechansim is fast, but tends to waste huge amounts of
     space since the old memo data is never removed during normal
     operation. Calling this routine removes all unused blocks and
     updates the memo fields in the datafile as the new memo file is
     written.

     NdxRebuild: integer;

          If an index file becomes unsynchronized with its associated
     data file, it will become necessary to rebuild it to resynchronize
     the files. NdxRebuild uses the current index field as its key. The
     old index file is overwritten by the new one.

          When a database record is deleted, the record is marked as
     deleted, but remains stored in the database. This mechansim makes
     record undeletion possible. Use the following function when you
     want to physically remove all deleted records from a DBF file:

     DbPackFile(FName: PathStr; IndexField: integer): integer;

          Removes all records that are currently marked as deleted. Also
     rebuilds the associated index using the current index field.
     Remember that they must remain synchronized.

Managing Database Colors

          Database colors are actually quite simple to deal with. There
     are none!

Error Recovery

          One quality which separates the professional from the novice
     is the ability to gracefully recover from an error. GOLDDB contains
     a wealth of  internal error checking. If you call a routine which
     has the potential to fail, call the function LastDbError to make
     sure the operation was successful.

