                           dBASE III MEMO FIELDS

                                    by

                            P. L. Olympia, Ph.D.
                            Darwin Systems, Inc.
               Sysop, Darwin PCBoard, 301-251-9206, 12/2400 bps
                                05/01/85


NOTE: This paper recently appeared in the February 1986 issue of
      Data Based Advisor, Vol 4, No.2, pp 54-58.

      A related article, "Fast Screens and Assembly Language Interfaces
      in dBASE II/III and Clipper" will appear in the March 1986 issue.


PURPOSE

     As  I mentioned in my review of dBASE III last July,  1984,  the  MEMO 
field  of  dBASE III is a potentially useful feature but is not  very  well 
implemented  even  in  version 1.1.   Memo fields allow you  to  have  text 
information of unlimited length as part of your data base.   Of course,  if 
you  use the built-in dBASE editor to enter your memo field data,  you  are 
limited to only about 4000 characters worth of text data per  field.   Many
people are under the mistaken notion that the limitation is inherent to the 
memo  field concept per se.   Not so.  The limitation is that of the  dBASE 
editor, not the memo field.

     The purpose of this article is to explain the structure of dBASE III's 
DBF and DBT files (versions 1.0,  1.1 and 1.2), the difficulties associated 
with working with MEMO fields and the solutions to those.  I have written a 
program  that  solves the most serious constraints on  memo  fields,  e.g., 
loading memo fields with data from an external text file created elsewhere, 
even a mainframe.  The program is based on information described here.  


MEMO FIELDS FEATURES (AND MISFEATURES)


     Memo fields are defined by you when you create your database file.  In 
principle,  the number of memo fields in a data base is subject only to the 
maximum number of fields that dBASE III allows.  A memo field occupies only 
10 bytes in the DBF file; these 10 bytes are used to store the block number 
where  the actual memo field text data are stored in the corresponding  DBT 
file (more on this later).   Thus, a data base with memo fields really have 
two files:  the usual DBF file,  whose first byte is 83 hex (signifying the 
existence of memo fields;  this value is 03 hex otherwise),  and a DBT file 
where the actual memo field data are stored.

     How do you enter data into the field?   At present, the only way to do 
that  is  to go into EDIT mode,  place the cursor in the memo  field,  then 
press CTRL-PgDn (or CTRL-Home).   At that point,  either the built-in dBASE 
editor  or the text editor of your choice (installed in the CONFIG.DB  file 
with  the  line WP=<editor name>) opens up to accept your  keyboarded  text 
data.  Of  course,  if your text editor can do an external file read  (like 
WordStar,  for example) you can simply enter previously prepared data  into 
the  memo  field.   Your inability to load data into the memo field by  any 
other  way  except  for  this cumbersome procedure  is  one  of  the  worst 
limitations of the present implementation.


     Another  serious  deficiency is your inability to display the data  in 
the  exact form that you entered them except via the REPORT  FORM  command.  
If you give the LIST or DISPLAY commands,  the memo data will not show. The 
only  way  to display them is via the commands LIST/DISPLAY/?  <memo  field 
name>.  And, then, dBASE imposes its own display format so you will not see 
a  line  end  where you expect it to  end.   Note,  however,  that  in  the 
Developer's  release of dBASE III,  Version 1.2,  you can now override  the 
default  display  width  of 55 characters.   It costs $200  to  upgrade  to 
version 1.2.

     If you accidentally lose your DBT file,  your DBF file is  temporarily 
unusable  since you cannot open it via the USE command.   Luckily,  all you 
have  to  do is change the first byte of the DBF file from 83H  to  03H  to 
recover  the rest of the data base.  You can change the byte either with  a 
program or DEBUG (provided the DBF file is small enough).  Perhaps the most 
expedient  way around the problem of losing your DBT file is to copy any of 
your old DBT files (or create a new one) to one with the same primary  name 
as your DBF file.   Thus,  if you have a file called ONE.DBF whose DBT file 
you lost,  and you have a completely unrelated DBT file,  say, TWO.DBT, you 
can get going with the DOS command:

          C>copy two.dbt one.dbt

     What  makes  this  seemingly senseless operation work  is  that  there 
really is nothing in a DBT file that uniquely identifies it as belonging to 
a specific data base file.   All dBASE III looks for is a matching  primary 
filename.   Of  course,  all  your memo field operations will not make  any 
sense until you can reconstruct the proper record pointers. 

     The  way  memo data are stored in the DBT file is simplistic  and  can 
needlessly gobble up precious disk space. If you edit a memo field multiple 
times, the old data continue to occupy space even though they are no longer 
needed or even accessible. Ashton-Tate suggests that you periodically issue 
a COPY TO command to free up this space.   That is a painfully slow process 
that programs like DBMEMDPO can handle better.

     One  other major problem with memo fields is your inability to  search 
them  for text strings within dBASE III.   This is really not  a  difficult 
problem  to solve.   It is easy enough to write a program that  can  handle 
that task once you have an understanding of memo file structure.



STRUCTURE OF A DBASE III DATA BASE (DBF) FILE 

Consider a sample data base with three memo fields:

Structure for database : A:pdscomnt.dbf
Number of data records :     144
Date of last update    : 04/17/85
Field  Field name  Type       Width    Dec
    1  COMNTCATEG  Character     40
    2  COMNTID     Character     16
    3  COMNTDATE   Date           8
    4  COMNTWHO    Character     60
    5  COMMENT_1   Memo          10
    6  COMMENT_2   Memo          10
    7  CMNTSEC_PG  Character     50
    8  RESP_ROUTE  Character     40
    9  RESP_REQD   Character      1
   10  RESPONSE    Memo          10
** Total **                     246

 . display

Record#  COMNTCATEG                               COMNTID          COMNTDATE 
COMNTWHO                                                     COMMENT_1 
COMMENT_2 CMNTSEC_PG                                         RESP_ROUTE        
                       RESP_REQD RESPONSE
      1  OTHER                                    DDS0185/ARR-001  03/04/85  
U. S. NUCLEAR RADIOACTIVE WASTE CLN                          Memo      
Memo                                                                           
                                 Memo


     Note  that the LIST or DISPLAY command without any field argument will 
only show the word "Memo" instead of the actual data.

     The  DBF  structure contains of a header and actual data  (except  for 
memo field data):  The header structure is as follows:

     BYTE      CONTENTS       DESCRIPTION

     0         1 byte         03H without DBT; 83H otherwise
     1-3       3 bytes        Date of last update, YY MM DD
     4-7       32-bit #       number of data base records 
     8-9       16-bit #       length of header structure
     10-11     16-bit #       length of data base record
     12-31     20 bytes       Reserved
     32-n      32 bytes       Field descriptor array (described below)
     n+1       1 byte         0DH (header structure terminator)


     Each  field occupies 32 bytes in the DBF header structure and arranged 
as follows:

     BYTE      CONTENTS       DESCRIPTION

     0-10      11 bytes       Field name
     11        1 byte         Field type, ASCII (C,N,L,D,M)
     12-15     32-bit #       Field data address
     16        1 byte         Field length
     17        1 byte         Decimal count (if field is numeric)
     18-31     14 bytes       Reserved


     The  data  records immediately follow the  header.  Each  data  record 
begins with 20H.   All data are in ASCII form.   The 10 bytes allocated for 
the record's memo field contain the block number in the DBT file where that 
field's data begin.  A DBT block consists of 512 bytes.

     Here is the actual layout of the sample DBF data base:


  Date last upd         Length of Header
             |                  |
 03=w/o Memo |                  | Length of
 83=w/ Memo  |    # records     | Data recd
      |      |    in DBF        |     |
OFFST |  +------+ +---------+ +---+ +---+
      |  |      | |         | |   | |   |
0100  83 55 04 11 05 00 00 00-62 01 F6 00 00 00 00 00   .U......b.v.....
0110  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
0120  43 4F 4D 4E 54 43 41 54-45 47 00 43 0F 00 F5 67   COMNTCATEG.C..ug
      |                              | |  |         |
      +-- Field Name (left just) ----+ |  +---------+
                                       |   Field data address
                                   Field type (C,D,L,M)
 Field length
      |   ___ Field decimal count
      |  |
0130  28 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   (...............
0140  43 4F 4D 4E 54 49 44 00-00 00 00 43 37 00 F5 67   COMNTID....C7.ug
0150  10 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
0160  43 4F 4D 4E 54 44 41 54-45 00 00 44 47 00 F5 67   COMNTDATE..DG.ug
0170  08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
0180  43 4F 4D 4E 54 57 48 4F-00 00 00 43 4F 00 F5 67   COMNTWHO...CO.ug
0190  3C 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   <...............
01A0  43 4F 4D 4D 45 4E 54 5F-31 00 00 4D 8B 00 F5 67   COMMENT_1..M..ug
01B0  0A 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
01C0  43 4F 4D 4D 45 4E 54 5F-32 00 00 4D 95 00 F5 67   COMMENT_2..M..ug
01D0  0A 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
01E0  43 4D 4E 54 53 45 43 5F-50 47 00 43 9F 00 F5 67   CMNTSEC_PG.C..ug
01F0  32 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   2...............
0200  52 45 53 50 5F 52 4F 55-54 45 00 43 D1 00 F5 67   RESP_ROUTE.CQ.ug
0210  28 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   (...............
0220  52 45 53 50 5F 52 45 51-44 00 00 43 F9 00 F5 67   RESP_REQD..Cy.ug
0230  01 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
0240  52 45 53 50 4F 4E 53 45-00 00 00 4D FA 00 F5 67   RESPONSE...Mz.ug
0250  0A 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 | ................
           +-- END OF HEADER RECORD ------------------+
 Hdr recd  | 
 terminator|  _____ Start of data (2A=deleted, 00=not deleted)
      |    | |
0260  0D 00 2A 4F 54 48 45 52-20 20 20 20 20 20 20 20   ..*OTHER        
0270  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0280  20 20 20 20 20 20 20 20-20 20 20 50 44 53 30 31              DDS01
0290  38 35 2F 4E 52 43 2D 30-30 31 20 31 39 38 35 30   85/ARR-001 19850
02A0  33 30 34 55 2E 20 53 2E-20 4E 55 43 4C 45 41 52   304U. S. NUCLEAR
02B0  20 52 45 47 55 4C 41 54-4F 52 59 20 43 4F 4D 4D    RADIOACTIVE WAS
02C0  49 53 53 49 4F 4E 20 20-20 20 20 20 20 20 20 20   TE CLN          
                                                  +---+
02D0  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20|20 |                 
     +---------------------------=====================+
02E0 |20 20 20 20 20 20 20 20-31|20 20 20 20 20 20 20           1       
     +-- -----------------------+ 
         10-digit number where first
         memo field (COMMENT_1) content is located

02F0  20 20 32 20 20 20 20 20-20 20 20 20 20 20 20 20     2             
0300  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0310  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0320  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20  RESPONSE memo field
0330  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20  is empty for this recd
0340  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20          |        
0350  20 20 20 20 20 20 20 30-20 53 43 48 45 44 55 4C          0 SCHEDUL
0360  45 2F 52 45 56 49 45 57-2D 50 45 52 49 4F 44 20   E/REVIEW-PERIOD 
0370  44 55 52 41 54 49 4F 4E-53 20 20 20 20 20 20 20   DURATIONS       
0380  20 50 44 53 30 31 38 35-2F 4E 52 43 2D 30 30 32    DDS0185/ARR-002
0390  20 31 39 38 35 30 33 30-34 55 2E 53 2E 20 4E 55    19850304U.S. NU
03A0  43 4C 45 41 52 20 52 45-47 55 4C 41 54 4F 52 59   CLEAR RADIOACTIV
03B0  20 43 4F 4D 4D 49 53 53-49 4F 4E 20 20 20 20 20   E WASTE CLN     
03C0  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
03D0  20 20 20 20 20 20 20 20-20 20 20 20 20 20 33 20                 3 
03E0  20 20 20 20 20 20 20 20-30 20 20 20 20 20 20 20           0       
03F0  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0400  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0410  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0420  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0430  20 20 20 20 20 20 20 20-20 20 20 20 20 20 20 20                   
0440  20 20 20 20 20 20 20 20-20 20 20 20 20 30 20 53                0 S
0450  43 48 45 44 55 4C 45 20-4D 49 4C 45 53 54 4F 4E   CHEDULE MILESTON
     ....... rest of data base ....



STRUCTURE OF A DBASE III MEMO (DBT) FILE 

     A DBT file consists of blocks of 512 bytes each. Block 0 is the header 
block of which only the first FOUR bytes are used.  These four bytes hold a 
32-bit number signifying the next available block number in the DBT file.


     All  succeeding  blocks are the data blocks containing the memo  field 
data.   If a memo field's data consist of more than 512  bytes,  succeeding 
blocks are used.  For example, if the data consist of 1025 bytes, three DBT 
blocks are used, where the third block contains only one useful byte.  Each 
memo field is terminated with two ^Z's (1A hex). 

     The  block number containing the start of this data base record's memo 
field is recorded (in ASCII) in the 10 bytes of the corresponding DBF file.

     If you edit a memo field, the entire memo field data are re-written at 
the  end  of the DBT file.  The old data remain in the  DBT  file,  wasting 
space.  This wasted space is released when you do a COPY TO command on your 
DBF file.

     Here is how the DBT file of the above data base looks:


           _____ Next available (512-byte) block in DBT
OFFST      |
 |    +---------+
0100  0B 00 00 00 05 00 00 00-01 00 03 00 01 00 04 00   ................ _____
0110  00 00 F6 00 62 01 33 00-03 00 00 00 39 00 04 00   ..v.b.3.....9...   | 
0120  6E 01 17 68 96 2A CD 5C-7A 24 CD 5C 64 3A 70 64   n..h.*M\z$M\d:pd   |
0130  73 74 69 6E 79 2E 64 62-66 00 00 00 00 00 00 00   stiny.dbf.......
0140  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   H
0150  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   E
0160  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   A
0170  00 00 00 00 00 00 00 00-00 00 00 00 64 3A 70 64   ............d:pd   D
0180  73 74 69 6E 79 2E 64 62-74 00 00 00 00 00 00 00   stiny.dbt.......   E
0190  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   R
01A0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
01B0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   B
01C0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   L
01D0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   O
01E0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   C
01F0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   K
0200  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................
0210  00 00 00 00 00 00 00 00-00 00 00 00 33 2E 00 17   ............3...  (0)
0220  68 39 56 29 CD 5C 33 4E-00 17 68 39 76 29 CD 5C   h9V)M\3N..h9v)M\   |
0230  33 6E 00 17 68 39 96 29-CD 5C 33 8E 00 17 68 39   3n..h9.)M\3...h9   |
0240  B6 29 CD 5C 33 EE 00 17-68 39 16 2A CD 5C 33 0E   6)M\3n..h9.*M\3.   |
0250  01 17 68 39 36 2A CD 5C-33 2E 01 17 68 39 56 2A   ..h96*M\3...h9V*   |
0260  CD 5C 00 00 00 00 00 00-00 00 00 00 00 00 00 00   M\..............   | 
0270  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
0280  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
0290  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
02A0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
02B0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
02C0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
02D0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
02E0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................   |
02F0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................ __|__

     What follows is DBT Block 1 containing the data contents of  COMMENT_1 
memo field in the first data base record 

------------------------------------------------------------------------
0300  53 69 6E 63 E5 20 74 68-E5 20 6C 69 63 65 6E 73   Since the licens
0310  69 6E E7 20 70 72 6F 63-65 73 F3 20 75 6E 64 65   ing process unde
0320  F2 20 31 B0 20 43 46 D2-20 50 61 72 F4 20 36 B0   r XX CFR Part YY
0330  20 69 F3 20 63 65 6E 74-72 61 EC 20 74 EF 20 8D    is central to .
0340  0A 74 68 E5 20 4E 52 C3-20 73 63 68 65 64 75 6C   .the ARR schedul
0350  65 F3 20 61 6E E4 20 74-69 6D E5 20 72 65 71 75   es and time requ
0360  69 72 65 6D 65 6E 74 73-AC 20 77 E5 20 62 65 6C   irements, we bel
0370  69 65 76 E5 20 61 6C EC-20 74 68 E5 20 6B 65 F9   ieve all the key
0380  20 8D 0A 73 74 65 70 F3-20 69 EE 20 74 68 69 F3    ..steps in this
0390  20 70 72 6F 63 65 73 F3-20 73 68 6F 75 6C E4 20    process should 
03A0  62 E5 20 69 64 65 6E 74-69 66 69 65 E4 20 69 EE   be identified in
03B0  20 74 68 E5 20 50 72 6F-6A 65 63 F4 20 8D 0A 44    the Project ..D
03C0  65 63 69 73 69 6F 6E 20-53 63 68 65 64 75 6C 65   ecision Schedule
03D0  2E 1A 1A 1A 1A 1A 1A 1A-1A 1A 1A 1A 1A 1A 1A 1A   ................
03E0  1A 1A 1A 1A 1A 1A 1A 1A-1A 1A 1A 1A 1A 1A 1A 1A   ................
03F0  1A 1A 1A 1A 1A 1A 1A 1A-1A 1A 1A 1A 1A 1A 1A 1A   ................
       .... rest of DBT file ...


SAMPLE APPLICATION - SEARCH MEMO FIELD FOR STRINGS

     Armed  with a knowledge of DBF and DBT file structures,  you  can  now 
write  a simple program to search your memo fields for a specified  string.  
The  trick is simply to use a string search function like INSTR in BASIC to 
find the string in the DBT file.  After you locate the string, you can just 
display it and its surrounding text.   If you need to also display  certain 
fields  of  the  data base,  you have to compute which 512-byte  block  the 
string is in and then use the DBF file to retrieve the remaining fields  of 
the data base record that contains this memo field string.

     Your  program  does not need dBASE III to do its work,  and so may  be 
invoked at the DOS prompt.  If you are so inclined, the program may also be 
executed  from  within dBASE by way of the RUN command.  If you choose  the 
latter approach, you should close the affected data base before running the 
program.
 


POSTSCRIPT

     I have written a demonstration program that allows you to:

          o pack and reorganize your DBT files FAST!
          o load your memo fields with text data generated elsewhere
            either with your PC text editor or captured from a mainframe
          o export memo data to a text file so you can manipulate them
            with your text editor


     from DOS (without you needing dBASE III).  All can be done by one
     command invocation with all parameters passed in the command line.
     Thus, the program can be run from a batch file and need not be
     attended.

     The file is called MEMODPO.ARC which is available for downloading from
     Darwin PCBoard, and discriminating BBS's around the country.


