0% found this document useful (0 votes)
11 views179 pages

تنظيم الملفات

The document discusses file structure design and specification, emphasizing the importance of efficient data access due to the slow nature of disk storage compared to RAM. It outlines the evolution of file structures from sequential access to tree structures, highlighting the development of B-trees and hashing techniques for improved performance. The text also introduces an object-oriented approach using C++ to implement file structures, focusing on class definitions and the integration of data and behavior.

Uploaded by

abdelrahman53502
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
11 views179 pages

تنظيم الملفات

The document discusses file structure design and specification, emphasizing the importance of efficient data access due to the slow nature of disk storage compared to RAM. It outlines the evolution of file structures from sequential access to tree structures, highlighting the development of B-trees and hashing techniques for improved performance. The text also introduces an object-oriented approach using C++ to implement file structures, focusing on class definitions and the integration of data and behavior.

Uploaded by

abdelrahman53502
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 179

File Organization

An Object-Oriented
Approach with C++
Contents

 Chapter 1 Introduction to the Design and

Specification of File Structures

 Chapter 2 Fundamental File Processing Operations

 Chapter 3 Secondary Storage and System Software

 Chapter 4 Fundamental File Structure Concepts

 Chapter 5 Managing Files of Records

 Chapter 6 Organizing Files for Performance

 Chapter 7 Indexing
CHAPTER 1

Introduction to the
Design and Specification
of File Structures

1.1 T h e Heart of File Structure Design

Disks are slow. They are also technological marvels: one can pack
thousands of megabytes on a, disk that fits into a notebook computer. Only
a few years ago, disks with that kind of capacity looked like small washing
machines. However, relative to other parts of a computer, disks are slow.
How slow? The time it takes to get information back from even
relatively slow electronic random access memory (RAM) is about 120
nanoseconds, or
120 billionths of a second. Getting the same information from a typical
disk
might take 30 milliseconds, or 30 thousandths of a second. To understand
the size of this difference, we need an analogy. Assume that memory
access is like finding something in the index of this book. Let’s say that
this local, book-in-hand access takes 20 seconds. Assume that accessing
a disk is like sending to a library for the information you cannot find here
in this book. Given that our “memory access” takes 20 seconds, how long
does the “disk access” to the library take, keeping the ratio the same as that
of a real memory access and disk access? The disk access is a quarter of a
million times longer than the memory access. This means that
getting'information back from the library takes 5 million seconds, or
almost 58 days. Disks are very slow compared with memory.
On the other hand, disks provide enormous capacity at much less cost
than memory. They also keep the information stored on them when they
are turned off. The tension between a disk’s relatively slow access time
and its enormous, nonvolatile capacity is the driving force behind file
structure design. Good file structure design will give us access to all the
capacity without making our applications spend a lot of time waiting for
the disk.
A file structure is a combination of representations for data in files and
of operations for accessing the data. A file structure allows applications to
read, write,, and modify data. It might also support finding the data that
matches some search criteria or reading through the data in some
particular order. An improvement in file structure design may make an
application hundreds of times faster. The details of the representation of
the data and the implementation of the operations determine the efficiency
of the file structure for particular applications.
A tremendous variety in the types of data and in the needs of applications
makes file structure design very important. What is best for one situation may
be terrible for another.

1.2 A Short History of File Structure Design

Our goal is to show you how to think creatively about file structure design
problems. Part of our approach draws on history: after introducing basic
principles of design, we devote the last part of this book to studying some of
the key developments in file design over the last thirty years. The problems that
researchers struggle with, reflect the same issues that you confront in addressing
any substantial file design problem. Working through the approaches to major
file design issues shows you a lot about how to approach new design problems.
The general goals of research and development in file structures can be
drawn directly from our library analogy.
■ Ideally, we would like to get the information we need with one access to
the disk. In terms of our analogy, we do not want to issue a series of
fifty-eight-day requests before we get what we want.
If it is impossible to getwhat we need in one accessT we want

It is relatively easy to come up with file structure designs that meet these goals when
we have files that never change. Designing file structures that maintain these qualities
as files change, grow, or shrink when information is added and deleted is much more
difficult.
Early work with files presumed that files were on- tape, since most files were.
Access was sequential, and the .cost to the size of the file. As files grew
intolerably large for unaided sequential access and as storage devices such as disk drives
became available, indexes were added to files. The indexes made it possible to keep a list
of keys and pointers in a smaller file that could be searched more quickly. With the key
and pointer, the user had direct access to the large, primary file.
Unfortunately, simple indexes had some of the same sequential flavor as the data
files, and as the indexes grew, they too became difficult to manage, especially for
dynamic files in which the set of keys changes. Then, in the early 1960s, the idea of
applying tree structures emerged. Unfortunately, trees can grow very unevenly as
records are added and deleted, resulting in long searches requiring many disk accesses
to find a record.
In 1963 researchers developed the tree, an elegant, self-adjusting binary tree
structure, called an AVL tree, for data in memory. Other researchers' began to look for
ways to apply AVL trees, or something like them, to files. The problem was that even
with a balanced binary tree, dozens of accesses were required to find a record in even
moderate-sized files. A method was needed to keep a tree balanced when each node of
the tree was not a single record, as in a binary tree, but a file block containing dozens,
perhaps even hundreds, of records.
It took nearly ten more years of design work before a solution emerged in the form
of the B-tree. Part of the reason finding a solution took so long was that the approach
required for file structures was very different from the approach that worked in memory.
Whereas AVL trees grow from the top down as records are added, B-trees grow from
the bottom up.
Bytrees provided excellent access performance, but there was a cost: no longer
could a file be,accessed_sequenfially^ifhLefficienQ:. Fortunately, this problem was
solved almost immediately bv adding a linked list structure at thebottom level of the B-
tree. The combination of a Brtriee and a.sequen- tial linked list is called a B+ tree.
Over the next ten years, B-trees and B+ trees became the basis for many
commercial file systems, since they provide access times that grow in
proportion to logwhere N is the number of entries in the file and k is the number of
entries indexed in a single block of the B-tree structure. In practical terms, this means
that B-trees can guarantee that you can find one file entry among millions of others
with only three or four trips to the disk. Further, B-trees guarantee that as you add and
delete entries, performance stays about the same.
Being able to retrieve information with just three or four accesses is
pretty good. But how about our goal of being able to get what we want with a
single request? An approach called hashing is_a.good .way to do that with
files that do not change size greatiX-Qxenfime. From early on, hashed indexes
were used to provide fast access to files. However, until recently, hashing did
not work well with volatile, dynamic files. After, the development of B-trees,
researchers turned to work on systems for extendible, dynamic hashing that
could retrieve information with one or, at most, two disk accesses no matter
how big the file became.

1.3 A Conceptual Toolkit: File Structure Literacy


As we move through the developments in file structures over the last three
decades, watching file structure design evolve as it addresses dynamic files
first sequentially, then through tree structures, and finally through direct
access, we see that the same design problems and design tools keep emerging.
We decrease the number of-disk-access. we manage the growth of these
collections by splitting them, which requires that we find a way to increase
our address or index space, and so on. Progress takes the form of finding new
ways to combine these basic tools of file design.
We think of these tools as conceptual tools. They are methods of framing
and addressing a design problem. Each tool combines ways of representing
data with specific operations. Our own work in file structures has shown us
that by understanding the tools thoroughly and by studying how the tools have
evolved to produce such diverse approaches as B-trees and extendible hashing,
we develop mastery and flexibility in our own use of the tools. In other words,
we acquire literacy with regard to file structures. This text is designed to help
readers acquire file structure literacy. Chapters 2 through 6 introduce the basic
tools; Chapters 7 through 11 introduce readers to the highlights of the past
several decades of file structure design, showing how the basic tools are used to
handle efficient
Chapter 1 Introduction to the Design and Specification of File Structures

sequential access—B-trees, Bf trees, hashed indexes! and extendible, dynamic


hashed files.

1.4 An Object-Oriented Toolkit: Making File


Structures Usable

Making file structures usable in application development requires turning this


conceptual toolkit into application programming interfaces— collections of
data types and operations that can be used in applications. We have chosen to
employ an object-oriented. approach in which data types and operations are
presented in a unified fashion as class definitions. Each particular approach to
representing some aspect of a file structure is represented by one or more
classes of objects.
A major problem in describing the classes that can be used for file
structure design is that they are complicated and progressive. New classes are
often modifications or extensions of other classes, and the details of the data
representations and operations become ever more complex. The most effective
strategy for describing these classes is to give specific representations in the
simplest fashion. In this text, use the C++ programming language to give
precise specifications to the file structure classes. From the first chapter to the
last, this allows us to build one class on top of another in a concise and
understandable fashion.

1.5 Using Objects in C++

In an object-oriented information system, data content and behavior are


integrated into a single design. The objects of the system are divided into
classes of objects with common characteristics. Each class is described by its
members, which are either data attributes (data members) or functions
(member functions or methods). This book illustrates the principles of
object-oriented design through implementations of file structures and file
operations as C++ classes. These classes are also an extensive presentation of
the features of C++. In this section, we look at some of the features of objects
in C++, including class definitions, constructors, public and private sections,
and operator overloading. Later chapters- show how to make effective use
of.inheritance, virtual functions, and templates.
An example of a very simple C++ class is Person, as given below.
class Person
{ public
// data members
char LastName [11], FirstName ■ [11], Address [16];
char City [16], State [3], ZipCode [10];
II method
Person (); 7/ default constructor
};
Each Person object has first and last names, address, city, state, and zip code,
which are declared as members, just as they would be in a C struct. For an
object p of type Person, p . LastName refers to its Las tName member.
The public label specifies that the following members and methods are
part of the interface to objects of the class. These members and methods can
be freely accessed by any users of Person objects. There are three levels of
access to class members; public, private, and protected. The last two restrict
access and will be described later in the book. The only significant difference
in C++ between struct and class is that for struct members the default access is
public, and for class members the default access is private.
Each of these member fields is represented by a character array of fixed
size. However, the usual style of dealing with character arrays in C++ is to
represent the value of the array as a nulhdelimited, variable-sized string with a
maximum length. The number of characters in the representation of a string is
one more than the number of characters in the string. The -LastName field, for
example, is represented by an array of eleven characters and can hold a string
of length between 0 and 10. Proper use of strings in C++ is dependent on
ensuring that every string variable is initialized before it is used.
C++ includes special methods called constructors that are used to provide
a guarantee that.every object is properly initialized.1 A constructor is a
method with no return type whose name is the same as the class. Whenever an
object is created, a constructor is called. The two ways that objects are created
in C++ are by the declaration of a variable (automatic creation) and by the
execution of a new operation (dynamic creation):

Person p; / / automatic creation


Person * p_ptr = new Person; // dynamic creation
Execution of either of the object creation statements above includes the
execution of the Person constructor. Hence, we-are sure that every Person
object has been properly initialized before it is used. The code for the Person
constructor initializes each member to an empty string by assigning 0 (null) to
the first character:
Person::Person ()
•{ // set each field to an empty string
LastName [0] = 0; FirstName [0] =0; Address [0] = 0;
City '[0] = 0; State [0] = 0; ZipCode [0] = 0;
}

The symbol:: is the scope resolution operator. In this case, it tells us that
Person () is a method of class Person. Notice that within the method code, the
members can be referenced without the.dot (.) operator. Every call on a
member function has a pointer to an object as a hidden argument. The implicit
argument can be explicitly referred to with the keyword this. Within the
method, this->LastName is the same as LastName.
Overloading of symbols in programming languages allows a particular
symbol to have more than one meaning. The meaning of each instance of the
symbol depends on the context. We are very familiar with overloading of
arithmetic operators to have different meanings depending on the operand type.
For example, the symbol + is used for both integer and floating point addition.
C++ supports the use of overloading by programmers for a wide variety of
symbols. We can create new meanings for operator symbols and for named
functions.
The following class String illustrates extensive use of overloading: there
are three constructors, and the operators = and == are overloaded with new
meanings; .'
class String
{public:
String (}; // default constructor String (const
StringSc); //copy constructor String (const char *); //
create from C string -String (); // destructor
String & operator = (const String &); // assignment int
operator == (const String &) const; // equality char *
operator char*() // conversion to char *
{return strdup(string);} // inline body of method
private:char * string; // represent value as C string
int MaxLength;
,
The data members, string and MaxLength, of class String are in the
private section of the class. Access to these members is restricted. They can
be referenced only from inside the code of methods of the class. Hence, users
of String objects cannot directly manipulate these members..A conversion
operator (operator char *) has been provided to allow the use of the value of
a String object as a C string. The body of this operator is given inline, that is,
directly in the class definition. To protect the value of the String from direct
manipulation, a copy of the string value is returned. This operator allows a
String object to be used as a char *. For example, the following code creates
a String object s 1 and. copies its value to normal C string:
String si ("abcdefg"); // uses String::String (const char *) char
str[10];
strcpy (str, si); //uses String::operator char * ()
The new definition of the assignment operator (operator -) replaces the
standard meaning, which in C and C++ is to copy the bit pattern of one object
to another. For two objects si and s2 of class String, si = s2 would copy the
value of si. string (a pointer) to s2 . string. Hence, si. string and s2 . string
point to the same character array. In essence, si and s2 become aliases. Once
the two fields point to the same array, a change in the string value of si would
also . change s2. This is contrary to how we expect variables to behave. The
implementation of the assignment operator and an example of its use are:
String & String::operator = (const String & str)
{ // code for assignment operator
strcpy (string, str.string);
■ return *this;
1
String si, s2;
si = s2; // using overloaded assignment

In the assignment si = s2, the hidden argument (this) refers to si, and the
explicit argument str refers to s2. The line strcpy ( string, str . string); copies
the contents of the string member of s2 to the string member of si. This
assignment operator does not create the alias problem that occurs with the
standard meaning of assignment.
Chapter 1 introduction to the Design and Specification of File Structures

To complete the class String, we add the copy constructor, which is used
whenever a copy of a string is needed, and the equality operator (operator = = ),
which makes two String objects equal if the array contents are the same. The
predefined meaning for these operators performs pointer copy and pointer
comparison, respectively.
CHAPTER 2

Fundamental File
Processing Operations

2.1 Physical Files and Logical Files

When we talk about ajfile on a disk or tape, we refer to a particular collection


of bytes stored there. A file, when the word is used' in this sense, phys- ically
exists. A disk drive might contain hundreds, even thousands, of these physical
files. From the standpoint of an application program, the notion of a file is
different. The program knows only about its own end of the phone line.
Moreover, even though there may be thousands of physical files on a disk, a
single program is usually limited to the use of only about twenty files.
The application program relies on the operating system to take care of the
details of the telephone switching system, as illustrated in Fig. 2.1. It could be
Opening Files

a physical file or that they come from the keyboard or some other input device.
Similarly, the bytes the program sends down the line might end up in a file, or
they could appear on the terminal screen. Although the program often doesn’t
know where bytes are coming from or where they are going, it does know
which line it is using. This line is usually referred to as the logical file to
distinguish it from the physical files on the disk or tape.
Before the program can open a file for use, the operating system must
receive instructions about making a hookup between a logical file (for
example, a phone line) and some physical file or device. When using oper-
ating systems such as IBM’s OS/MVS, these instructions are provided through
job control language (JCL). On minicomputers and microcomputers, more
modern operating systems such as Unix, MS-DOS, and VMS provide the
instructions within the program. For example, in Cobol, 1 the association
between a logical file called inp_file and a physical file called my file .da t is
made with the following statement:
select- inp_file assign to "iayfile.dat".
This statement asks the.operating system to find the physical file named myf
ile . dat and then to make the hookup by assigning a logical file (phone line) to
it. The number identifying the particular phone line that is assigned is
returned.through the variable inp_f ile, which is the file’s logical name. This
logical name is what we use to refer to the file inside the program. Again, the
telephone analogy applies: My office phone is connected to six telephone
lines. When I receive a call I get an intercom message such as, “You have a
call on line three.” The receptionist does not say, “You have a call from
918-123-4567.” I need to have the call identified logically, not physically.

2.2 Opening Files

Once we have a logical file identifier hooked up to a physical file or device,


we need to declare what we intend to do with the file. In general, we have two
options: (1) open an existing file, or (2) create a new file, deleting any
es^fijQgXimle^^ Opening a filTmakeTit ready for use
by the program. We are positioned at the beginning of the file and are

1. These values are defined in an “include" file packaged with your Unix system or C compiler. The
name of the include file is often f c n t l . h or f i l e . h, but it can vary from system to system.
16 Ch a pt e r 2 Fundamental File Processing Operations Op e n in g Fi le s 17

Figure 2.1 The


program relies
on the operating
system to make
connections
between logical
files and physical
files and devices.

ready to start reading or writing. The file contents are not disturbed by the The return value fd and the arguments filename, flags, and pmode
open statement. Creating a file also opens the file in the sense that it is ready have the following meanings:
for use after creation. Because a newly created file has no contents, writing is Argument Type Explanation
initially the only use that makes sense. fd int The file descriptor. Using our earlier analogy, this is
As an example of opening an existing file or creating a new one in C and the phone line (logical file identifier) used to refer to the file
C++, consider the function open, as defined in header file fcntl .h. Although within the program. It is an integer. If there is an error in the
this function is based on a Unix system function, many C++ implementations attempt to open the file, this value is negative.
filename char * A character string containing the physical file name.
for MS-DOS and Windows, including Microsoft Visual C++, also support (Later we discuss pathnames that include directory
open and the other parts of f cntl. h. This function takes two required information about the file’s location. This argument can
arguments and a third argument that is optional: be a pathname.)
fd = open(filename, flags [, pmode]); (continued)
18 Chapter 2 Fundamental File Processing Operations

Argument Type Explanation


flags int The flags argument controls the operation of the open
function, determining whether it opens an existing file for
reading or writing. It can also be used to indicate that you
want to create a new file or open an existing file but delete its
contents. The value of flags is set by performing a bit-wise
OR of the following values, among others.

0 APPEND Append every write operation to the . end of


the file.
O^CREAT
Create and open a file for writing. This has
rio effect if the file already
exists.
0_EXCL
Return an error if O.CREATE is specified
and the file exists.
0_RDONLY Open a file for reading only.

0_RDWR Open a file for reading and writing.


0_TRUNC If the file exists, truncate it to a length of
zero, destroying its contents.
0_WRONLY Open a file for writing only.
Some of these flags cannot be used in combination
with one another. Consult your documentation for details
and for other options.

pmode int ^ O-CREAT is specified, pmode is required. This integer


argument specifies the protection mode for the file. In Unix,'the
pmode is a three-digit octal number that indicates how the file
can be used by the owner (first digit), by members of the owners
group (second digit), and by everyone else (third digit). The
first bit of each octal digit indicates read permission, the second
write permission, and the third execute permission. So, if
pmode is the octal number 0751, the file’s owner has read,
write, and execute permission for the file; the owner’s group
has read and execute permission; and everyone else has only
execute permission:
rwe rwe rwe
pmode = 0751 = -1 1 1 ‘ 1 0 1 001
owner group world
Closing Files 19

Given this description of the open function, we can develop some


examples to show how it can be used to open and create files in C. The
following function call opens an existing file for reading and writing or creates
a new one if necessary. If the file exists, it is opened without change; reading or
writing would start at the file’s first byte, fd = open(filename, 0_RDWR I
0_CREAT, 0751);
The following call creates a new file for reading and writing. If there is already
a file with the name specified in filename, its contents are truncated, fd = open
(filename, O^RDWR I 0_CREAT i 0_TRUNC, 0751);
Finally, here is a call that will create a new file only if there is not already a file
with the name specified in filename. If a file with this name exists, it is not
opened, and the function returns a negative value to indicate an error.
fd = open(filename, 0_RDWR | 0_CREAT I 0_EXCL, 0751);
File protection is tied more to the host operating system than to a specific
language..For example, implementations of C running on systems that support
file protection, such as VAX/VMS, often include extensions to standard C that
let you associate a protection status with a file when you create it.

2.3 Closing Files

In terms of our telephone line analogy, closing a file is like hanging up the
phone. When you hang up the phone, the phone line is available for taking or
placing another call; when you close a file, the logical file name or file
descriptor is available for use with another file. Closing a file that has been used
for output also ensures that everything has been written to the file. As you will
learn in a later chapter, it is more efficient to move data to and from secondary
storage in blocks than it is to move data one byte at a time. Consequently, the
operating system does not immediately send off the bytes we write but saves
them up in a buffer for transfer as a block of data. Closing a file ensures that the
buffer for that file has been flushed of data and that everything we.have written
has been sent to the file.
Files are usually closed automatically by the operating system when a
program terminates normally. Consequently, the execution of a close statement
within a program-is needed only to protect it against data loss in the event that
the program is interrupted and to free up logical filenames for reuse.
20 Chapter 2 Fundamental File Processing Operations

Now that you know how to connect and.disconnect programs to and from
physical files and how to open the files, you are ready to start sending and
receiving data.

2.4 Reading and Writing

Reading and writing are fundamental to file processing; they are the
actions that make file processing an input/output (I/O) operation. The
form of the read and write statements used in different languages varies.
Some languages provide very high-level access to reading and writing and
automatically take care of details for the programmer. Other languages
provide access at a much low;er level. Our use of C and C++ allows us to
explore some of these differences.2

2.4.1 Read and Write Functions


We begin with reading and writing at a relatively low level. It is useful to
have a kind of systems-level understanding of what happens when we send
and receive information to and from a file.
A low-level read call requires three pieces of information, expressed
here as arguments to a generic Read function:
Destination_addr, Size)
Read (Source file,
The Read call must know where it is to read from.
Source_file
We specify the source by logical file name (phone
line) through which data is received. (Remember,
before we do any reading, we must have already
opened the file so the connection between a logical
file and a specific physical file or device exists.)
Read must know where to place the information it
Destination_addr
reads from the input file. In this generic function we
specify the destination by giving the first address of
the memory block where we want to store the data.

Size Finally, Read must know how much information to


bring in from the file. Here the argument is supplied
as a byte count.

2. To accentuate the differences and view I/O operations at something close to a systems level, we use the
fread and fwrite functions in C rather than the higher-level functions such as fgetc, fgets, and so on.
Reading and Writing 21

A Write statement is similar; the only difference is that the data moves in
the other direction:
Write(Destination_file, Source_addr, Size)

Des t ination_f i le The logical file name that is used for sending the data.

Source_addr Write must know where to find the information it


will send. We provide this specification as the first
address of the memory block where the data is stored.
Size The number of bytes to be written must be supplied.

2.4.2 Files with C Streams and C++ Stream Classes .


I/O operations in C and C++ are based on the concept of a stream, which
can be a file or some other source or consumer of data. There are two
different styles for manipulating files in C++. The first uses the standard C
functions defined in header file s tdio . h. This is often referred to as C
streams or C input/output. The second uses the stream classes of header
files iostream.hand f stream, h. We refer to this style as C++ stream
classes.
The header file stdio . li contains definitions of the types and the
operations defined on C streams. The standard input and output of a C
program are streams called stdin and stdout* respectively. Other files
can be associated with streams through the use of the fopen function:
file = fopen (filename, type);

e and the arguments filename and type have


The return value f i l the;s:
following meaning Explanation

Argument Type A pointer to the file descriptor. Type FILE is another


file FILE . name for struct _iobuf. If there is an error in
the attempt to open the file, this value is null, and the
variable errno is set with the error number.
The file name, just as in the Unix open function.
filename char * type char The type argument controls the operation of the open
function, much like the flags argument to open.
*
. The following values are supported:
" r" Open an existing file for input.

" w" Create a new file, or truncate an existing one,


for output.
Chapter 2 Fundamental File Processing Operations

" a" Create a new file, or append to an existing one, for


output.
” r +" Open an existing file for input and output.
” w+" Create a new file, or truncate an existing one, for input
and output.
" a + " Create a new file, or append to an existing one, for
input and output.

Read and write operations are supported by functions fread, fg.et, fwrite, and
fput. Functions fscanf and fprintf are used for. formatted input and output.
Stream classes in C++ support open, close, read, and write operations that
are equivalent to those in stdio. h, but the syntax is considerably different.
Predefined stream objects cin and cout represent the standard input and standard
output files. The main class for access to files, f stream, as defined in header
files iostream. h and f stream. h, has two constructors and a wide variety of
methods. The following constructors and methods are included in the class:
fstream (); // leave the stream unopened 'fstream (char * filename, int
mode); int open (char * filename, int mode); int read (unsigned char *
dest_addr, int size); int write (unsigned char * source_addr,• int size);
The argument f ilename of the second constructor and the method open are just
as we’ve seen before. These two operations attach the fstream to a file. The
value of mode controls the way the file is opened, like the flags and type
arguments previously described. The value is set with a bit-wise or of constants
defined in class ios. Among the options are ios : : in (input), ios : : out (output),
ios : :nocreate (fail if the file does not exist), and ios : : noreplace (fail if the file
does exist). One additional, nonstandard option, ios : : binary, is supported on
many systems to specify that a file is binary. On MS-DOS systems, if ios : :
binary is not specified, the file is treated as a text file. This can have some
unintended consequences, as we will see later.
A large number of functions are provided for formatted input and output.
The overloading, capabilities of C++ are used to make sure that objects are
formatted according to their types. The infix operators >>(extraction) and
«(insertion) are overloaded for input and output, respectively. The header file
iostream. h includes the following overloaded definitions of the insertion
operator (and many others):
Reading and Writing 23

ostream&. operator« (char c) ; ostream& operator<<(unsigned char c);


ostream& operator<<(signed char c); ostream& operator« (const char
*s) ; ostream& operator<<(const unsigned char *s); ostream&
operator<<(const signed char *s); ostream& operator<<(const void *p); -
ostream& operator« (int n) ; ostreamk operator« (unsigned int n) ;
ostream& operator« (long n) ; ostream& operator« (unsigned long n) ;
The overloading resolution rules of C++ specify which function is selected for
a particular call depending on the types of the actual arguments and the types
of the formal parameters. In this case, the insertion function that is used to
evaluate an expression depends on the type of the arguments, particularly the
right argument. Consider the following statements that include insertions into
cout (an object of class ostream):
int n = 25;
cout << "Value of n is "<< n « endl;

The insertion operators are evaluated left to right, and each one returns its left
argument as the result. Hence, the stream cout has first the string “Value of n is
” inserted, using the fourth function in the list above, then the decimal value of
n, using the eighth function in the list. The last operand is the I/O manipulator
endl, which causes an end-of-line to be inserted. The insertion function that is
used for << endl is not in the list above. The header file ios tream. h includes
the definition of endl and the operator that is used for this insertion.
Appendix C includes definitions and examples of many of the formatted
input and output operations.

2.4.3 Programs in C++to Display the Contents of a File


Let’s do some reading and writing to see how these functions are' used. This
first simple file processing program opens a file for input and reads it,
character by character, sending each character to the screen after it 'is read
from the file. This program includes the following steps;
1. Display a prompt for the name of the input file.
2. Read the user’s response from the keyboard into a variable called
filename.
3. Open the file for input.
4. While there are still characters to be read from the input file,
a. read a character from the file;
b. write the character to the terminal screen.
. 5. Close the input file.
Figures 2.2 and 2.3 are C++ implementations of this program using C
streams and C++ stream classes, respectively. It is instructive to look at the
differences between these implementations. The full implementations of these
programs are included in Appendix D.
Steps 1 and 2 of the program involve writing and reading, but in each of
the implementations this is accomplished through the usual functions for
handling the screen and keyboard. Step 4a, in which we read from the input
file, is the first instance of actual file I/O. Note that the f read call using C
streams parallels the low-level, generic Read statement we described earlier;
in truth, we used the f r e a d function as the model for our low-level Read. The
function’s first argument gives the address of a character variable used as the
destination for the data, the second and third arguments are the element size
and the number of elements (in this case the size is 1 byte, and the number of
elements is one), and the fourth argument gives a pointer to the file descriptor
(the C stream version of a logical file name) as the source for the input.

// listc.cpp
// program using C streams to read characters from a file // and write
them to the terminal screen #include <stdio.h> .
main( ) { char ch;
FILE * file; // pointer to file descriptor char
filename[20];
printf("Enter the name of the-file: "); // Step 1
gets(filename); // Step 2
file =fopen(filename, "r"); // Step 3
while (fread(Scch, 1, 1, file) != 0) // Step 4a
fwrite(&ch, 1, 1, stdout); // Step 4b
fclose (file); // Step 5
}

■ Figure 2.2 The file listing program using C streams (listc. cpp).
// listcpp.cpp
// list contents of file using C++.stream classes #include <fstream.h> main () {
char ch;
fstream file; // declare unattached fstream char filename[20];
cout «"Enter the name of the file: " // Step 1 <<flush; // force output . cin »
filename; // Step 2
file . open(filename, ios::in); // Step 3
file . unsetf(ios::skipws);// include white space in read while (1)
{
file >> ch; " // Step 4a
if (file.fail()) break;
cout « ch; // Step 4b
}
file . close(); // Step 5

Figure 2.3 The file listing program using C++ stream classes (listcpp. cpp).

The arguments for the call to operator » communicate the same


information at a higher level. The first argument is the logical file name for
the input source. The second argument is the name of a character variable,
which is interpreted as the address of the variable. The overloading resolution
selects the >> operator whose right argument is a char variable. Hence, the
code implies that only a single byte is to be transferred. In the C++version,
the call file .unsetf (ios : :skipws) causes operator » to include white space
(blanks, end-of-line, tabs, and so on). The default for formatted read with
C++ stream classes is to skip white space.
After a character is read, we write it to standard output in Step 4b. Once
again the differences between C streams and C++ stream classes indicate the
range of approaches to I/O used in different languages. Everything must be
stated explicitly in the f write call. Using the special assigned file descriptor
of st clout to identify the terminal screen as the destination for our writing,
fwrite(&ch, 1, 1, stdout);
Chapter 2 Fundamental File Processing Operations

means: “Write to standard output the contents from memory starting at the
address &ch. Write only one element of one byte.” Beginning C++
programmers should pay special attention to the use of the & symbol in the
fwri te call here. This particular call, as a very low-level call, requires that the
programmer provide the starting address in memory of the bytes to be
transferred.
Stdout, which stands for “standard output,” is a pointer to a struct defined
in the file stdio . h, which has been included at the top of the program. The
concept of standard output and its counterpart standard input are covered
later in Section 2.8 “Physical and Logical Files.”
Again the C++ stream code operates at a higher level. The. right operand
of operator << is a character value. Hence a single byte is transferred to cout.
cout « ch;

As in the call to operator », C++ takes care of finding the address of the
bytes; the programmer need specify only the name of the variable ch that is
associated with that address. .

2.4.4 Detecting End-of-File


The programs in Figs. 2.2 and 2.3 have to know when to end the while loop
and stop reading characters. C streams and C++ streams signal the end-of-file
condition differently. The function f read returns a value that indicates
whether the read succeeded. However, an explicit test is required to see if the
C++ stream read has failed.
The f read call returns the number of elements read as its valnp Tn this
case, if f read returns a value of zero, the program has reached the end of the
Tile. So we construct the while loop to run as long as the f read call finds
something to read.
Each C++ stream has a state that can be queried with function calls.
Figure 2.3 uses the function fail, which returns true (1) if the previous
operation on the stream failed. In this case, f ile . fail'() returns false if the
previous read failed because of trying to read past end-of-file. The following
statement exits the while loop when end-of-file is encountered:
if (file.fail ()) break;
In some languages, including Ada,.a function end_of_f ile can be used to
test for end-of-file. As we read from a file, the operating system keeps track
of our location in the file with a read/write pointer. This is
Seeking

necessary: when the next byte is read, the system knows where to get it. The
end_of_file function queries the system to see whether the read/write
pointer’has moved past the last element in the file. If it has, end_of_f ile
returns true; otherwise it returns false. In Ada, it is necessary to call end_of_f
ile before trying to read the next byte. For an empty file, end_of_f ile
immediately returns true, and no bytes can be read.

2.5 Seeking

In the preceding sample programs we read through the file sequentially,


reading one byte after another until we reach the end of the file. Every time a
byte is read, the operating system moves the read/write pointer ahead, and we
are ready to read the next byte.
Sometimes we want to read or write without taking the time to go through
every byte sequentially. Perhaps we know that the next piece of information
we need is ten thousand bytes away, so we want to jump there. Or perhaps we
need to jump to the end of the file so we can add new information there. To
satisfy these needs we must be able to control the movement of the read/write
pointer.
The action of moving directly to a certain position in a file is often called
seeking. A seek requires at least two pieces of information, expressed here as
arguments to the generic pseudocode function Seek:
Seek(Source_file, Offset)

Source_file The logical file name in which the seek will occur.
Offset The number of positions in the file the pointer is to be
moved from the start of the file.
Now, if we want to move directly from the origin to the 373d position in a file
called data, we don’t have to move sequentially through the first 372
positions. Instead, we can say
Seek(data, 373)

2.5.1 Seeking with C Streams


One of the features of Unix that has been incorporated into C streams is the
ability to view a file as a potentially very large array of bytes that just
Chapter 2 Fundamental File Processing Operations

happens to be kept on secondary storage. In an array of bytes in memory, we


can move to any particular byte using a subscript. The C stream seek function, f
seek, provides a similar capability for files. It lets us set the read/write pointer
to any byte in a file.
The f seek function has the following form:
'pos = fseek(file, byte_offset, origin)
where the variables have the following meanings:
pos A long integer value returned by f seek equal to the posi
tion (in bytes) of the read/write pointer after it has been moved.
file The file descriptor of the file to which the f seek is to be
applied.
by t e_of f set The number of bytes to move from some origin in the file.
The byte offset must be specified as a long integer, hence the • name f
seek for long seek. When appropriate, the by t e_o f f s e t can be
negative.
origin A value that specifies the starting position from which the
byte_of f set is to be taken. The origin can have the value 0, 1, or 2 3
‘ |
0- fseek from the beginning of the file;
1- fseek from the current position;
2- fseek from the end of the file.
The following definitions are included in stdio. h to allow symbolic reference
to the origin values.
#define SEEK_SET 0
ttdefine SEEK_CUR 1
#define SEEK_END 2

The following program fragment shows how you could use f seek to move to a
position that is 373 bytes into a file. .
long pos;
fseek(File * file, long offset, int origin);
File * file;
pos=fseek(file, 373L, 0) ;

3. Although the values 0,1, and 2 are almost always used here, they are not guaranteed to work for all C
implementations. Consult your documentation.
Special Characters in Files

2.5.2 Seeking with C++ Stream Classes


Seeking in C++ stream classes is almost exactly the same as it is in C
streams. There are two mostly syfitactic differences:
■ An object of type f stream has two file pointers: a get pointer for input and
a put pointer for output. Two functions are supplied for seeking: seekg
which moves the get pointer, and seekp which moves the put pointer. It is
not guaranteed that the pointers move separately, but they might. We have
to be very careful in our use of these seek functions and often call both
functions together.
■ The seek operations are methods of the stream classes. Hence the syntax
is file. seekg (byte_offset, origin) and file . seekp (byte_off set, origin).
The value of origin comes from class ios, which is described in more detail
in Chapter 4. The values are ios::beg (beginning of file), iosucur (current
position), and ios::end (end of file).
The following moves both get and put pointers to a byte 373:
file.seekg(373, ios::beg);
file.seekp(373, ios::beg);

2.6 Special Characters in Files

As you create the file structures described in this text, you may
encounter some difficulty with extra, unexpected characters that
turn up in your files with characters that disappear and with
numeric counts that are inserted into your files. Here are some
examples of the kinds of things you might encounter:
■ On many computers you may find that a Control-Z (ASCII
value of 26)Ts appended at the end of your files. Some
applications use this to indicate end-of-file even if you have not
placed it there. This is most likely to happen on MS-DOS
systems.
■ Some systems adopt a convention of indicating end-of-line in a
text file* 4 as a pair of characters consisting of a carriage return

4. When we use the term “text file” in this text, we are referring to a file consisting
entirely of characters from a specific standard character set, such as ASCII or
EBCDIC. Unless otherwise specified, the ASCII character set will be assumed.
Appendix B contains a table that describes the ASCII character set
30 Chapter 2 Fundamental File Processing Operations

value of 13) and a line feed (LF: ASCII value of 10). Sometimes I/O
procedures written for such systems automatically expand single CR
characters or LF characters into CR-LF pairs. This unrequested addition
of characters can cause a great deal of difficulty. Again, you are most
likely to encounter this phenomenon on MS-DOS systems. Using flag
“b” in a C file or mode ios::bin in a C++ stream will suppress these
changes.
■ Users of larger systems, such as VMS, may find that they have just the
opposite problem. Certain file formats under VMS remove carriage
return characters from your file without asking you, replacing them with
a count of the characters in what the system has perceived as a line of text.
These are just a few examples of the kinds of uninvited modifications that
record management systems or that I/O support packages might make to your
files. You will find that they are usually associated with the concepts of a line
of text or the end of a file. In general, these modifications to your files are an
attempt to make your life easier by doing things for you automatically. This
might, in fact, work out for those who want to do nothing more than store
some text in a file. Unfortunately, however, programmers building
sophisticated file structures must sometimes spend a lot of time finding ways
to disable this automatic assistance so they can have complete control over
what they are building. Forewarned is forearmed: readers who encounter these
kinds of difficulties as they build the file structures described in this text can
take some comfort from the knowledge that the experience they gain in
disabling automatic assistance will serve them well, over and over, in the
future.

2.7 The Unix Directory Structure

No matter what computer system you have, even if it is a small PC,


chances are there are hundreds or even thousands of files you have
access to. To provide.convenient access to such large numbers of
files, your computer has some method for organizing its files. In
Unix this is called the file system.
The Unix file system is a tree-structured organization of
directories, with the root of the tree signified bv the
characterTTArTTlTr^^ ing the root, caux:Q.nta.in-t-wo^kinds-Qf
The Unix Directory Structure 31

data, and directories (Fig. 2.4). Since devices such as tape drives are also
treated like files in Unix, directories can also contain references to devices, as
shown in the dev directory in Fig. 2.4. The file name stored in a Unix
directory corresponds to what we call its physical name.
Since every file in a Unix system is part of the file system that begins with
the root, any file can be uniquely identified by giving its absolute pathname.
For instance, the true, unambiguous name of the file “addr” in Fig. 2.4 is /usr6
/mydir/addr. (Note that the / is used both to indicate the root directory and to
separate directory names from the file name.)
When you issue commands to a Unix system, you do so within a direc-
tory, which is called your current directory. A pathname for a file that does
not begin with a / describes the location of a file relative to the current
directory. Hence, if your current directory in Fig. 2.4 is mydir, addr uniquely
identifies the file /usr6/mydir/addr.
The special filename . stands for the current directory, and .. stands for
the parent of the current directory. Hence, if your current directory is
/usr6/mydir/DF,. . /addr refers to the file /usr6/mydir/addr.
32 Chapter 2 Fundamental File Processing Operations

2.8 Physical Devices and Logical Files

2.8.1 Physical Devices as Files


One of the most powerful ideas in Unix is reflected in its notion of what a file
is. In Unix, a file is a sequence of bytes without any implication of how or
wheretHe15ytes are^toredmwhere they originate. This simple concep-
tuaTview of aide makes it possible to do with very few operations what might
require several times as many operations on a different operating system. For
example, it is easy to think of a magnetic disk as the source of a file because
we are used to the idea of storing such things on disks. But in Unix, devices
like the keyboard and the console are also files—in Fig. 2.4, /dev/kbd and
/dev/console, respectively. The keyboard produces a sequence of bytes that
are sent to the computer when keys are pressed; the console accepts a
sequence of bytes and displays their corresponding symbols on a screen.
How can we say that the Unix concept of a file is simple when it allows so
many different physical things to be called files? Doesn’t this make the
situation more complicated, not less so? The trick in Unix is that no matter
what physical representation a file may take, the logical view of a Unix file is
the same. In its simplest form, a Unix file is represented logically by an
integer-the file descriptor. This integer is an index to an array of more
complete information about the file. A keyboard, a disk file, and a magnetic
tape are all represented by integers. Once the integer that describes a file is
identified, a program can access that file. If it knows the logical name of a file,
a program can access that file without knowing whether the file comes from a
disk, a tape, or a connection to another computer.
Although the above discussion is directed at Unix files, the same capa-
bility is available through the stdio functions fopen, fread, and so on. Similar
capabilities are present in MS-DOS, Windows, and other operating systems.

2.8.2 The Console,the Keyboard, and Standard Error


We see an example of the duality between devices and files in the listc.cpp
program in Fig. 2.2:
f i l e =fopen(filename, " r " ) ; / / Step 3
while (fread(&ch, 1, 1, f i l e ) ! = ' 0 ) / / Step 4a
fwrite(&ch, 1,. 1, stdout); // Step 4b
Physical Devices and Logjcal Files 33

The logical file is represented by the value returned by the f open call. We
assign this integer to the variable file in Step 3. In Step 4b, we use the value
stdout, defined in stdi o. h> to identify the console as the file to be written to.
There are two other files that correspond to specific physical devices in
most implementations of C streams: the keyboard is called stdin (standard
input), and the error file is called stderr (standard error). Hence, stdin is the
keyboard on your terminal. The statement fread(&ch, 1, 1, stdin);
reads a single character from your terminal. Stderr is an error file which, like
stdout, is usually just your console. When your compiler detects an error, it
generally writes the error message to this file, which normally means that the
error message turns up on your screen. As with stdin, the values stdin and
stderr are usually defined in stdio .h.
Steps 1 and 2 of the file listing.program also involve reading and writing
from stdin or stdout. Since an enormous amount of I/O involves these
devices, most programming languages have special functions to perform
console input and output—in list.cpp, the C functions print f and gets are
used. Ultimately, however, printf and gets send their output through stdout
and stdin, respectively. But these statements hide important elements of the
I/O process. For our purposes, the second set of read and write statements is
more interesting and instructive.

2.8.1 I/O Redirection and Pipes


Suppose you would like to change the file listing program so it writes its
output to a regular file rather than to stdout. Or suppose you wanted to use
the output of the file listing program as input to another program. Because it
is common to want to do both of these, operating systems provide convenient
shortcuts for switching between standard I/O (stdin and stdout) and regular
file I/O. These shortcuts are called I/O redirection and pipes.5
I/O redirection lets you specify at execution time alternate files for input
or output. The notations for input and output redirection on the command
line in Unix are
< file (redirect stdin to "file")
> file (redirect stdout to "file")

5. Strictly speaking, I/O redirection and pipes are part of a Unix shell, which is the
command interpreter that sits on top of the core Unix operating system, the kernel.
For the purpose of this discussion, this distinction is not important
34 Chapter 2 Fundamental File Processing Operations

For example, if the executable file listing program is called “list.exe,” we


redirect the output from stdout to a file called “myfile” by entering the line
list.exe > myfile

What if, instead of storing the output from the list program in a file, you
wanted to use it immediately in another program to sort the results? Pipes let
you do this. The notation for a pipe in Unix and in MS-DOS is I. Hence,
programl I program2
means take any stdout output from programl and use it in place of any stdin
input to program2. Because Unix has a special program called sort, which
takes its input from stdin, you can sort the output from the list program,
without using an intermediate file, by entering
list | sort

Since sort writes its output to stdout, the sorted listing appears on your terminal
screen unless you use additional pipes or redirection to send it elsewhere.

2.8 File-Related Header Files

Unix, like all operating systems, has special names and values that
you must use when performing file operations. For example, some
C functions return a special value indicating end-of-file (EOF)
when you try to read beyond the end of a file.
Recall the flags that you use in an open call to indicate
whether you want read-only, write-only, or read/write access.
Unless we know just where to look, it is often not easy to find
where these values are defined. Unix handles the problem by
putting such definitions in special header files such as
/usr/include, which can be found in special directories.
Header files relevant to the material in this chapter are stdio .
h, iostream. h, f stream.h, fcnt 1. h, and file . h. The C streams are
in stdio . h; C++ streams in iostream. h and f stream. h. Many
Unix operations are in fcntl. h and file.h. EOF, for instance, is
defined on many Unix and MS-DOS systems in stdio . h5 as ar
i
CHAPTER 3

Secondary Storage and


System Software
Good design is always responsive to the constraints of the medium and to the
environment. This is as true for file structure design as it is for carvings in
wood and stone. Given the ability to create, open, and close files, and to seek,
read, and write, we can perform the fundamental operations of file
construction. Now we need to look at the nature and limitations of the devices
and systems used to store and retrieve files in order to prepare ourselves for
file design.
If files were stored just in memory, there would be no separate discipline
called file structures. The general study of data structures would give us all the
tools we need to build file applications. But secondary storage devices are very
different from memory. One difference, as already noted, is that accesses to
secondary storage take much more time than do accesses to memory. An even
more important difference, measured in terms of design impact, is that not all
accesses are equal. Good file structure design uses knowledge of disk and tape
performance to arrange data in ways that minimize access costs. ;
In this chapter we examine the characteristics of secondary storage
devices. We focus on the constraints that shape our design work in the chapters
that follow. We begin with a look at the major media used in the storage and
processing of files, magnetic disks, and tapes. We follow this with an overview
of the range of other devices and media used for secondary storage. Next, by
following the journey of a byte, we take a brief look at the many pieces of
hardware and software that become involved when a byte is sent by a program
to a file on a disk. Finally, we take a closer look .at one of the most important
aspects of file management—buffering. '
46 Chapter 3 Secondary Storage and System Software

3.1 Disks

Compared with the time it takes to access an item in memory, disk accesses are
always expensive. However, not all disk accesses are-equally expensive. This
has to do with the way a disk drive works. Disk drives1 belong to a class of
devices known as direct access storage devices (DASDs) because they make it
possible to access data directly. DASDs are contrasted with serial devices, the
other major class of secondary storage devices. Serial devices use media such
as magnetic tape that permit only serial access,. which means that a particular
data item cannot be read or written until all of the data preceding it on the tape
have been read or written in order.
Magnetic disks come in many forms. So-called hard disks offer high
capacity and low cost per bit. Hard disks are the most common disk used in
everyday file processing. Floppy disks are inexpensive, but they are slow and
hold relatively little data. Floppies are good for backing up individual files or
other floppies and for transporting small amounts of data. Removable disks use
disk cartridges that can be mounted on the same drive at different times,
providing a convenient form of backup storage that also makes it possible to
access data directly. The Iomega Zip (100 megabytes per cartridge) and Jaz (1
gigabyte per cartridge) have become very popular among PC users.
Nonmagnetic disk media, especially optical discs, are becoming
increasingly important for secondary storage. (See Sections 3.4 and 3.5 and
Appendix A for a full treatment of optical disc storage and its applications.)

3.1.1 The Organization of Disks


The information stored on a disk is stored on the surface of one or more platters
(Fig. 3.1). The arrangement is such that the information is stored in successive
tracks on the surface of the disk (Fig. 3.2). Each track is often divided into a
number of sectors. A sector is the smallest addressable portion of a disk. When
a read statement calls for a particular byte from a disk file, the computer
operating system finds the correct surface, track, and sector, reads the entire
sector into a special area in memory called a buffer, and then finds the
requested byte within that buffer.

1. When we use the terms disks or disk drives, we are referring to magnetic disk media.
Disks 47

Figure 3.1 Schematic illustration of disk drive.

Tracks Sectors

Disk drives typically have a number of platters. The tracks that are directly
above and below one another form a cylinder (Fig. 3.3). The signif- icance of
the cylinder is that all of the information on a single cylinder can
48
Chapter 3 Secondary Storage and System Software

be accessed without moving the arm- that holds the read/write heads. Moving
this arm is called seeking. This arm movement is usually the slowest part of
reading information from a disk.

3.1.2 Estimating Capacities and Space Needs


Disks range in storage capacity from hundreds of millions to billions of bytes.
In a typical disk, each platter has two surfaces, so the number of tracks per
cylinder is twice the number of platters. The number of cylinders is the same as
the number of tracks on a single surface, and each track has the same capacity.
Hence the capacity of the disk is a function of the number of cylinders, the
number of tracks per cylinder, and the capacity of a track.

Figure 3.3 Schematic illustration of disk drive viewed as a set of seven


cylinders.
Disks 49

The amount of data that can be held on a track and the number of tracks
on a surface depend on how densely bits can be stored on the disk surface.
(This in turn depends on the .quality of the recording medium and the size of
the read/write heads.) In 1991, an inexpensive, low-density disk held about 4
kilobytes on a track and 35 tracks on a 5-inch platter. In 1997, a Western
Digital Caviar 850-megabyte disk, one of the smallest disks being
manufactured, holds 32 kilobytes per track and 1,654 tracks on each surface
of a 3-inch platter. A Seagate Cheetah high performance 9-gigabyte disk (still
3-inch platters) can hold about 87 kilobytes on a track and 6526 tracks on a
surface. Table 3.1 shows how a variety of disk drives compares in terms of
capacity, performance, and cost.
Since a cylinder consists of a group of tracks, a track consists of a group
of sectors, and a sector consists of a group of bytes, it is easy to compute
track, cylinder, and drive capacities.
Track capacity = number of sectors per track X bytes.per sector Cylinder
capacity = number of tracks per cylinder X track capacity Drive capacity =
number of cylinders X cylinder capacity.

Table 3.1 Specifications of the Disk Drives


:
Western Digital Caviar Western Digital
Seagate Cheetah 9
AC22100 Caviar AC2850
Characteristic

Capacity 9000 MB 2100 MB 850 MB


Minimum (track-to-track) seek
time
0.78 msec 1 msec 1 msec
Average seek time 8 msec 12 msec 10 msec
Maximum seek time 19 msec 22 msec 22 msec

Spindle speed 10000 rpm. 5200 rpm 4500 rpm


Average rotational delay 3 msec 6 msec 6.6 msec
Maximum transfer rate
6 msec/track, or 14 506 12 msec/track, or 2796 13.3 msec/track, or
bytes/msec bytes/msec 2419,bytes/msec

Bytes per sector - 512 512 512


Sectors per track 170 63 63

Tracks per cylinder 16 16 16


Cylinders 526 4092 1654
If we know the number of bytes in a file, we can-use these relationships
to compute the amount of disk space the file is likely to require. Suppose, for
instance, that we want to store a file with fifty thousand fixed-length data
records on a "typical” 2.1-gigabyte small computer disk with the following
characteristics:
Number of bytes per sector =512
Number of sectors per track = 63
Number of tracks per cylinder = 16
Number of cylinders = 4092
How many cylinders does the file require if each data record requires 256
bytes? Since each sector can hold two records, the file requires

^-QOO = 25 000
sectors 2
One cylinder can hold ■ •
63 X 16 = 1008 sectors
so the number of cylinders required is approximately
= 24.8 cylinders
1008 ■

Of course, it may be that a disk drive with 24.8 cylinders of available space
does, not have 24.8 physically contiguous cylinders available. In this likely
case, the file might, in fact, have to be spread out over dozens, perhaps
even.hundreds, of cylinders.

3.1.2 Organizing Tracks by Sector


There are two basic ways to organize data on a disk: by sector and by user-
defined block. So far, we have mentioned only sector organizations. In this
section we examine sector organizations more closely. In the following
section we will look at block organizations.

The Physical Placement of Sectors


There are several views that one can have of the organization of sectors on a
track. The simplest view, one that suffices for most users most of the time, is
that sectors are adjacent, fixed-sized segments of a track that happen to hold a
file (Fig. 3.4a). This is often a perfectly adequate way to view a file logically,
but it may not be a good way to store sectors physically.
Figure 3.4 Two views of the organization of sectors on a thirty-two-sector
track.

When you want to read a series of sectors that are all in the same track,
one right after the other, you often cannot read adjacent sectors. After reading
the data, it takes the disk controller a certain amount of time to process the
received information before it is ready to accept more. If logically adjacent
sectors were placed on the disk so they were also physically adjacent, we
would miss the start of the following sector while we were processing the one
we had just read in. Consequently, we would be able to read only one sector
per revolution of the disk.
I/O system designers have approached this problem by interleaving the
sectors: they leave an interval of several physical sectors between logically
adjacent sectors. Suppose our disk had an interleaving factor of 5. The
assignment of logical sector content to the thirty-two physical sectors in a
track is illustrated in Fig. 3.4(b). If you study this figure, you can see that it
takes five revolutions to read the entire thirty-two sectors of a track. That is a
big improvement over thirty-two revolutions.
In the-early 1990s, controller speeds improved so that disks can now
offer 1:1 interleaving. This means that successive sectors are physically
adjacent, making it possible to read an entire track in a single revolution of the
disk.
52 Chapter 3 Secondary Storage and System Software

Clusters

Another view of sector organization, also designed to improve performance,


is the view maintained by the part of a computer’s operating system that we
call the file manager. When a program accesses a file, it is the file manager’s
job to map the logical parts of the file to their corresponding physical
locations. It does this by viewing the file as a series of clusters of sectors. A
cluster is a fixed number of contiguous sectors.2 Once a given cluster has been
found on a disk, all sectors in that cluster can be accessed without requiring an
additional seek.
To view a file as a series of clusters and still maintain the sectored view,
the file manager ties logical sectors to the physical clusters they belong to by
using a file allocation table (FAT). The FAT contains a list of all the clusters
in a file, ordered according to the logical order of the sectors they contain.
With each cluster entry in the FAT is an entry giving the physical location of
the cluster (Fig. 3.5).
On many systems, the system administrator can decide how many sectors
there should be in a cluster. For instance, in the standard physical disk
structure used by VAX systems, the system administrator sets the cluster size
to be used on a disk when the disk is initialized. The default value is
3512-byte sectors per cluster, but the cluster size may be set to any value
between 1 and 65 535 sectors. Since clusters represent physically contiguous
groups of sectors, larger clusters will read more sectors without seeking, so
the use of large clusters can lead to substantial performance gains when a file
is processed sequentially.

Extents
Our final view of sector organization represents a further attempt to emphasize
physical contiguity of sectors in a file and to minimize seeking even more. (If
you are getting the idea that the avoidance of seeking is an important part of
hie design, you are right.) If there is a lot of free room on a disk, it may be
possible to make a file consist entirely of contiguous clusters. When this is the
case, we say that the file consists of one extent: all of its sectors, tracks, and
(if it is large enough) cylinders form one contiguous whole (Fig. 3.6a on page
54). This is a good situation, especially if the file is to be processed
sequentially, because it means that the whole file can be accessed with a
minimum amount of seeking.

2. It is not always physically contiguous; the degree of physical contiguity is determined by the
interleaving factor.
Disks 53

File allocation table


(FAT)

Figure 3.5 The file manager determines which cluster in the file has the
sector that is to be accessed.

If there is not enough contiguous space available to contain an entire file,


the file is divided into two or more noncontiguous parts. Each part is an extent.
When new clusters are added to a file, the file manager tries to make them
physically contiguous to the previous end of the file, but if space is
unavailable, it must add one or more extents (Fig. 3.6b). The most important
thing to understand about .extents is that as the number of extents in a file
increases, the file becomes more spread out on the disk, and the amount of
seeking required to process the file increases.

Fragmentation
Generally, all sectors on a given drive must contain the same number of bytes.
If, for example, the size of a sector is 512 bytes and the size of all records in a
file is 300 bytes, there is no convenient fit between records and sectors. There
are two ways to deal with this situation: store, only one record per sector, or
allow records to span sectors so the beginning of a record might be found in
one sector and the end of it in another (Fig. 3.7).
The first option has the advantage that any record can be retrieved by
retrieving just one sector, but it has the disadvantage that it might leave an
enormous amount of unused space within each sector. This loss of space
54 Chapter 3 Secondary Storage and System Software

Figure 3.6 File extents (shaded area represents space on disk used by a
single file).

within a sector is called internal fragmentation. The second option has the
advantage that it loses no space from internal fragmentation, but it has the
disadvantage that some records may be retrieved only by accessing two sectors.
Another potential source of internal fragmentation results from the . use
of clusters. Recall that a cluster is the smallest unit of space that can be
allocated for a file. When the number of byte's in a file is not an exact multiple
of the cluster size, there will be internal fragmentation in the last extent of the
file. For instance, if a cluster consists of three 512-byte sectors, a file containing
1 byte would use up 1536 bytes on the disk; 1535 bytes would be wasted due
to internal fragmentation.
Clearly, there are important trade-offs in the use of large cluster sizes. A
disk expected to have mainly, large files that will often be processed
sequentially would usually be given a large cluster size, since internal frag-
mentation would not be a big problem and the performance gains might be
great. A disk holding smaller files or files that are usually accessed only
randomly would normally be set up with small clusters. -
Disks 55

(b)

Figure 3.7 Alternate record organization within sectors (shaded areas


represent data records, and unshaded areas represent unused space).

3.1.4 Organizing Tracks by Block


Sometimes disk tracks are not divided into sectors, but into integral numbers of
user-defined blocks whose sizes can vary. (Note: The word block has a different
meaning in the context of the Unix I/O system. See Section 3.7 for details.)
When the data on a track is organized by block, this usually means that the
amount of data transferred in a single I/O operation can vary depending on the
needs of the software designer, not the hardware. Blocks can normally be either
fixed or variable in length, depending on the requirements of the file designer
and the capabilities of the operating system. As with sectors, blocks are often
referred to as physical records. In this context, the physical record is the smallest
unit of data that the operating system supports on a particular drive. (Sometimes
the word block is used as a synonym for a sector or group of sectors. To avoid
confusion, we do not use it in that way here.) Figure 3.8 illustrates the
difference between one view of data on a sectored track and that on a blocked
track.
A block organization does not present the sector-spanning and frag-
mentation problems of sectors because blocks can vary in size to fit the logical
organization of the data. A block is usually organized to hold an integral number
of logical records. The term blocking factor is used to indicate the number of
records that are to be stored in each block in a file.
56 Chapter 3 Secondary Storage and System Software

(b)

Fi g ure 3 ,8 Sector organization versus block organization.

Hence, if we had a file with 300-byte records, a block-addressing scheme


would let us define a block to be some convenient multiple of 300 bytes,
depending on the needs of the program. No space would be lost to internal
fragmentation, and there would be no need to load two blocks to retrieve one
record.
Generally speaking, blocks are superior to sectors when it is desirable to
have the physical allocation of space for records correspond to. their logical
organization. (There are disk drives that allow both sector addressing and block
addressing, but we do not describe them here. See Bohl, 1981.)
In block-addressing schemes, each block of data is usually accompanied
by one or more subblocks containing extra information about the data block.
Typically there is a count subblock that contains (among other things) the
number of bytes in the accompanying data block (Fig. 3.9a). There may also be a
key subblock containing the key for the last record in the data block (Fig.
3.9b). When key subblocks are used, the disk controller can search a track for a
block or record identified by a given key. This means that a program can ask its
disk drive to search among all the blocks on a track for a block with a desired
key. This approach can result in much more efficient searches than are
normally possible with sector-addressable schemes, in which keys generally
cannot be interpreted without first loading them into primary memory.

3.1.5 Nondata Overhead


Both blocks and sectors require that a certain amount of space be taken up on
the disk in the form of nondata overhead. Some of the overhead consists of
information that is stored on the disk during preformatting, which is done
before the disk can be used.
Disks 57

Count Data Count Data


subblock subblock subblock subblock

(a)

Count Key Data Count Key Data


subblock subblock subblock subblock subblock subblock

(b)

Figure 3.9 Block addressing requires that each physical data block be accompanied by one
or more subblocks containing information about its contents.

On sector-addressable disks, preformatting involves storing, at the .


beginning of each sector, information such as sector address, track address, and
condition (whether the sector is usable or defective). Preformatting also
involves placing gaps and synchronization marks between fields of information
to help the read/write mechanism distinguish between them. This nondata
overhead usually is of no concern to the programmer, When the sector size is
given for a certain drive, the programmer can assume that this is the amount
of actual data that can be stored in a sector.
On a block-organized disk, some of the nondata overhead is invisible to
the programmer, but some of it must be accounted for. Since subblocks and
interblock gaps have to be provided with every block, there is generally more
nondata information provided with blocks than with sectors. Also, since the
number and size of blocks can vary, from one application to another, the
relative amount of space taken up by. overhead can vary when block
addressing is used. This is illustrated in the following example.
Suppose we have a block-addressable disk drive with 20 000 bytes per
track and the amount of space taken up by subblocks and interblock gaps is
equivalent to 300 bytes per block. We want to store a file containing 100-
byte records on the disk. How many records can be stored per track if the
blocking factor is 10? If it is 60?
of data and
1. If there are ten 100-byte records per block, each block holds 1000 bytes
uses 300 + 1000, or 1300, bytes of track space when overhead is taken
into account. The number of blocks that can fit on a 20 000-byte track can
be expressed as
Chapter 3 Secondary Storage and System Software

20 000 = 15.38 = 15
1300

So fifteen blocks, or 150 records, can be stored per track. (Note that we
have to take the floor of the result because a block cannot span two
tracks.)
2. If there are sixty 100-byte records per block, each block holds 6000 bytes
of data and uses 6300 bytes of track space. The number of blocks per track
can be expressed as •
20 000 _ 3
6300

So three blocks, or 180 records, can be stored per track.


Clearly, the larger blocking factor can lead to more efficient use of
storage. When blocks are larger, fewer blocks are required to hold a file, so
there is less space consumed by the 300 bytes of overhead that accompany
each block.
Can we conclude from this example that larger blocking factors always
lead to more, efficient storage? Not necessarily. Since we can put only an
integral number of blocks on a track and since tracks are fixed in length, we
almost always lose some space at the end of a track. Here we have the internal
fragmentation problem again, but this time it applies to fragmentation within a
track. The greater the block size, the greater potential amount of internal track
fragmentation. What would have happened if we had chosen a blocking factor
of 98 in the preceding example? What about 97?
The flexibility introduced by the use of blocks, rather than sectors, can
save time, since it lets the programmer determine to a large extent how data is
to be organized physically on a disk. On the negative side, blocking schemes
require the programmer and/or operating system to do the extra work of
determining the data organization. Also, the very flexibility introduced by the
use of blocking schemes precludes the synchronization of I/O operations with
the physical movement of the disk, which sectoring permits. This means that
strategies such as sector interleaving cannot be used to improve performance.

3.1.1 The Cost of a Disk Access


To give you a feel for the factors contributing to the total amount of time
needed to access a file on a fixed disk, we calculate some access times. A disk
access can be'divided into three distinct physical operations, each with its own
cost: seek time, rotational delay, and transfer time.
Disks 59

Seek Time
Seek time is the time required to move the access arm to the correct cylinder.
The amount of time spent seeking during a disk access depends, of course, on
how far the arm has to move. If we are accessing a file sequentially and the
file is packed into several consecutive cylinders, seeking needs to be done
only after all the tracks on a cylinder have been processed, and then the
read/write head needs to move the width of only one track. At the other
extreme, if we are alternately accessing sectors from two files that are stored
at opposite extremes on a disk (one at the innermost cylinder, one at the
outermost cylinder), seeking is very expensive. •
Seeking is likely to be more costly in a multiuser environment, where
several processes are contending for use of the disk at one time, than in a
single-user environment, where disk usage is dedicated to one process.
Since seeking can be very costly, system designers often go to great
extremes to minimize seeking. In an application that merges three files, for
example, it is not unusual to see the three input files stored on three different
drives and the output file stored on a fourth drive, so no seeking need be done
as I/O operations jump from file to file.
Since it is usually impossible to know exactly how many tracks will be
traversed in every seek, we usually try to determine the average seek time
required for a particular file operation. If the starting and ending positions for
each access are random, it turns out that the average seek traverses one- third
of the total number of cylinders that the read/write head ranges over.3
Manufacturers’ specifications for disk drives often list this figure as the
average seek time for the drives. Most hard disks available today have
average-seek times of less than 10 milliseconds (msec), and high-perfor-
mance disks have average seek times as low as 7.5 msec.

Rotational Delay

Rotational delay refers to the time it takes for the disk to rotate so the sector
we want is under the read/write head. Hard disks usually rotate at about 5000
rpm, which is one revolution per 12 msec. On average, the rotational delay is
half a revolution, or about 6 msec. On floppy disks, which often rotate at only
360 rpm, average rotational delay is a sluggish
83.3 msec.

3. Derivations of this result, as well as more detailed and refined models, can be found in Wiederhold
(1983), Knuth (1998), Teory and Fry (1982), and Salzberg (1988).
60 Chapter 3 Secondary Storage and System Software

As in the case of seeking, these averages apply only when the read/write
head moves from some random place on the disk surface to the target track. In
many circumstances, rotational delay can be much less than the average. For
example, suppose that you have a file that requires two or more tracks, that
there are plenty of available tracks on one cylin- .der, and that you write the
file to disk sequentially, with one write call. When the first track is filled, the
disk can immediately begin writing to the second track, without any rotational
delay. The “beginning” of the second track is effectively staggered by just the
amount of time it takes to switch from the read/write head on the first track to
the read/write head on the ' second. Rotational delay, as it were, is virtually
nonexistent. Furthermore, when you read the file back, the position of data on
the second track ensures that there is no rotational delay in switching from one
track to another. Figure 3.10 illustrates this staggered arrangement.

TransferTime .
Once the data we want is under the read/write head, it can be transferred. The
transfer time is given by the formula
„ r . number of bytes transferred w
Transfer time = ------------------ J ------------- : ---- X rotation time
number of bytes on a track
If a drive is sectored, the transfer time for one sector depends on the number of
sectors on a track. For example, if there are sixty-three sectors per track, the
time required to transfer one sector would be 1/63 of a revo-

Figure 3.10 When a single file can span several tracks on a cylinder, we can
stagger the beginnings of the tracks to avoid rotational delay when moving
from track to track during sequential access.
Disks 61

lution, or 0.19 msec. The Seagate Cheetah rotates at 10 000 rpm. The transfer
time for a single sector (170 sectors per track) is 0.036 msec. This results in a
peak transfer rate of more than 14 megabytes per second.

Some Timing Computations


Let’s look at two different file processing situations that show how different
types of file access can affect access times. We will compare the time it takes
to access a file in sequence with the time it takes to access all of the records in
the file randomly. In the former case, we use as much of the file as we can
whenever we access it. In the random-access case, we are able to use only one
record on each access.
The basis for our calculations is the high.performance Seagate Cheetah
9-gigabyte fixed disk described in Table 3.1. Although it is typical only of a
certain class of fixed disk, the observations we draw as we perform these
calculations are quite general. The disks used with personal computers are
smaller and slower than this disk, but the nature and relative costs of the factors
contributing to total access times are essentially the same.
The highest performance for data transfer is achieved when files are in
one-track units. Sectors are interleaved with an interleave factor of 1, so data
on a given track can be transferred at the stated transfer rate.
Let’s suppose that we wish to know how long it will take, using this drive,
to read an 8 704 000-byte file that is divided into thirty-four thousand 256-byte
records. First we need to know how the file is distributed on the disk. Since the
4096-byte cluster holds sixteen records, the file will be stored as a sequence of
2125 4096-byte sectors occupying one hundred tracks.
This means that the disk needs one hundred tracks to hold the entire 8704
kilobytes that we want to read. We assume a situation in which the one hundred
tracks are randomly dispersed over the surface of the disk. (This is an extreme
situation chosen to dramatize the point we want to make. Still, it is not so
extreme that it could not easily occur on a typical overloaded disk that has a
large number of small files.)
Now we are ready to calculate the time it would take to read the 8704-
kilobyte file from the disk. We first estimate the time it takes to read the file
sector by sector in sequence. This process involves the following operations
for each track:

Average seek 8 msec


Rotational delay 3 msec
Read one track 6 msec
Total 17
Chapter 3 Secondary Storage and System Software

We want to find and read one hundred tracks, so


Total time = 100 X 1.7 msec = 1700 msec = 1.7 seconds
Now lets calculate the time it would take to read in the same thirty- four
thousand records using random access instead of sequential access. In other
words, rather than being able to read one sector right after another, we assume
that we have to access the records in an order that requires jumping from track
to track every time we read a new sector. This process involves the following
operations for each record:
Average seek 8 msec
Rotational delay 3, msec
Read one cluster (1/21.5 X 6 msec) 0.28 msec
Total 11.28 msec
Total time = 34 000 X 11.28 msec = 9250 msec = 9.25 seconds
This difference in performance between sequential access and random
access is very important. If we can get to the right location on the disk and read
a lot of information sequentially, we are clearly much better off than if we have
to jump around, seeking every time we need a new record. Remember that seek
time is very expensive; when we are performing disk operations, we should try
to minimize seeking.

3.2 Magnetic Tape

Magnetic tape units belong to a class of devices that provide no direct


accessing facility but can provide very rapid sequential access to data. Tapes
are compact, stand up well under different environmental conditions, are easy
to store and transport, and are less expensive than disks. Many years ago tape
systems were widely used to store application data. An application that needed
data from a specific tape would issue a request

4. The term cache (as opposed to disk cache) generally refers to a very high-speed block of primary memory
that performs the same types of performance-enhancing operations with respect to memory that a
disk cache does with respect to secondary memory.
for the tape, which would be mounted by an operator onto a tape drive. The
application could then directly read and write on the tape. The tremendous
reduction in the cost of disk systems has changed the way tapes are used. At
present, tapes are primarily used as archival storage. That is, data is written to
tape to provide low cost storage and then copied to disk whenever it is needed.
Tapes are very common as backup devices for PC systems. In high
performance and high volume applications, tapes are commonly stored in
racks and supported by a robot system that is capable of moving tapes between
storage racks and tape drives.

3.2.1 Types of Tape Systems


There has been tremendous improvement in tape technology in the past few
years. There are now a variety of tape formats with prices ranging from $150
to $150,000 per tape drive. For $150, a PC owner can add a tape backup
system, with sophisticated backup software, that is capable of storing 4
gigabytes of data on a single $30 tape. For larger systems, a high performance
tape system could easily store hundreds of terabytes in a tape robot system
costing millions of dollars. Table 3.3 shows a comparison of some current
tape systems.
In the past, most computer installations had a number of reel-to-reel tape
drives and large numbers of racks or cabinets holding tapes. The primary
media was one-half inch magnetic tape on 10.5-inch reels with 3600 feet of
tape. In the next section we look at the format and data transfer capabilities of
these tape systems which use nine linear tracks and are usually referred to as-
nine-track tapes.

Table 3.3 Comparison of some current tape systems

Tape Model Media Format Loading Capacity Tracks Transfer Rate

9-track one-half inch reel autoload 200 MB 9 linear . 1 MB/sec


DLT cartridge robot 35 GB 36 linear . 5 MB/sec

Digital linear
tape
HP Colorado one-quarter inch manual 1.6 GB helical 0.5 MB/sec

T3000 cartridge
robot silo
StorageTek 50 GB helical . 10 MB/sec

Redwood one-half inch cartridge


Magnetic Tape 67

Newer tape systems are usually based on a tape cartridge medium where
the tape and its reels are contained in a box. The tape media formats that are
available include 4 mm, 8 mm, VHS, 1/2 inch, and 1/4 inch.

3.2*2 An Example of a High-Performance Tape System


The StorageTek'Redwood SD3 is one of the highest-performance tape systems
available in 1997. It is usually configured in a silo that contains storage racks, a
tape robot, and multiple tape drives. The tapes are 4-by-4- inch cartridges with
one-half inch tape. The tapes are formatted with heli-. cal tracks. That is, the
tracks are at an angle to the linear direction of the tape. The number of
individual tracks is related to the length of the tape rather than the width of the
tape as in linear tapes. The expected reliable storage time is more than twenty
years, and average durability is 1 million head passes.
The performance of the SD3 is achieved with tape capacities of up to 50
gigabytes and a sustained transfer rate of 11 megabytes per second. This
transfer rate is necessary to store and retrieve data produced by the newest
generation of scientific experimental equipment, including the Hubbell
telescope, the Earth Observing System (a collection of weather satellites),
seismographic instruments, and a variety of particle accelerators.
An important characteristic of a tape silo system is the speed of seeking,
rewinding, and loading tapes. The SD3 silo using 50-gigabyte tapes has an
average seek time of 53 seconds and can rewind in a maximum of 89 seconds.
The load time is only 17 seconds. The time to read or write a full tape is about
75 minutes. Hence, the overhead to rewind, unload, and load is only 3 percent.
Another way to look at this is that any tape in the silo can be mounted in under
2 minutes with no operator intervention.

3.2.3 Organization of Data on Nine-Track Tapes


Since tapes are accessed sequentially, there is no need for addresses to identify
the locations of data on a tape. On a tape, the logical position of a byte within a
file corresponds directly to its physical position relative to the start of the file.
We may envision the surface of a typical tape as a set of parallel tracks, each of
which is a sequence of bits. If there are nine tracks (see Fig. 3.11), the nine bits
that are at corresponding positions in the nine respective tracks are taken to
constitute 1 byte, plus a parity bit. So a byte can be thought of as a one-bit-wide
slice of tape. Such a slice is called a frame.
68 Chapter 3 Secondary Storage and System Software

Figured.! 1 Nine-track tape.

The parity bit is not part of the data but is used to check the validity of the
data. If odd parity is in effect, this bit is set to make the number of 1 bits in the
frame odd. Even parity works similarly but is rarely-used with tapes.
Frames (bytes) are grouped into data blocks whose size can vary from a
few bytes to many kilobytes, depending on the needs of the user. Since tapes
are often read one block at a time and since tapes cannot stop or start
instantaneously, blocks are separated by interblock gaps, which contain no
information and are long enough to permit stopping and starting. When tapes
use odd parity, no valid frame can contain all 0 bits, so a large number of
consecutive 0 frames is used to fill the interrecord gap.
Tape drives come in many shapes, sizes, and speeds. Performance
differences among drives can usually be measured in terms of three quantities:
Tape density—commonly 800,1600, or 6250 bits per inch (bpi) per track, but
recently as much as 30 000 bpi;
Tape speed—commonly 30 to 200 inches per second (ips); and Size of
interblock gap—commonly between 0.3 inch and 0.75 inch..
Note that a 6250-bpi nine-track tape contains .6250 bits per inch per track, and
6250 bytes per inch when the full nine tracks are taken together. Thus in the
computations that follow, 6250 bpi is usually taken to mean 6250 bytes of data
per inch.
Ma gn e ti c Ta p e 69

3.2.4 Estimating Tape Length Requirements


Suppose we want to store a backup copy of a large mailing-list file with one
million 100-byte records; If we want to store the file on a 6250-bpi tape that
has an interblock gap of 0.3 inches, how much tape is needed?
To answer this question we first need to determine what takes up space
on the tape. There are two primary contributors: interblock gaps and data
blocks. For every data block there is an interblock gap. If we let
b = the physical length of a data block, g
= the length of an interblock gap, and n =
the number of data blocks
then the space requirement s for storing the file is
. 5 = n X (b + g)
We know that gis 0.3 inch, but we do not know what b and n are. In fact, b is
whatever we want it to be, and n depends on our choice of b. Suppose we
choose each data block to contain one 100-byte record. Then b, the length of
each block, is given by
block size (bytes per block) 100
b------------------- — --------------------------- = 0.016 inch
tape density (bytes per inch). 6250
and n, the number of blocks, is 1 million (one per record).
The number of records stored in a physical block is called the blocking
factor. It has the same meaning it had when it was applied to the use of blocks
for disk storage. The blocking factor we have chosen here is 1 because each
block has only one record. Hence, the space requirement for the file is

s = 1 000 000 X (0.016 + 0.3) inch =


1 0.00 000 X 0.316 inch = 316 000
inches = 26 333 feet
Magnetic tapes range in length from 300 feet to 3600 feet, with 2400 feet
being the most common length. Clearly, we need quite a few 2400-foot tapes
to store the file. Or do we? You may have noticed that our choice of block size
was not a very smart one from the standpoint of space usage. The interblock
gaps in the physical representation of the file take up about nineteen times as
much space as the data blocks do. If we were to take a snapshot of our tape, it
would look something like this:
Chapter 3 Secondary Storage and System Software

IT IT ...................... - ... -...... ..- . n .. . ........ .... ~rr-


Data Gap Data Gap Data
A Data
Gap

Most of the space on the tape is not used!


Clearly, we should consider increasing the relative amount of space used
for actual data if we want to try to squeeze the file onto one 2400-foot tape. If
we increase the blocking factor, we can decrease the number of blocks, which
decreases the number of interblock gaps, which in turn decreases the amount
of space consumed by interblock gaps. For example, if we increase the
blocking factor from 1 to 50, the number of blocks becomes
1 000 000
n= = 20 000
50

and the space requirement for interblock gaps decreases from 300 000 inches
to 6000 inches. The space requirement for the data is of course the same as was
previously. What has changed is the relative amount of space occupied by the
gaps, as compared to the data. Now a snapshot of the tape would look much
different:

TT Y ..... H..........V A
rc Gap Data
I..
Data Gap Data Gap Data Gap Data

We leave it to you to show that the file can fit easily on one 2400-foot tape
when a blocking factor of 50 is used.
When we compute the space requirements for our file, we produce
numbers that are quite specific to our file. A more general measure of the effect
of choosing different block sizes is effective recording density. The effective
recording density is supposed to reflect the amount of actual data
.that can be stored per inch of tape. Since this depends exclusively on the
relative sizes of the interblock gap and the data block, it can be defined as
number of bytes per block ;
number of inches required to store a block
When a blocking factor of 1 is used in our .example, the number of bytes per
block is 100, and the number of inches required to store a block is 0.316.
Hence, the effective recording density is
100 bytes
------ --------- -- 316.4 bpi
0.316 inches
which is a far cry from the nominal recording density of 6250 bpi.
MagneticTape 71

Either way you look at it, space utilization is sensitive to the relative sizes
of data blocks and interblock gaps. Let us now see how they affect the amount
of time it takes to transmit tape data.

3.2.5 Estimating Data Transmission Times


If you understand the role of interblock gaps and data block sizes in deter-
mining effective recording density, you can probably see immediately that
these two factors also affect the rate of data transmission. Two other factors
that affect the rate of data transmission to or from tape are the nominal
recording density and the speed with which the tape passes the read/write head.
If we know these two values, we can compute the nominal data transmission
rate:
Nominal rate = tape density (bpi) X tape speed (ips)
Hence, our 6250-bpi, 200-ips tape has a nominal transmission rate of
6250 X 200 = 1 250 000 bytes/sec =
1250 kilobytes/sec
This rate is competitive with most disk drives.
But what about those interblock gaps? Once our data gets dispersed by
interblock gaps, the effective transmission rate certainly suffers. Suppose, for
example, that we use our blocking factor of 1 with the same file and tape drive
discussed in- the preceding section (one million 100-byte records, 0.3-inch
gap). We saw that the effective recording density for this tape organization is
316.4 bpi. If the tape is moving at a rate of 200 ips, then its’ effective
transmission rate is
316.4 X 200 = 63 280 bytes/sec =
63.3 kilobytes/sec
a rate that is about one-twentieth the nominal rate!
It should be clear that a blocking factor larger than 1 improves on this
result and that a substantially larger blocking factor improves on it
substantially.
Although there are other factors that can influence performance, block
size is generally considered to be the one variable with the greatest influence
on space utilization and data transmission rate. The other factors we have
included—gap size,-tape speed, and recording density—are often beyond, the
control, of the user. Another factor that can sometimes be important is the time
it takes to start and stop the tape. We consider start/stop time in the exercises at
the end of this chapter.
72 Chapter 3 Secondary Storage and System Software

33 Disk versus Tape

In the past, magnetic tape and magnetic disk accounted for the lion’s share of
all secondary storage applications. Disk was excellent for random access 'and
storage of files for which immediate access was desired; tape was ideal for
processing data sequentially and for long-term storage of files. Over time,
these roles have changed somewhat in favor of disk.
. The major reason that tape was preferable to disk for sequential processing
is that tapes are dedicated to one process, while disk generally serves several
processes. This means that between accesses a disk read/write head tends to
move away from the location where the next sequential access will occur,
resulting in an expensive seek; the tape drive, being dedicated to one process,
pays no such price in seek time.
This problem of excessive seeking has gradually diminished, and disk has
taken over much of the secondary storage niche previously occupied by tape.
This change is largely because of the continued dramatic decreases in the cost
of disk and memory storage. To understand this change fully, we need to
understand the role of memory buffer space in performing I/O. 5 Briefly, it is
that performance depends largely on how big a chunk of file we can transmit at
any time; as more memory space becomes available for I/O buffers, the
number of accesses decreases correspondingly, which means that the number
of seeks required goes down as well. Most systems now available, even small
systems, have enough memory to decrease the number of accesses required to
process most files that disk becomes quite competitive with tape for sequential
processing. This change, along with the superior versatility and decreasing
costs of disks, has resulted in use of disk for most sequential processing, which
in the past was primarily the domain of tape.
This is not to say that tapes should not be used for sequential processing.
If a file is kept on tape and there are enough drives available to use them for
sequential processing, it may be more efficient to process the file directly from
tape than to stream it to disk and process it sequentially. .
Although it has lost ground to disk in sequential processing applications,
tape remains important as a medium for long-term archival storage. Tape is
still far less expensive than magnetic disk, and it is very easy and fast to stream
large files or sets of files between tape and disk. In this context, tape has
emerged as one of our most important media (along with CD-ROM) for
tertiary storage.

5. Techniques for memory buffering are covered in Section 3.9.


Introduction to CD-ROM 73

3.4 Introduction to CD-ROM

CD-ROM is an acronym for Compact Disc, Read-Only Memory.6 It is a CD


audio disc that contains digital data rather than digital sound. CD- ROM is
commercially interesting because it can hold a lot of data and can be
reproduced cheaply. A single disc can hold more than 600 megabytes of data.
That is approximately two hundred thousand printed pages, enough storage to
hold almost four hundred books the size of this one. Replicates can be stamped
from a master disc for about only a dollar a copy.
CD-ROM is read-only (or write-once) in the same sense as a CD audio
disc: once it has been recorded, it cannot be changed. It is a publishing medium,
used for distributing information to many users, rather than a data storage and
retrieval medium like magnetic disks. CD-ROM has become the preferred
medium for distribution of all types of software and for publication of database
information such as telephone directories, zip codes, and demographic
information. There are also many CD-ROM products that deliver textual data,
such as bibliographic indexes, abstracts, dictionaries, and encyclopedias, often
in association with digitized images stored on the disc. They are also used to
publish video information and, of course, digital audio.

3.4.1 A Short History of CD-ROM


CD-ROM is the offspring of videodisc technology developed in the late 1960s
and early 1970s, before the advent of the home VCR. The goal was to store
movies on disc. Different companies developed a number of methods for
storing video signals, including the use of a needle to respond mechanically to
grooves in a disc, much like a vinyl LP record does. The consumer products
industry spent a great deal of money developing the different technologies,
including several approaches to optical storage, then spent years fighting over
which approach should become standard. The surviving format is one called
LaserVision. By the time LaserVision emerged as the winner, the competing
developers had not only spent enormous sums of money but had also lost
important market opportunities. These hard lessons were put to use in the
subsequent development of CD audio and CD-ROM.

6. Usually we spell disk with a k, but the convention among optical disc manufacturers is to spell it with a c.
74 Chapter 3 Secondary Storage and System Software

From the outset, there was an interest in using Laser Vision discs to do
more than just record movies. The LaserVision format supports recording in
both a constant linear velocity (CLV) format that maximizes storage capacity
and a constant angular velocity (CAV) format that enables fast seek
performance. By using the CAV format to access individual video frames
quickly, a number of organizations, including the MIT Media Lab, produced
prototype interactive video discs that could be used to teach and entertain.
In the early 1980s, a number of firms began looking at the possibility of
storing digital, textual information on LaserVision discs. LaserVision stores
data in an analog form; it is, after all, storing an analog video signal. Different
firms came up with different ways of encoding digital information in analog
form so it could be stored on the disc. The capabilities demonstrated in the
prototypes and early, narrowly distributed products were impressive. The
videodisc has a number of performance characteristics that make it a
technically more desirable medium than the CD- ROM; in particular, one can
build drives that seek quickly and deliver information from the disc at a high
rate of speed. But, reminiscent.of the earlier disputes over the physical format
of the videodisc, each of these pioneers in the use of LaserVision discs as
computer peripherals had incompatible encoding schemes and error correction
techniques. There was no standard format, and none of the firms was large
enough to impose its format over the others through sheer marketing muscle.
Potential buyers were frightened by the lack of a standard; consequently, the
market never grew.
During this same period Philips and Sony began work on a way to store
music on optical discs. Rather than storing the music in the kind of analog form
used on videodiscs, they developed a digital data format. Philips and Sony had
learned hard lessons from the expensive standards battles over videodiscs. This
time they worked with other players in the consumer products industry to
develop a licensing system that resulted in the emergence of CD audio as a
broadly accepted, standard format as soon as the first discs and players were
introduced. CD audio appeared in the United States in early 1984. CD-ROM,
which is a digital data format built On top of the CD audio standard, emerged
shortly thereafter. The first commercially available CD-ROM drives appeared
in 1985.
Not surprisingly, the firms that were delivering digital data on
LaserVision discs saw CD-ROM as a threat to their existence. They also
recognized, however, that CD-ROM promised to provide what had always
Introduction to CD-ROM 75

eluded them in the past: a standard physical format. Anyone with a CD- ROM
drive was guaranteed that he or she could find and read a sector off of any disc
manufactured by any firm. For a storage medium to be used in publishing,
standardization at such a fundamental level is essential.
What happened next is remarkable considering the history of standards and
cooperation within an industry. The firms that had been working on products to
deliver computer data from videodiscs recognized that a standard physical
format, such as that provided by CD-ROM, was not enough. A standard
physical format meant that everyone would be able to read sectors off of any
disc. But computer applications do not work in terms of sectors; they store data
in files. Having an agreement about finding sectors, without further agreement
about how to organize the sectors into files, is like everyone agreeing on an
alphabet without having settled on how letters are to be organized into words on
a page. In late 1985 the firms emerging from the videodisc/digital data industry,
all of which were relatively small, called together many of the much larger firms
moving into the CD-ROM industry to begin work on a standard file system that
would be built on top of the CD-ROM format. In a rare display of cooperation,
the different firms, large and small, worked out the main features of a file system
standard by early summer of 1986; that work has become an official
international standard for organizing files on CD-ROM.
The CD-ROM industry is still young, though in the past years it has begun
to show signs of maturity: it is moving away from concentration on such.matters
as disc formats to a concern with CD-ROM applications. Rather than focusing
on the new medium in isolation, vendors are seeing it as an enabling mechanism
for new systems. As it finds more uses in a broader array of applications,
CD-ROM looks like an optical publishing technology that will be with us over
the long term.
Recordable CD drives make it possible for users to store information on
CD. The price of the drives and the price of the blank recordable CDs make this
technology very appealing for backup. Unfortunately, while the speed of CD
readers has increased substantially, with 12X (twelve times CD audio speed) as
the current standard, CD recorders work no faster than 2X, or about 300
kilobytes per second.
The latest new technology for CDs is the DVD, which stands for Digital
Video Disc, or Digital Versatile Disk. The Sony Corporation has developed
DVD for the video market, especially for the new high definition TVs, but DVD
is also available for storing files. The density of both tracks and bits has been
increased to yield a sevenfold increase in storage
76 Chapter 3 Secondary Storage and System Software

capacity. DVD is also available in a two-sided medium that yields 10 gigabytes


per disc.

3.4.2 CD-ROM as a File Structure Problem


■ CD-ROM presents interesting file structure problems because it is a medium
with great strengths and weaknesses. The strengths of CD-ROM include its
high storage capacity its inexpensive price, and its durability. The key
weakness is that seek performance on a CD-ROM is very slow, often taking
from a half second to a second per seek. In the introduction to this textbook we
compared memory access and magnetic disk access and showed that if memory
access is analogous to your taking twenty seconds to look up something in the
index to this textbook, the equivalent disk access would take fifty-eight days, or
almost 2 months. With a CD- ROM the analogy stretches the disc access to
more than two and a half years! This kind-of performance, or lack of it, makes
intelligent file structure design a critical concern for CD-ROM applications.
CD-ROM provides an excellent test of our ability to integrate and adapt the
principles we have developed in the preceding chapters of this book. .

3.5 Physical Organization of CD-ROM

CD-ROM is the child of CD audio. In this instance, the impact of heredity is


strong, with both positive and negative aspects. Commercially, the CD audio
parentage is probably wholly responsible for CD-ROMs viability. It is because
of the enormous CD audio market that it is possible to make these discs so
inexpensively. Similarly, advances in the design and decreases in the costs of
making CD audio players affect the performance and price of CD-ROM drives.
Other optical disc media without the benefits of •this parentage have not
experienced the commercial success of CD-ROM.
On the other hand, making use of the manufacturing capacity associated
with CD audio means adhering to the fundamental physical organization of the
CD audio disc. Audio discs are designed to play music, not to provide fast,
random access to data. This biases CD toward having high storage capacity and
moderate data transfer rates and against decent seek performance. If an
application requires good random-access performance, that performance has to
emerge from our file structure design efforts; it won’t come from anything
inherent in the medium.
Physjcal Organization of CD-ROM 77

3.5.1 Reading Fits and Lands


CD-ROMs are stamped from a master disc. The master is formed by using the
digital data we want to encode'to turn a powerful laser on and off very quickly.
The master disc, which is made of glass, has a coating that is changed by the
laser beam. When the coating is developed, the areas hit by the laser beam turn
into pits along the track followed by the beam. The smooth, unchanged areas
between the pits are called lands. The copies formed from the master retain this
pattern of pits and lands.
When we read the stamped copy of the disc, we focus a beam of laser
light on the track as it moves under the optical pickup. The pits scatter the light,
but the lands reflect most of it back to the pickup. This alternating pattern of
high- and low-intensity reflected light is the signal used to reconstruct the
original digital information. The encoding scheme used for this signal is not
simply a matter of calling a pit a 1 and a land a 0. Instead, the Is are represented
by the transitions from pit to land and back again. Every time the light intensity
changes, we get a 1. The Os are represented by the amount of time between
transitions; the longer between transitions, the more Os we have.
If you think about this encoding scheme, you realize that it is not possible
to have two adjacent Is—-Is are always separated by Os. In fact, due to the
limits of the resolution of the optical pickup, there must be at least two Os
between any pair of Is. This means that the raw pattern of Is and Os has to be
translated to get the 8-bit patterns of Is and Os that form the bytes of the original
data. This translation scheme, which is done through a lookup table, turns the
original 8 bits of data into 14 expanded bits that can be represented in the pits
and lands on the disc; the reading process reverses this translation. Figure 3.12
shows a portion of the lookup table values. Readers who have looked closely
at the specifications for CD players may have encountered the term EFM
encoding. EFM stands for “eight to fourteen modulation” and refers to this
translation scheme.

Figure 3.12 A portion of the


EFM encoding table.
Chapter 3 Secondary Storage and System
Software

It is important to realize that since we represent the Os in the EFM-


encoded data by the length of time between transitions, our ability to read
the data is dependent on moving the pits and lands under the optical pickup
at a precise and constant speed. As we will see, this affects the CD-ROM
drive’s ability to seek quickly.

3.5.4 Structure of a Sector


It is interesting to see how the fundamental design of the CD disc, initially
intended to deliver digital audio information, has been adapted for computer
data storage. This investigation will also help answer the question: If the disc
is capable of storing a quarter of a million printed pages, why does it hold only
an hour’s worth of Roy Orbison?
When we want to store sound, we need to convert a wave pattern into
digital form. Figure 3.14 shows a wave. At any given point in time, the wave
has a specific amplitude. We digitize the wave by measuring the amplitude at
very frequent intervals and storing the measurements. So, the question of how
much storage space we need to represent a wave digitally .turns into two other
questions: How much space does it take to store each amplitude sample? How
often do we take samples?

Figure 3.14 Digital sampling of a wave.


Physical Organization of CD-ROM 81

CD audio uses 16 bits to store each amplitude measurement; that means


that the “ruler” we use to measure the height of the wave has 65 536 different
gradations. To approximate a wave accurately through digital sampling, we
need to take the samples at a rate that is more than twice as frequent as the
highest frequency we want to capture. This makes sense if you look at the wave
in Fig. 3.15. You can see that if we sample at less than twice the frequency of
the wave, we lose information about the variation in the wave pattern. The
designers of CD audio selected a sampling frequency of 44.1 kilohertz, or 44
100 times per second, so they could record sounds with frequencies ranging up
to 20 kilohertz (20 000 cycles per second), which is toward the upper bound of
what people can hear.
So, if we are taking a 16-bit, or 2-byte, sample 44 100 times per second,
we need to store 88 200 bytes per second. Since we want to store stereo sound,
we need double this and store 176 400 bytes per second. You can see why
storing an hour of Roy Orbison takes so much space. .
If you divide the 176 400-byte-per-second storage capacity of the CD
into seventy-five sectors per second, you have 2352 bytes per sector. CD-
ROM divides up this “raw” sector storage as shown in Fig. 3.16 to provide 2
kilobytes of user data storage, along with addressing information, error
detection, and error correction information. The error correction information is
necessary because, although CD audio contains redundancy for error
correction, it is not adequate to meet computer data storage needs.
82 Chapter 3 Secondary Storage and System Software

12 bytes 4 bytes 2,048 bytes 4 bytes 8 bytes 276 bytes


synch sector ID user data error null error
detection correction

Figure 3.T6 Structure of a CD-ROM sector.

The audio error correction would result in an average of one incorrect byte for
every two discs. The additional error correction information stored within the
2352-byte sector decreases this error rate to 1 uncor- rectable byte in every
twenty thousand discs.

3.6 CD-ROM Strengths and Weaknesses

As we say throughout this book, good file design is responsive to the nature of
the medium, making use of strengths and minimizing weaknesses. We begin,
then, by cataloging the strengths and weaknesses of CD-ROM.

3.6.1 Seek Performance


The chief weakness of CD-ROM is the random-access performance. Current
magnetic disk technology is such that the average time for a random data
access, combining seek time and rotational delay, is about 30 msec. On a
CD-ROM, this average access takes 500 msec and can take up to a second or
more. Clearly, our file design strategies must avoid seeks to an even greater
extent than on magnetic disks.

3.6.2 Data Transfer Rate


A CD-ROM drive reads seventy sectors, or 150 kilobytes of data per second.
This data transfer rate is part of the fundamental definition of CD-ROM; it
can’t be changed without leaving behind the commercial advantages of
adhering to the CD audio standard. It is a modest transfer rate, about five times
faster than the transfer rate for floppy disks, and an order of magnitude slower
than the rate for good Winchester disks. The inadequacy of the transfer rate
makes itself felt when We are loading large files, such as those associated with
digitized images. On the other hand, the
CD-ROM Strengths and Weaknesses 83

transfer rate is fast enough relative to the CD-ROM’s seek performance that
we have a design incentive to organize data into blocks, reading more data
with each seek in the hope that we can avoid as much seeking as possible.

3.6.3 Storage Capacity


A CD-ROM holds more than 600 megabytes of data. Although it is possible to
use up this storage area very quickly, particularly if you are storing raster
images, 600 megabytes is big when it comes to text applications. If you decide
to download 600 megabytes of text with a 2400-baud modem, it will take
about three days of constant data transmission, assuming errorless
transmission conditions. Many typical text databases and document collec-
tions published on CD-ROM use only a fraction of the disc’s capacity.
The design benefit arising from such large capacity is that it enables us to
build indexes and other support structures that can help overcome some of the
limitations associated with CD-ROM’s poor seek performance.

3.6.4 Read-Only Access


From a design standpoint, the fact that CD-ROM is a publishing medium, a
storage device that cannot be changed after manufacture, provides significant
advantages. We never have to worry about updating. This not only simplifies
some of the file structures but also means that it is worthwhile to optimize our
index structures and other aspects of file organization. We know that our
efforts to optimize access will not be lost through later additions or deletions.

3.6.5 Asymmetric Writing and Reading


For most media, files are written and read using the same computer system.
Often, reading and writing are both interactive and are therefore constrained
by the need to provide quick response to the user. CD-ROM is different. We
create the files to be placed on the disc once; then we distribute the disc, and it
is accessed thousands, even millions, of times. We are in a position to bring
substantial computing power to the task of file organization and creation, even
when the disc will be used on systems with much less capability. In fact, we
can use extensive batch-mode processing on large computers to try to provide
systems that will perform well on small machines. We make the investment in
intelligent, carefully designed file
84 Chapter 3 Secondary Storage and System Software

structures only once; users can enjoy the benefits of this investment again and
again.

3.7 Storage as a Hierarchy

Although the best mixture of devices for a computing system depends on the
needs of the system’s users, we can imagine any computing system as a
hierarchy of storage devices of different speed, capacity, and cost. Figure
3.17 summarizes the different types of storage found at different levels in
such hierarchies and shows approximately how they compare in terms of access
time, capacity, and cost.

Figure 3.17 Approximate comparisons of types of storage.


A Journey of a Byte 85

3.8 A Journey of a Byte

What happens when a program writes a byte to a file on a dish? We know what
the program does (it makes a call to a write function), and we now know
something about how the byte is stored on a disk. But we haven’t looked at
what happens between the program and the disk. The whole story of what
happens to data between program and disk is not one we can tell here, but we
can give you an idea of the many different pieces of hardware and software
involved and the many jobs that have to be done by looking at an example of a
journey of 1 byte.
Suppose we want to append a byte representing the character P stored in
a character variable ch to a file named in the variable text f i l e stored
somewhere on a disk. From the program’s point of view, the entire journey that
the byte will take might be represented by the statement
write(textfile, ch, 1)
but the journey is much longer than this simple statement suggests.
The write statement results in a call to the computer’s operating system,
which has the task of seeing that the rest of the journey is completed
successfully (Fig. 3.18). Often our program can provide the operating
system'with information that helps it carry out this task more effectively, but
once the operating system has taken over, the job of overseeing the rest of the j
ourney is largely beyond our program’s control.

Figure 3.18 The write statement tells the operating system to send one
character to disk and gives the operating system the location of the
character. The operating system takes over the job of writing, and then
returns control to the calling program.
oo Chapter3 Secondary Storage and System Software

3.8,1 The Fife Manager


An operating system is not a single program but a collection of programs,
each one designed to manage a different part of the computer’s resources.
Among these programs are ones that deal with file-related matters and I/O
devices. We call this subset of programs the operating system’s file manager.
The file manager may be thought of as several layers of procedures (Fig.
3.19), with the upper layers dealing mostly with symbolic, or logical, aspects
of file management, and the lower layers dealing more with the

Logical
1. Theprogram asks the operating system to write the contents of the f
variable c to the next available position in TEXT. . ; j _ - ' v ..y

2. The operating system passes the job on to the file manager. ' " ' ‘ A

3. The ?file manager looks up TEXT in a table'containing information ~-


about .it, such as whether the file is open and available for use, what types of
access are allowed, if any, and what physical file the logical ; name-TEXT
corresponds to.

4. The file manager searches a file allocation table for the physical
location of the-sector that is to contain the byte. .;. -

5. The file managermakes sure that the last sector in the,file has been
stored..ina system I/O buffer in RAM, then deposits the‘P’ in to. its proper
position in the buffer.

6. The file manager gives instructions to the I/O processor about where the
byteis storedin RAM and where it needs to be sent on'the d i s k . -

7. The T/O processor, finds a time when the drive ls available to receive the
.data .and puts the data in proper format for the disk. It may' also ■,
bufferrthe data to send it out in chunks of the proper size for the -
disk. . • -VV:Vr

8. The I/O processor sends the data to the disk controller.

9. The controller instructs: the drive to move the read/write head to the :,
proper track, waits for the desired, sector to come under the *1 < -
read/write head, then sends the byte to the drive to be-deposited, bit-V
' by-bity: on the surface of the disk.

. y
Physical

Figure 3.19 Layers of procedures involved in transmitting a byte from a


program's data area to a file called textfile on disk.
A Journey of a Byte 87

physical aspects. Each layer calls the one below it, until, at the lowest level,
the byte is written to the disk.
. The file manager begins by finding out whether the logical characteristics
of the file are consistent with what we are asking it to do with the file. It may
look up the requested file in a table, where it finds out such things as whether
the file has been opened, what type of file the byte is being sent to (a binary
file, a text file, or some other organization), who the file’s owner is, and
whether write access is allowed for this particular user of the file.
The file manager must also determine where in the file t e x t f i l e the P is
to be deposited. Since the P is to be appended to the file, the file manager
needs to know where the end of the file is—the physical location of the last
sector in the file. This information is obtained from the file allocation table
(FAT) described earlier. From the FAT, the file manager locates the drive,
cylinder, track, and sector where the byte is to be stored.

3.8.2 The I/O Buffer


Next, the file manager determines whether the sector that is to contain the P is
already in memory or needs to be loaded into memory. If the sector needs to
be loaded, the file manager must find an available system I/O buffer space for
it and then read it from the disk. Once it has the sector in a buffer in memory,
the file manager can deposit the P into its proper position in the buffer (Fig.
3.20). The system I/O buffer allows the file manager to read and write data in
sector-sized or block-sized units. In other words, it enables the file manager to
ensure that the organization of data in memory conforms to the organization it
will have on the disk.
Instead of sending the sector immediately to the disk, the file manager
usually waits to see if it can accumulate more bytes going to the same sector
before transmitting anything. Even though the statement w r i t e ( t e x t f i l e
, c h , l ) seems to imply that our character is being sent immediately to the disk,
it may in fact be kept in memory for some time before it is sent. (There are
many situations in which the file manager cannot wait until a buffer is filled
before transmitting it. For instance, if text f i l e were closed, it would have to
flush all output buffers holding data waiting to be written to text f i l e so the
data would not be lost.)

3.8.3 The Byte Leaves Memory:The I/O Processor


and Disk Controller
So far, all of our bytes activities have occurred within the computer’s primary
memory and have probably been carried out by the computer’s
88
Chapter 3 Secondary Storage and System Software •

Figure 3.20 The file manager moves P from the program's data ar.ea to a
system output buffer where it may join other bytes headed for the same place
on the disk. If necessary, the file manager may have to load the corresponding
sector from the disk into the system output buffer.

central processing unit. The byte has traveled along data paths that are
designed to be very fast and are relatively expensive. Now it is time for the byte
to travel along a data path that is likely to be slower and narrower than the one
in primary memory. (A typical computer might have an internal data-path
width of 4 bytes, whereas the width of the path leading to the disk might be
only 2 bytes.)
Because of bottlenecks created by these differences in speed and datapath
widths, our byte and its companions might have to wait for an external data
path to become available. This also means that the CPU has extra time on its
hands as it deals out information in small enough chunks and at slow enough
speeds that the world outside can handle them. In fact, the differences between
the internal and external speeds for transmitting data are often so great that the
CPU can transmit to several external devices simultaneously.
The processes of disassembling and assembling groups of bytes for
transmission to and from external devices are so specialized that it is
unreasonable to ask an expensive, general-purpose CPU to spend its valu
A Journey of a Byte 89

able time doing I/O when a simpler device could do the job and free the CPU to
do the work that it is most suited for. Such a special-purpose device is called an
I/O processor.
An I/O processor may be anything from a simple chip capable of taking a
byte and passing it along one cue, to a powerful, small computer capable of
executing very sophisticated programs and communicating with many devices
simultaneously. The I/O processor takes its instructions from the operating
system, but once it begins processing I/O, it runs independently, relieving the
operating system (and the CPU) of the task of communicating with secondary
storage devices. This allows I/O processes and internal computing to overlap. 7
In a typical computer, the file manager might now tell the I/O processor
that there is data in the buffer to be transmitted to the disk, how much data there
is, and where it is to go on the disk. This information might come in the form of
a little program that the operating system constructs and the I/O processor
executes (Fig. 3.21).
The job of controlling the operation of the disk is done by a device called
a disk controller. The I/O processor asks the disk controller if the disk drive is
available for writing. If there is much I/O processing, there is a good chance
that the drive will not be available and that our byte will have to wait in its
buffer until the drive becomes available.
What happens next often makes the time spent so far seem insignificant in
comparison: the disk drive is. instructed to move its read/write head to the track
and sector on the drive where our byte and its companions are to be stored. For
the first time, a device is being asked to do something mechanical! The.
read/write head must seek to the proper track (unless it is already there) and
then wait until the disk has spun around so the desired sector is under the head.
Once the track and sector are located, the I/O processor (or perhaps the
controller) can send out bytes, one at a time, to the drive. Our byte waits until
its turn comes; then it travels alone to the drive, where it probably is stored in a
little 1-byte buffer while it waits to be deposited on the disk.
Finally, as the disk spins under the read/write head, the 8 bits of our byte
are deposited, one at a time, on the surface of the disk (Fig. 3.21). There the P
remains, at the end of its journey, spinning at a leisurely 50 to 100 miles per
hour.

7. On many systems the I/O processor can take data directly from memory, without further involvement
from the CPU. This process is called direct memory access (DMA). On other systems, the CPU must
place the 'data in special I/O registers before the I/O processor can have access to it.
90 Chapters Secondary Storage and System Software

Figure 3.21 The file manager sends the I/O processor instructions in the
form of an I/O processor program.The I/O processor gets the data from
the system buffer, prepares it for storing on the disk, then sends it to the
disk controller, which deposits it on the surface of the disk.

3.9 Buffer Management

Any user of files can benefit from some knowledge of what happens to data
traveling between a program’s data area and secondary storage. One aspect of
this process that is particularly important is the use of buffers. Buffering
involves working with large chunks of data in memory so the number of
accesses to secondary storage can be reduced. We concentrate on the operation
of system I/O buffers; but be aware that the use of buffers within programs can
also substantially affect performance.

3.9.1 Buffer Bottlenecks


We know that a file manager allocates I/O buffers that are big enough to hold
incoming data, but we have said nothing so far about how many buffers are
used. In fact, it is common for file managers to allocate several buffers for
performing I/O.
Buffer Management 91

To understand the need for several system buffers, consider what happens
if a program is performing both input and output on one character at a time and
only one I/O buffer is available. When the program asks for its first character,
the I/O buffer is loaded with the sector containing the character, and the
character is transmitted to the program. If the program then decides to output a
character, the I/O buffer is fdled with the sector into which the output character
needs to go, destroying its original contents. Then when the next input character
is needed, the buffer contents have to be written to disk to make room for the
(original) sector containing the second input character, and so on.
Fortunately, there is a simple and generally effective solution to this
ridiculous state of affairs, and that is to use more than one system buffer. For
this reason, I/O systems almost always use at least two buffers—one for input
and one for output.
Even if a program transmits data in only one direction, the use of a single
system I/O buffer can slow it down considerably. We know, for instance, that
the operation of reading a sector from a disk is extremely slow compared with
the amount of time it takes to move data in memory, so we can guess that a
program that reads many sectors from a file might have to spend much of its
time waiting for the I/O system to fill its buffer every time a read operation is
performed before it can begin processing. When this happens, the program that
is running is said to be I/O bound— the CPU spends much of its time just
waiting for I/O to be performed. The solution to this problem is. to use more
than one buffer and to have the I/O system filling the next sector or block of data
while the CPU is processing the current one.

3.9.2 Buffering Strategies

Multiple Buffering
Suppose that a program is only writing to a disk and that it is I/O bound. The
CPU wants to be'filling a buffer at the same time that I/O is being performed. If
two buffers are used and I/O-CPU overlapping is permitted, the CPU can be
filling one buffer while the contents of the other are being transmitted to disk.
When both tasks are finished, the roles of the buffers can be exchanged. This
method of swapping the roles of two buffers after each output (or input)
operation is called double buffering. Double buffering allows the operating
system to operate on one buffer while the other buffer is being loaded or emptied
(Fig. 3.22).
92 Chapter 3 Secondary Storage and System Software

(a)

(b)

Figure 3.22 Double buffering: (a) the contents of system I/O buffer 1 are sent
to disk while I/O buffer 2 is.being filled; and (b) the contents of buffer 2 are
sent to disk while I/O buffer 1 is being filled.

This technique of swapping system buffers to allow processing and I/O to


overlap need not be restricted to two buffers. In theory, any number of buffers
can be used, and they can be organized in a variety of ways. The actual
management of system buffers is usually done by the operating system and can
rarely be controlled by programmers who do not work at the systems level. It is
common, however, for programmers to be able to control the number of
system buffers assigned to jobs.
Some file systems use a buffering scheme called buffer pooling: when a
system buffer is needed, it is taken from a pool of available buffers and used.
When the system receives a request to read a certain sector or block, it looks to
see if one of its buffers already contains that sector or block. If no buffer
contains it, the system finds from its pool of buffers one that is not currently in
use and loads the sector or block into it.
Several different schemes are used to decide which buffer to take from a
buffer pool. One generally effective strategy is to take the buffer that is least
recently used. When a buffer is accessed, it is put on a least-recently- used queue so it is
allowed to retain its data until all other less-recently- used buffers have been
accessed. The least-recently-used (LRU) strategy for replacing old data with
new data has many applications in computing.
Buffer Management 93

It is based on the assumption that a block of data that has been used recently is
more likely to be needed in the near future than one that has been used less
recently. (We encounter LRU again in later chapters.)
It is difficult to predict the point at which the addition of extra buffers
ceases to contribute to improved performance. As the cost of memory
continues to decrease, so does the cost of using more and bigger buffers. On
the other hand, the more buffers there are, the more time it takes for the file
system to manage them. When in doubt, consider experimenting with different
numbers of buffers.

Move Mode and Locate Mode


Sometimes it is not necessary to distinguish between a program’s data area
and system buffers. When data must always be copied from a system buffer to
a program buffer (or vice versa), the amount of time taken to perform the move
can be substantial. This way of handling buffered data is called move mode, as
it involves moving chunks of data from one place in memory to another before
they can be accessed.
There are two ways that move mode can be avoided. If the file manager
can perform I/O directly between secondary storage and the program’s data
area, no extra move is necessary. Alternatively, the file manager could use
system buffers to handle all I/O but provide the program with the locations,
using pointer variables, of the system buffers. Both techniques are examples of
a general approach to buffering called locate mode. When locate mode is used,
a program is able to operate directly on data in the I/O buffer, eliminating the
need to transfer data between an I/O buffer and a program buffer.

Scatter/Gather I/O
Suppose you are reading in a file with many blocks, and each block consists of
a header followed by data. You would like to put the headers in one buffer and
the data in a different buffer so the data can be processed as a single entity. The
obvious way to do this is to read the whole block into a single big buffer; then
move the different parts to their own buffers. Sometimes we can avoid this
two-step process using a technique called scatter input. With scatter input, a
single read call identifies not one, but a collection of buffers into which data
from a single block is to be scattered.
The converse of scatter input is gather output. With gather output,
several buffers can be gathered and written with a single write call; this avoids
the need to copy them to a single output buffer. When the cost of
94 Chapter 3 Secondary Storage and System Software

copying several buffers into a single output buffer is high, scatter/gather


can have a significant effect on the running time of a program.
It is not always obvious when features like scatter/gather, locate mode,
and buffer pooling are available in an operating system. You often have to
go looking for them. Sometimes you can invoke them by communicating
with your operating system, and sometimes you can cause them to be
invoked by organizing your program in ways that are compatible with the
way the operating system does I/O. Throughout this text we return many
times to the issue of how to enhance'performance by thinking about how
buffers work and adapting programs and file structures accordingly.'
CHAPTER 4

Fundamental File
Structure Concepts

4.1 Field and Record Organization

When we build file structures, we are making it possible to make data


persistent. That is, one program can create data in memory and store it in a
file and another program can read the file and re-create the data in its memory.
The basic unit of data is the field, which contains a single data value. Fields are
organized into aggregates* either as many copies of a single field (an array)
or as a list of different fields (a reicord). Programming language type
definitions allows us to define the structure of records. When a. record is stored
in memory, we refer to it as an object and refer to its fields as members. When
that object is stored in a file, we call it simply a record.
In this chapter we investigate the many ways that objects can be repre-
sented as records in files. We begin by considering how to represent fields
and continue with representations of aggregates. The simplest representation
is with a file organized as a stream of bytes.
4.1.1 A Stream File
Suppose the objects we wish to store contain name and address information
about a collection of people. We will use objects of class Person, from Section
1.5, “Using Objects in C++” to store information about individuals. Figure 4.1
(and file writestr. cpp) gives a C++function (operator «) to write the fields of a
Person to a file as a stream of bytes.
File writ strm. cpp in Appendix D includes this output function, together
with a function to accept names and addresses from the keyboard and a main
program. You should compile and run this program. We use it as the basis for a
number of experiments, and you can get a better feel for the differences
between the file structures we are discussing if you perform the experiments
yourself.
The following names and addresses are used as input to the program: Mary
Ames Alan Mason

1.23 Maple 90 Eastgate

Stillwater, OK 74075 Ada, OK 74820

When we list the output file on our terminal screen, here is what we see: AmesMaryl23
MapleStillwaterOK74075MasonAian90 EastgateAdaOK74820
The program writes the information out to the file precisely as specified, as a
stream of bytes containing no added information. But in meeting our
specifications, the program creates a kind of reverse FIumpty-Dumpty
problem. Once we put all that information together as a single byte stream,
there is no way to get it apart again.

ostream & operator « (ostream & outputFile, Person & p) { // insert


(write) fields into stream outputFile <<.p.LastName << p.FirstName
<< p.Address << p.City « p.State « p.ZipCode; return outputFile;
}

Fi g ure 4.1 Function to write (<<) a'Person as a stream of bytes.


120 Chapter 4 Fundamental File Structure Concepts

We have lost the integrity of the fundamental organizational units of our


input data; these fundamental units are not the individual characters but
meaningful aggregates of characters, such as “Ames” or “123 Maple.” When we
are working with files, we call these fundamental ag-

A field is a logical notion; it is a conceptual tool. A field does not necessarily


exist in any physical sense, yet it is important to the file’s structure. When we write
out our name and address information as a stream of ' undifferentiated bytes, we
lose track of the fields that make the information meaningful. We need to organize
the file in some way that lets us keep ■ the information divided into fields.

There are many ways of adding structure to files to maintain the identity of
fields. Four of the most common methods follow:
■ Force the fields into a predictable length.
■ Begin each field with a length indicator.
■ Place a delimiter at the end of each field to separate it from the next, field.
■ Use a “keyword = value”, expression to identify each field and its contents.

Method 1: Fix the Length of Fields


The fields in our sample file vary in length. If we force the fields- into
predictable lengths, we can pull them back out of the file simply by counting
our way to the end of the field. We can define a struct in C or a class in C++ to
hold these fixed-length fields, as shown in Fig. 4.2. As you can see, the only
difference between the C and C++ versions is the use of the keyword.struct or
class and the designation of the fields of class Person as public in C++.

1. Readers should not confuse the terms /ie/d and record with the meanings given to them by some
programming languages, including Ada. In Ada, a record is an aggregate data structure that can contain members
of different types, where each member is referred to as a field. As we shall see, there is often a direct
correspondence between these definitions of the terms and the fields and records that are used in files.
However, the terms field and record as we use them have much more general meanings than they do in
Ada.
Field and Record Organization 121

In C: In C++;
struct Person{ class Person { public: char
char last [11]; char last [11]; char first
first [11]; char [11] ; char address
address [16]; char [16]; char city [16];
city [16]; char state char state [3]; char
[3];1 char zip [10]; zip [10] ;
}; };

Figure 4.2 Definition of record to hold person information.

In this example; each field is a character array that can hold a


string value of some maximum size. The size of the array is one
larger than the longest string it can hold. This is because strings in C
and C++ are. stored with a terminating 0 byte. The string “Mary”
requires five characters to store. The functions in string . h assume
that each string is stored this way. A fixed-size field in a file does
not need to add this extra character. Hence, an object of class Person
can be stored in 61 bytes: 10+10+15+15+2+9.
Using this kind of fixed-field length structure changes our
output so it looks like that shown in Fig. 4.3(a). Simple arithmetic is
sufficient to let us recover the data from the original fields.
One obvious disadvantage of this approach is that adding all
the padding required to bring the fields up to a fixed length makes
the file much larger. Rather than using 4 bytes to store the last name
“Ames,” we use 10. We can also encounter problems with data that
is too long to fit into the allocated amount of space. We could solve
this second problem by fixing all the fields at lengths that are-large
enough to cover all cases, but this would make the first problem of
wasted space in the file even worse.
Because of these difficulties, the fixed-field approach to
structuring data is often inappropriate for data that inherently
contains a large amount of variability in the length of fields, such as
names and addresses. But there are kinds of data for which
fixed-length fields are highly appropriate. If every field is already
fixed in length or if there is very little variation in field lengths,
using a file structure consisting of a continuous stream of bytes
organized into fixed-length fields is often a very good solution.
122 Chapter 4 Fundamental File Structure Concepts

Ames Mary .123 Maple Stillwater OK74075


Mason Alan 90 Eastgate Ada OK74820
(a)

04Ames04Mary09123 Maplel0Stillwater02OK0574075
05Mason04Alanll90 Eastgate03Ada020K0574820
—^

AmesIMary1123 Maple I Stillwater I OK I 74075 I


Mason|Alan I 9 0 EastgateI Ada|OK I 74820 j
(c)

last=AmesIfirst=MaryIaddress=123 Maple
Icity=StillwaterI
state=OKIzip=74075I
.-—

Figure 4.3 Four methods for organizing fields within records, (a) Each field is
of fixed length, (b) Each field begins with a length indicator, (c) Each field
ends with a delimiter |. (d) Each field is identified by a key word.

Method 2: Begin Each Field with a Length Indicator


Another way to make it possible to count to the end of a field is to store the field
length just ahead of the field, as illustrated in Fig. 4.3(b). If the fields are not
too long (less than 256 bytes), it is possible to store the length in a single byte at
the start of each field. We refer to these fields as length-based.

Method 3:Separate the Fields with Delimiters


We can also preserve the identity of fields by separating them with delimiters.
All we need to do is choose some special character or sequence of characters
that will not appear within a field and then insert that delimiter into the file after
writing each field. •
The choice of a delimiter character can be very important as it must be a
character that does not get in the way of processing. In many instances
white-space characters (blank, new line, tab) make excellent delimiters
because they provide a clean separation between fields when we list them
Field and Record Organization 123

on the console. Also, most programming languages include I/O statements


which, by default, assume that fields are separated by white space.

original stream-of-bytes program, w r i t s t r m . cpp, so that it places a delimiter


after each field. We use this delimited field format in the next few sample
programs.

Method 4: Use a "Keyword = Value"Expression to Identify Fields

many applications. It is easy to tell which fields are contained in a: file, even
if we don’t know ahead of time which fields the file is supposed to contain. It
is also a good format for dealing with missing fields. If a field is missing, this
format makes it obvious, because the keyword is simply not there.
You may havenoticed in Fig. 4.3(d) that this format is used in combi-
nation with another format, a delimiter to separate fields. While this may not
always be necessary, in this case it is helpful because it shows the division
between each value and the keyword for the following field.
Unfortunately, for the address file this format also wastes a lot of space:
50 percent or more of the file’s space could be taken up by the keywords. But
there are applications in which this format does not demand so much
overhead. We discuss some of these applications in Section 5.6: “Portability
and Standardization.”

4.1.3 Reading a Stream of Fields


Given a modified version of operator << that uses delimiters to separate
fields, we can write a function that overloads the extraction operator (operator
>>) that reads the stream of bytes back in, breaking the stream into fields and
storing it as a Person object. Figure 4.4 contains the implementation of the
extraction operation. Extensive use is made of the isCream method
getline. The arguments to get line are a character array to hold the
string, a maximum length, and a delimiter. Getline reads up to the first
occurrence of the delimiter, or the end-of-line,
12 4 Chapter 4 Fundamental File Structure Concepts

istream & operator » (istream & stream, Person & p) { // read


delimited fields from file char delim;
stream.getline(p.LastName, 30,'I'); • if
(strlen(p.LastName)==0) return stream;
stream.getline(p.FirstName,30, 'I 1);
stream.getline(p.Address,30,'I');
stream.getline(p.City, 3 0, 1 I ' ) ;
stream.getline(p.State,15,'|');
stream.getline(p.ZipCode,10,'|'); return stream;

Figure 4.4 Extraction operator for reading delimited fields into a Person
object.

whichever comes first. A full implementation of the program to read a stream


of delimited Person objects in C++, readdel. cpp, is included in Appendix D.
.
When this program is run using our delimited-field version of the file
containing data for Mary Ames and Alan Mason, the output looks like this:

Last Name 'Ames'


First Name 'Mary'
Address City '123 Maple’
State Zip 'Stillwater'
Code Last 'OK'
Name Fijrst '74075'
Name Address 'Mason'
City State 'Alan'
Zip Code '90 Eastgate
'Ada'
' OK ’
'74820'

Clearly, we now preserve the notion of. a field as we store and retrieve
this data. But something is still missing. We do not really think of this file as a
stream of fields. In fact, the fields are grouped into records. The first six fields
form a record associated with Mary Ames. The next six are a record associated
with Alan Mason.
Field and Record Organization 125

a record is another conceptual tool. It is another level of organization that we


impose on the data to preserve meaning.. Records do not necessarily exist in
the File in any physical sense, yet they are an important logical notion included
in the file's structure.

saving the state (or value) of an object that is stored m memory; Reading a
record from a file into a memory resident object restores the state of the
object. It is our goal in designing file structures to facilitate this transfer of
information between memory and files. We will use the term object to refer to
data residing in memory and the term record to refer to data residing in a file.
In C++ we use class declarations to describe objects that reside in
memory. The members, or attributes, of an object of a particular class
correspond to the. fields that need to be stored in a file record. The C++
programming examples are focused on adding methods to classes to support
using files to preserve the state of objects.
Following are some of the most often used methods for organizing the
records of a file:

■ Require that the records be a predictable number of bytes in length.


■ Require that the records be a predictable number of fields in length.
■ Begin each record with a length indicator consisting of a count of the
number of bytes that the record contains.
■ Use a second file to keep track of the beginning byte address for each
record.
■ Place a delimiter at the end of each record to separate it from the next
record.

Method 1: Make Records a Predictable Number of Bytes


(Fixed-Length Records)
A fixed-length record file is one in which each record contains the same
number of bytes. This method of recognizing records is analogous to the first
method we discussed formaking fields recognizable. As we will see in
126 Chapter 4 Fundamental File Structure Concepts

the chapters that follow, fixed-length record structures are among the most
commonly used methods for organizing files.
The C structure Person (or the C++ class of the same name) that we define
in our discussion of fixed-length fields is actually an example of a fixed-length
record as well as an example of fixed-length fields. We have a fixed number
of fields, each with a predetermined length, that combine to make a fixed-
length record. This kind of field and record structure is illustrated in Fig. 4.5(a).
It is important to realize, however, that fixing the number of bytes in a
record does not imply that the size or number of fields in the record must be
fixed. Fixed-length records are frequently used as containers to hold variable
numbers of variable-length fields. It is also possible to mix fixed- and variable-
length fields within a record. Figure 4.5(b) illustrates how variable-
length fields might be placed in a fixed-length record.

Method 2: Make Records a Predictable Number of Fields


Rather than specify that each record in a file contain some fixed number of
bytes, we can specify that it will contain a fixed number of fields. This is a
good way to organize the records in the name and address file we have been
looking at. The program in writstrm. cpp asks for six pieces of information for
every person, so there are six contiguous fields in the file for each record (Fig.
4.5c). We could modify r eaddel to recognize fields simply by counting the
fields modulo six, outputting record boundary information to the screen every
time the count starts over.

Method 3: Begin Each Record with a Length Indicator


We can communicate the length of records by beginning each record with a
field containing an integer that indicates how many bytes there are in the rest of
the record (Fig. 4.6a on page 128). This is a commonly used method for
handling variable-length records. We will look at it more closely in the next
section.

Method 4: Use an Index to Keep Track of Addresses


We can use an index to keep a byte offset for each record in the original file.
The byte offsets allow us to find the beginning of each successive record and
compute the length of each record. We look up the position of a record in the
index then seek to the record in the data file. Figure 4.6(b) illustrates this
two-file mechanism.
Field and Record Organization 127

Ames Mary 123 Maple Stillwater 0K74075

Mason Alan 90 Eastgate Ada OK74820

(a)

(b)

Figure 4.5 Three ways of making the lengths of records constant and predictable,
(a) Counting bytes:fixed-length records with fixed-length fields, (b) Counting
bytes: fixed-length records with variable-length fields, (c) Counting fields: six
fields per record. ■ •

Method 5: Place a Delimiter at the End of Each Record


This option, at a record level, is exactly analogous to the solution we used to
keep the fields distinct in the sample program we developed. As with fields,
the delimiter character must not get in the way of processing. Because we often
want to read files directly at our console, a common choice of a record
delimiter for files that contain readable text is the end- of-line character
(carriage return/new-line pair or, on Unix systems, just a new-line character:
\n). In Fig 4.6(c) we use a # character as the record delimiter.

4.1.5 A Record Structure That Uses a Length Indicator


None of these approaches to preserving the idea of a record in a file is
appropriate for all situations. Selection of a method for record organization
depends on the nature of the data and on what you need to do with it. We begin
by looking at a record structure that uses a record- length field at the.
beginning of the record. This approach lets us preserve the variability in the
length of records that is inherent in our initial stream file.
128 Chapter 4 Fundamental I'.L. ' tructure Concepts

Figure 4.6 Record structures for variable-length records, (a) Beginning each record with
a length indicator, (b) Using an index file to keep track of record addresses, (c) Placing
the delimiter # at the end of each record.

Writing the Variable-Length Records to the File


Implementing variable-length records is partially a matter of building on the
program in wri t s t r m . cpp that we created earlier in this chapter, but it also
involves addressing some new problems:

U In what form should we write the record-length field to the file? As a


binary integer? As a series of ASCII characters?
The concept of buffering is one we run into again and again as we work
with files. In this case, the buffer can simply be a character array into which we
place the fields and field delimiters as we collect them. A C++ function Wr
itePerson, written using the C string functions, is found in Figure 4.7. This
function creates a buffer; fills it with the delimited field values using strcat,
the string concatenation function; calculates the length of the of the buffer
using strlen; then writes the buffer length ' and the buffer to the output stream.
129
Field and Record Organization

const int MaxBufferSize = 200;


int WritePerson (ostream & stream, Person & p)
{ char buffer [MaxBufferSize]/•/ create buffer of fixed size strcpy.(buffer,
p.LastName); strcat(buffer,"|") ; strcat(buffer, p.FirstName);
strcat(buffer,"|"); strcat (buffer, p.Address-) ; strcat (buffer, " |
strcat (buffer,. p.City); strcat (buffer," I ; strcat(buffer, p.State);
strcat(buffer,"I");

Stream.write (&buffer, length); } ;

Figure 4.7 Function WritePerson writes a variable-length, delimited buffer to a file.

Representing the Record Length


The question of how to represent the record length is a little more difficult.
One option would be to write the length in the form of a 2-byte bina- ry
integer before each record. This is a natural solution in C, since it does not
require uTto go to the trouble of converting the record length into character
form. Furthermore, we can represent much bigger numbers with an integer
than we can with the same number of ASCII bytes (for example, 32 767
versus 99). It is also conceptually interesting, since it illustrates the use of a
fixed-length binary field in combination with variable- length character
fields.
Another option is to convert the length into a character string using
formatted output. With C streams, we use f p r i n t f ; with C++ stream
classes, we use the overloaded insertion operator (<<):
fprintf (file, "%d ", length);. // with C streams stream <<
length << ' '; // with C++ stream classes
Each of these lines inserts the length as a decimal string followed by a single
blank that functions as a delimiter.
In short, it is easy to store the integers in the file as fixed-length, 2-byte
fields containing integers. It is just as easy to make use of the automatic
conversion of integers into characters for text files. File structure design is
always an exercise in flexibility. Neither of these approaches is correct; good
design consists of choosing the approach that is most appropriate for a given
language and computing environment. In the functions included
130 Chapter 4 Fundamental File Structure Concepts

40 Ames I Mary1123 Maple I Stillwater I OK I 74075 I 36 Mason


I Alan I 90 Eastgate -I-Ada I OK I 74820

Figure 4.8 Records preceded by record-length fields in character form.

m program readvar. cpp in Appendix D, we have implemented our record


structure using binary field to hold the length. The output from an
implementation with a text length field is shown in Fig. 4.8. Each record now
has a record length field preceding the data fields. This field is delimited by a
blank. For example, the first record (for Mary Ames) contains 40

Since the implementation of variable-length records presented'in Section


4.2 uses binary integers for the record length, we cannot simply print it to a
console screen. We need a way to interpret the noncharacter portion of the
file. In the next section, we introduce the file dump, a valuable tool for
viewing the contents of files. But first, let’s look at how to read in any file
written with variable-length records.

Given our file structure of variable-length records preceded by record- length


fields, it is easy to write a program that reads through the file, record by
record, displaying the fields from each of the records on the

to read and break up the record is included in function ReadVariablePerson


in Fig. 4.9. The function is quite simple because it takes advantage of the
extraction operator that was previously defined for reading directly from a
file. The implementation of ReadVariablePerson may be hard to understand
because it uses features of C++ that we haven’t yet covered. In particular,
class is tr stream (input string, stream) is a type of input stream that uses the
same operators as other input streams but has its value stored in a character
string instead of in a file. The extraction operation of Figure 4.4 works just as
well on a string stream as it does on a file stream. This is a wonderful result of
the use of inheritance in C++ classes. We use inheritance extensively in' later
C++ classes, but that will have to wait for Section 4.3.
131
Field and Record Organization

int ReadVariablePerson (istream & stream, Person & p)


{ // read a variable' sized record from stream and store it in p short length;
. stream . read (klength, sizeof(length)); char * buffer = new char
[length+1] ;// create buffer space stream . read (buffer, length);.
buffer [length] = 0 - ; // terminate buffer with null istrstream
strbuff (buffer); 7/ create a string stream . strbuff » p; // use
the istream extraction operator return 1;

Figure 4.9 Function ReadVariablePerson that reads a variable-sized Person record. .

4.1.2 Mixing Numbers and Characters: Use of a File Dump


File dumps give us the ability7 to look inside a file at the actual bytes that are
stored there. Consider, for instance, the record-length information in the text
file that we were examining a moment ago. The length of the Ames record, the
first one in the file, is 40 characters, including delimiters. The actual bytes
stored in thefilelook like the representation in Fig. 4.10(a). In the mixed binary
and text implementation, where we choose to represent the length field as a
2-byte integer, the bytes look like the representation in Fig. 4.10(b).
As you can see, the number 40 is not the same as the set of characters 4
and 0. The 1-byte hex value of the binary integer 40 is 0x28; the hex values of
the characters 4 and 0 are 0x34 and 0x30. (We are using the C language
convention of identifying hexadecimal numbers through the use of the prefix
Ox.) So, when we are storing a number in ASCII form, it is the hex values of
the ASCII characters that go into the file, not the hex value of the number
itself.
Figure 4.10(b) shows the byte representation of the number 40 stored as an
integer (this is called storing the number in binary form, even though we
usually view the output as a hexadecimal number). Now the hexadecimal
value stored in the file is that of the number itself. The ASCII characters that
happen to be associated with the number’s hexadecimal value have no obvious
relationship to the number. Here is what the version of the. file that uses binary
integers for record lengths looks like if we simply print it on a terminal screen:
132 Chapter 4 Fundamental File Structure Concepts

Figure 4.10 The number 40, stored as ASCII characters and as a short integer.
pretations of the bytes in the file in ASCII and hexadecimal. These repre-
sentations were requested on the command line with the -xc flag (x = hex; c =
character).
Let’s look at the first row of ASCII values. As you would expect, the
data placed in the file in ASCII form appears in this row in a readable way.
But there are hexadecimal values for which there is no printable ASCII
representation. The only such value appearing in this file is 0 x 00. But there
could be many others. For example, the hexadecimal value of the number
500 000 000 is 0xlDCD6500. If you write this value out to a file, an od of the
file with the option-xc looks like this:
0000000 \03 5\315 e \0 .,
- Idcd 6500
The only printable byte in this file is the one with the value 0x65 (e). Od
handles all of the others by listing their equivalent octal values in the ASCII
representation.
The hex dump of this output from writrec. shows how this file structure
represents an interesting mix Qf a number of the organizational tools we have
encountered. In a single record we have both binary and ASCII data. Each
record consists of a fixed-length field (the byte count) and several delimited,
variable-length fields. This kind of mixing of different data types and
organizational methods is common in real-world file structures.

A Note about Byte Order


If your computer is a PC or a computer from DEC, such as a VAX, your octal
dump for this file will probably be different from the one we see here These
machines store the individual bytes of numeric values in a reverse order. For
example, if this dump were executed on a PC, using the MS- DOS debug
command,, the hex representation of the first 2-byte value in the file would
be 0x2800 rather than 0x0028.
This reverse order also applies to long, 4-byte integers on these ma-
chines. This is an aspect of files that you need to be aware of if you expect to
make sense out of dumps like this one. A more serious consequence of the
byte-order differences among machines occurs when we move files from a
machine with one type of byte ordering to one with a different byte ordering.
We discuss this problem and ways to deal with it in Section 5.6, "Portability
and Standardization.”
4.2 Using Classes to Manipulate Buffers

Now that we understand how to use buffers to read and write information, we
can use C++ classes to encapsulate the pack, unpack, read, and write
operations of buffer objects. An object of one of these buffer classes can be
used for output as follows: start with an empty buffer object, pack field values
into the object one by one, then write the buffer contents to an output stream.
For input, initialize a buffer object by reading a record from an input stream,
then extract the object’s field values, one by one. Buffer objects support only
this behavior. A buffer is not intended to allow' modification of packed values
nor to allow pack and unpack operations to be mixed. As the classes are
described, you will see that no direct access is allowed to the data members
that hold the contents of the buffer. A considerable amount of extra error
checking has been included in these classes.
There are three classes defined in this section: one for delimited fields,
one for length-based fields, and one for fixed-length fields. The first.two field
types use variable-length records for input and output. The fixed- length
fields are stored in fixed-length records, j

4.2.1 Buffer Class for Delimited Text Fields


The first buffer class, DelimTextBuffer, supports variable-length buffers
whose fields are represented as delimited text. A part of the class definition is
given as Fig. 4.11. The full definition is in file del text. h in Appendix E. The
full implementations of the class methods are in deltext. cpp. Operations on
buffers include constructors, read and write, and field pack and unpack. Data
members are used to store the"-- delimiter used in pack and unpack operations,
the actual and maximum number of bytes in the buffer, and the byte (or-
character) array that contains the value of the buffer. We have also included an
extension of the class Person from Fig. 4.2 to illustrate the use of buffer
objects.
The following code segment declares objects of class Person and class
DelimTextBuf fer, packs the person into the buffer, and writes the buffer to
a file:
Person MaryAmes;
DelimTextBuffer buffer;
buffer . Pack (MaryAmes . LastName);
buffer . Pack (MaryAmes . FirstName) ;•
Using Classes to Manipulate Buffers
135

class DelimTextBuffer { public:


'•
DelimTextBuffer (char Delim = ' [ ' , int maxBytes = 1000); int Read
(istream & file); int Write (ostream & file) const; int Pack (const char *
str, int size = -1); int Unpack (char * str); private;
* char Delim; // delimiter character'
char * Buffer; // character array to hold field values . int
BufferSize; // current size of packed fields int MaxBytes; // maximum number
of characters -in the
buffer
int NextByte; // packing/unpacking position in buffer
};

Figure 4.11 Main methods and members of class DelimTextBuf fer.

buffer . .Pack (MaryAmes . ZipCode); buffer


. Write (stream);
This code illustrates how default values are used in C++. The declaration of
object buffer has no arguments, but the only constructor for DelimTextBuf fer
has two parameters. There is no error here, since the constructor declaration
has default values for both parameters. A call that omits the arguments has the
defaults substituted. The following two declarations are completely
equivalent:
DelimTextBuffer buffer; // default arguments used
DelimTextBuffer buffer .( 1 |’, 1000); // arguments given explicitly
Similarly, the calls on the Pack method have only a single argument, so the
second argument.(size) takes on the default value -1.
The Pack method copies the characters of its argument str to the buffer
and then adds the delimiter character. If the size argument is not -1, it specifies
the number of characters to be written. If size is -1, the C function str len is
used to determine the number of characters to write. The Unpack function
does not need a size, since the field that is being unpacked consists of all of the
characters up to the next instance of the delimiter. The implementation of Pack
and Unpack utilize the private member NextByte to keep track of the current
position in the buffer. The Unpack method is implemented as follows:
136 Chapter 4 Fundamental File Structure Concepts

int DelimTextBuffer :: Unpack (char -* str).


// extract the value of the next field of the buffer
{
int len = -1; // length of packed string int start = NextByte; //
first character to be unpacked for (int i = start; i < BufferSize;
i++) if (Buffer[i] == Delim)
{len = i - start; break;}
if (len == -1) return FALSE; // delimeter not found NextByte
+= len +1;
if (NextByte > BufferS.ize) return FALSE; strncpy (str-,
^Buffer [start] , len); ■ str [len] = 0; // zero termination
for string return TRUE;
}
The Read and Write methods use the variable-length strategy as described
in Section 4.1.6. A binary value is used to represent the length of the record.
Write inserts the current buffer size, then the characters of the buffer. Read
clears the current buffer contents, extracts the record size, reads the proper
number of bytes into the buffer, and sets the buffer size:
int DelimTextBuffer :: Read (istream &' stream)
{
Clear (); '
stream . read ((char *)&BufferSize, sizeof(BufferSize));
if (stream.fail()) return FALSE;
if (BufferSize > MaxBytes) return FALSE; // buffer overflow stream . read
(Buffer, BufferSize); return stream . good ();
}

4.2.2 Extending Class Person with Buffer Operations


The buffer classes have the capability of packing any number and type of
values, but they do not record how these values are combined to make objects.
In order to pack and unpack a buffer for a Person object, for instance, we have
to specify the order in which the members of Person are packed and
unpacked. Section 4.1 and the code in Appendix D included operations for
packing and unpacking the members of Person objects in insertion («) and
extraction (») operators. In this, section and Appendix E, we add those
operations as methods of class Person. The
Using Classes to Manipulate Buffers 137

definition of the class has the following method for packing delimited text
buffers. The unpack operation is equally simple:
int Person::Pack (DelimTextBuffer & Buffer) const {// pack
the fields into a DelimTextBuffer int result;
result = Buffer . Pack (LastName); result =
result && Buffer . Pack (FirstName);
result = result && Buffer . Pack (Address);
result = result && Buffer . Pack (City);
'result = result && Buffer-. Pack (State);
result = result && Buffer . Pack (ZipCode);
return result;
}.

4.2.3 Buffer Classes for Length-Based and


Fixed-length Fields
Representing records of length-based fields and records of fixed-length
fields requires a change in the implementations of the Pack and Unpack
methods of the delimited field class, but the class definitions are almost
exactly the same. The main members and methods of class
LengthTextBuffer are given in Fig. 4.12. The full class definition and
method implementation are given in lent ext. h and lent ext. cpp

class LengthTextBuffer '


{ pub.lic
LengthTextBuffer (int maxBvtes = 1000); int Read
(istream & file); int Write (ostream & file) const; int Pack
(const char *-field, int size = -l); int Unpack (char *
field); private:
char * Buffer; // character array to hold field values int
BufferSize; I t size of packed fields
int MaxBytes; // maximum number of characters in the buffer int NextByte;
// packing/unpacking position in buffer .

Figure 4.12 Main methodsand members of'class LengthTextBuf fer.


138 Chapter 4 Fundamental File Structure Concepts

in Appendix E. The only changes that are apparent from this figure are the
name of the class and the elimination of the delim parameter on the
constructor. The code for the Pack and Unpack methods is substantially
different, but the Read and Write methods are exactly the same.
Class FixedTextBuffer, whose main members and methods are in Fig.
4.13 (full class in fixt ext .hand f ixtext. cpp),is different in two ways from the
other two classes. First, it uses a fixed collection of fixed-length fields. Every
buffer value has the same collection of fields, and the Pack method needs no
size parameter. The second difference is that it uses fixed-length records.
Hence, the Read and Write methods do not use a length indicator for buffer
size. They simply use the fixed size of the buffer to determine how many
bytes to read or write.
The method AddField is included to support the specification of the fields
and their sizes. A buffer for objects of class Person is initialized by the new
method Ini tBuffer of class Person:
int Person::InitBuffer (FixedTextBuffer & Buffer)
// initialize a FixedTextBuffer to be used for Person objects ------------
{.
Buffer . Init (6, 61);//6 fields, 61 bytes total
Buffer . AddField (10); // LastName [11] ;
Buffer . AddField (10); // FirstName [11];
Buffer . AddField (15); // Address [16];

class FixedTextBuffer {
public:
FixedTextBuffer (int maxBytes = 1000); int AddField (int
fieldSize); int Read (istream & file); int Write (ostream &
file) const; int Pack (const char * field); int Unpack (char
* field); private:
char * Buffer; // character array to hold field values int BufferSize;
// size of packed fields
int MaxBytes; // maximum number of characters in the buffer int NextByte;
// packing/unpacking position in buffer int * FieldSizes; // array of
field sizes

Figure 4.13 Main methods and members of class FixedTextBuf fer.


Using Inheritance for Record Buffer Clashes 139

Buffer . AddField (15); // City [16]; Buffer .


AddField (2); // State [3]; Buffer . AddField
(9); // ZipCode [10]; return 1;

4.3 Using Inheritance for Record Buffer Classes

A reading of the cpp files for the three classes above shows a striking simi-
larity: a large percentage of the code is duplicated. In this section, we elim-
inate almost all of the duplication through the use of the inheritance
capabilities of C++.

4.3.1 Inheritance in the C++Stream Classes


C++ incorporates inheritance to allow multiple classes to share members and
methods. One or more base classes define members and methods, which are
then used by subclasses. The stream classes are defined in such a hierarchy.
So far, our discussion has focused on class f stream, as though it stands alone.
In fact, f stream is embedded in a class hierarchy that contains many other
classes. The read operations, including the extraction operators are defined in
class is tream. The write operations are defined in class ostream. Class fstream
inherits these operations from its parent class ios tream, which in turn inherits
from is tream and ostream. The following definitions are included in ios tream.
h and fstream.h:
class istream: virtual public ios {. . .
class ostream: virtual public ios {. . .
class iostream: public istream, public ostream { . . .
class ifstream: public fstreambase, public istream { . . .
class ofstream: public fstreambase, public ostream {. . . .
class fstream: public fstreambase, public iostream { . . .
We can see that this is a complex collection of classes. There are two base
classes, ios and fstreambase, that provide common declarations and basic
stream operations (ios) and access to operating system file operations
(fstreambase). There are uses of multiple inheritance in these classes; that is,
classes have more than one base class. The keyword
140 Chapter 4 Fundamental File Structure Concepts

virtual is used to ensure that class i'os is included only once in the ancestry of
any of these classes.
Objects of a class are also objects of their base classes, and generally,
include members and methods of the' base classes.. An object of class
fstream, for example, is also an object of classes fstreambase, iostream,
istream, ostream, and ios and includes all of the members and methods of
those base classes. Hence, the read method and extraction (>>) operations
defined in istream are also available in iostream, if stream, and fstream. The
openand close operations of class f streambase are also members of class
fstream.
An important benefit of inheritance is that operations that work on base
class objects also work on derived class objects. We had an example of this
benefit in the function ReadVariablePerson in Section 4.1.5 that used an
istrstream object strbuf f to contain a string buffer. The code of that function
passed s trbuf f as an argument to the person extraction function that expected
an istream argument. Since istrstream is derived from istream, strbuff is an
istream object and hence can be manipulated by this istream operation.

4.3.2 A Class Hierarchy for Record Buffer Objects


The characteristics of the three buffer classes of Section 4.2 can be combined
into a single class hierarchy, as shown in Fig. 4.14. Appendix F has the full
implementation of these classes. The'members and methods that are common
to all of the three buffer classes are included in the base class IOBuffer.
Other methods are in classes VariableLengthBuffer and
FixedLengthBuffer, which support the read and write operations for
different types of records. Finally the classes LengthFieldBuffer,
DelimFieldBuf f er, and FixedFieldBuf f er have the pack
and unpack methods for the specify ic field representations.
The main members and methods of class IOBuffer are given in Fig.
4.15. The full class definition is in file iobuf f er. h, and the implementation
of the methods is in file iobuf fer . cpp. The common members of all
of the buffer classes, BufferSize, MaxBytes, NextByfe., and
Buffer, are declared in class IOBuffer. These members are in the '
protected Section of IOBuffer.
This is our first use of protected access, which falls between private (no
access outside the class) and public (no access restrictions). Protected
members of a class can be used by methods of the class and by methods of
Using Inheritance for Record Buffer Classes 141

Figure 4.14 Buffer class hierarchy

classes derived from the class. The protected members of IOBuffer can be
used by methods in all of the classes in this hierarchy. Protected members of
VariableLengthBuf f er can be used in its subclasses but not in classes
IOBuffer and FixedLengthBuf f er.
The constructor for class IOBuffer has a single parameter that specifies,
the maximum size of the buffer. Methods are declared for reading, writing,
packing, and unpacking. Since the implementation of these methods depends
on the exact nature of the record and its fields, IOBuffer must leave its
implementation to the subclasses.
Class IOBuffer defines these methods as virtual to allow each subclass
to define its own implementation. The = 0 declares a pure virtual

class IOBuffer {public-:


IOBuffer (int maxBytes = 1000); //a maximum of maxBytes virtual int Read
(istream &) = 0; // read a buffer virtual int Write (ostream &) const =0;
// write a buffer virtual int Pack (const void * field, int size = -1) =
0; virtual int Unpack (void * field, int maxbytes = -1). = 0; protected:
char * Buffer; // character array to hold field values
int BufferSize; // sum of the sizes of packed fields
int MaxBytes; // maximum number of characters in the buffer

Figure 4.15 Main members and methods of class IOBuffer.


142 Chapter 4 Fundamental File Structure Concepts

method. This means that the class IOBuf f er does not include an imple-
mentation of the method. A class with pure virtual methods is an abstract
class. No objects of such a class can be created, but pointers and references to
objects of this class can be declared.
The full implementation of read, write, pack, and unpack operations for
delimited text records is supported by two more classes. The reading and
writing of variable-length records are included in the class VariableLengthBuf
f er, as given in Figure 4.16 and files varlen. h and varlen.cpp. Packing and
unpacking delimited fields is in class . DelimitedPieldBuf f er and in files
delim.h and delim. cpp. The code to implement these operations follows the
same structure as in Section 4.2 but incorporates additional error checking.
The Write method of VariableLengthBuf fer is implemented as follows:
int VariableLengthBuf fer Write (ostream & stream) const // read the
length and buffer from the stream { '
int recaddr = stream , tellp (); unsigned short bufferSize =
BufferSize;
stream . write ((char *)&bufferSize, sizeof(bufferSize));
if ( Istream) return —1;
stream . write (Buffer, BufferSize);
if (1 stream.good ()) return -1;
return recaddr;

The method is implemented to test for all possible errors and to return
information to the calling routine via the return value. We test for failure in
the write operations using the expressions ! stream and I stream. good (),
which are equivalent. These are two different ways to test if the stream has
experienced an error. The Write method returns the address in the stream
where the record was written. The address is determined by calling stream,
tellg () at the beginning of the function. Tellg is a method of ostream that
returns the current location of the put pointer of the stream. If either of the
write operations fails, the value-1 is returned.
An effective strategy for making objects persistent must make it easy for
an application to move objects from memory to files and back correctly. One
of the crucial aspects is ensuring that the fields are packed and unpacked in
the same order. The class Person has been extended to include pack and
unpack operations. The main purpose of these operations is to specify an
ordering on the fields and to encapsulate error testing. The unpack operation
is:
Using Inheritance for Record Buffer Classes 143

class VariableLengthBuffer: public IOBuffer { public:


VariableLengthBuffer (int MaxBytes = 1000);
int'Read (istream &) ;
int Write (ostreaiu &) const;
int SizeOfBuffer () const; // return current size of buffer

};

class DelimFieldBuf fer: public VariableLengthBuffer { public:


DelimFieldBuffer (char Delim = -1, int maxBytes = 1000; int Pack (const
void*, int size = -1); int Unpack (void * field, int maxBytes = -1);
protected; char Delim;
};

Figure 4.16 Classes VariableLengthBuf fer and DelimFieldBuf fer .

iht Person::Unpack (IOBuffer & Buffer)


{
Clear (); int
numBytes
■ numBytes.= Buffer-. Unpack (LastName); if
(numBytes == -1) return FALSE;
LastName[numBytes] = 0; numBytes = Buffer .
Unpack (FirstName); if (numBytes == -1) return
FALSE;
. . . // -unpack the other fields
return.TRUE;,
}
This method illustrates the power of virtual functions. The parameter
of Person: :Unpack is an object of type IOBuffer, but a call to Unpack
supplies an argument that can be an object of any subclass of IOBuffer.
The calls to Buffer.Unpack in the method Person: : Unpack are virtual
function calls. In calls of this type, the determination of exactly which
Unpack method to call is not made during compilation as it is with
nonvirtual calls. Instead, the actual type of the object Buffer is. used to
determine which function to call. In the following example of calling
Unpack, the calls to B u f f e r . Unpack use the method DelimFieldBuf
f e r : : Unpack.
144 Chapter 4 Fundamental File Structure Concepts

Person MaryAmes;
DelimFieldBuffer Buffer;
MaryAmes . Unpack (Buffer);
The full implementation of the I/O buffer classes includes class
LengthFieldBuffer, which supports field packing with length plus ■ value
representation. This class is like DelimFieldBuf fer in that it is implemented by
specifying only the pack and unpack methods. The read and write operations are
supported by its base class, VariableLengthBuffer.

4.4 Managing Fixed-Length, Fixed-Field Buffers

Class FixedLengthBuf f er . is the subclass of IOBuffer that supports read and


write of fixed-length records. For this class, each record is of the same size.
Instead of storing the record size explicitly in the file along with the record, the
write method just writes the fixed-size record. The read method must know the
size in order to read the record correctly. Each FixedLengthBuf fer object has
aprotected field that records the record size.
Class FixedFieldBuf fer, as shown in Fig. 4.17 and files f ixf Id. h and f ixf
Id. cpp, supports a fixed set of fixed-length fields. One difficulty with this
strategy is that the unpack method.has to know the length of all of the fields. To
make it convenient to keep track of the

class FixedFieldBuffer: public FixedLengthBuffer public:


FixedFieldBuffer (int maxFields, int RecordSize = 1000); FixedFieldBuffer
(int maxFields, int’ * fieldSize); int AddField (int fieldSize); // define the
next field int Pack (const void * field,. int size = -1);. int Unpack (void *
field, int maxBytes = -1); int NumberOfFields () const; // return number of
defined fields protected:
int * FieldSize; // array to hold field sizes
int MaxFields; // maximum number of fields
int NumFields; // actual number of defined fields

Figure 4.17 Class FixedFieldBuffer.


Managing Fixed-Length, Fixed-Field Buffers 145

field lengths, class FixedFieldBuffer keeps track of the field sizes. The
protected member FieldSize holds the field sizes in an integer array. The
AddField method is used to specify field sizes. In the case of using a
FixedFieldBuf fer'to hold objects of class Person, the InitBuf f er method can
be used to fully initialize the buffer:
int Person:,: Ini tBuf fer (FixedFieldBuffer & Buffer)
// initialize a FixedFieldBuffer to be used for Persons { ■ int result;
result = Buffer . AddField (10); // LastName [11]; result =
result && Buffer . AddField(10); //. FirstName [11];
result = result && Buffer . AddField (15); //Address [16];
result = result && Buffer . AddField (15); // City [16] ;
result = result && Buffer .AddField (2); // State [3];
result = result && Buffer . AddField (9); // ZipCode [10]; return result;
) '/ '
Starting with a buffer with no fields, Ini tBuf fer adds the fields one at a time,
each with its own size. The following code prepares a buffer for use in
reading and writing objects of class Person:
FixedFieldBuffer Buffer(-6, 61) ; // 6 fields, 61 bytes total
HaryAmes.InitBuffer (Buffer);

Unpacking FixedFieldBuf fer objects has to be done carefully. The


object has to include information about the state of the unpacking. The
member NextByte records the next character of the buffer to be unpacked,
just asm all of the IOBuf f er classes. FixedFieldBuffer has additional
member NextField to record the next field to be unpacked. The method
FixedFieldBuffer: :Unpack is implemented as follows:
int FixedFieldBuffer :: Unpack (void * field, int maxBytes)
{
if (NextField == NumFields || Packing)
// buffer is full or not in unpacking mode return
-1;
int start = NextByte; // first byte to be unpacked int packSize =
FieldSize[NextField]; // bytes to be unpacked memcpy (field,
&Buffer[start]; packSize); //move the bytes NextByte += packSize; //
advance NextByte to following char NextField.'++; // advance NextField
if (NextField == NumFields) Clear (); // all fields unpacked
return packSize;
146 Chapter 4 Fundamental File Structure Concepts

4.4 An Object-Oriented Class for Record Files

Now that we know how to transfer objects to and from files, it is appropriate
to encapsulate that knowledge in a class that supports all of our file
operations. Class BufferFile (in files buffile.h and buf f ile . cpp of Appendix
F) supports manipulation of files that are tied to specific buffer types. An
object of class Buf ferFile is created from a specific buffer object and can be
used to open and create files and to read and write records. Figure 4.18 has the
main data methods and members of Buf ferFile.
Once a BufferFile object has been created and attached to an operating
system file, each read or write is performed using the same buffer. Hence,
each record is guaranteed to be of the same basic type. The following code
sample shows how a file can be created and used with a DelimFieldBuffer:
DelimFieldBuffer buffer;
BufferFile file (buffer); file .
Open (myfile); file . Read {) ;
buffer . Unpack (myobject);

class BufferFile
{public:
BufferFile (IOBuffer &); // create with a buffer int Open (char * filename,
int MODE); // open an existing file int Create (char * filename, int MODE);
// create a new file int Close () ;
int Rewind (); // reset to the first data record // Input
and Output operations
int Read (int recaddr = -1); ■ • ■
int Write (int recaddr = -1);
int Append (); // write the current buffer at the end of file protected: •
IOBuffer & Buffer; // reference to the file’s buffer fstream
File; // the C++ stream of the file

Figure 4.18 Main data members and methods of class BufferFile.


A buffer is created, and the Buf f erFile object file is attached to it Then
Open and Read methods are called for file. After the Read, buffer contains
the packed record, and buffer.Unpack puts the record into myobj ect.
When Buf f erFile is combined with a fixed-length buffer, the result is
a file that is guaranteed to have every record the same size. The full
implementation of BufferF'ile, which is described in Section 5.2, “More
about Record Structures,” puts a header record on the beginning of each file.
For fixed-length record files, the header includes the record size. Buf f
erFile : : Open reads the record sizefrom the file header and compares it
with the record size of the corresponding buffer. If the two are not the same,
the Open fails and the file cannot be used.
This illustrates another important aspect of object-oriented design.
Classes can be used to guarantee that operations on objects are performed
correctly. Ifs easy to see that using the wrong buffer to read a file record is
disastrous to an application. It is the encapsulation of classes like Buf f erFile
that add safety to our file operations.
CHAPTER 5

Managing F i l e s of
Records

5.1 Record Access

5.1.1 Record Keys


Since our new file structure so clearly focuses on a record as the quantity of
information that is being read or written, it makes sense to think in terms of
retrieving just one specific record rather than reading all the way through the
file, displaying everything. When looking for an individual record, it is
convenient to identify the record with a key based on the record’s contents. For
example, in our name and address file we might want to access the “Ames
record” or the “Mason record” rather than thinking in terms of the “first
record” or “second record.” (Can you remember
Record Access 155

which record comes first?) This notion of a key is another fundamental


conceptual tool. We need to develop a more exact idea of what a key is.
When we are looking for a record containing the last name Ames, we want
to recognize it even if the user enters the key in the form “AMES,” “ames,” or
“Ames.” To do this, we must define a standard form for keys, along with
associated rules and procedures for converting keys into this standard form. A
standard form of this kind is often called a canonical form for the key. One
meaning of the word canon is rule, and the word canonical means conforming
to the rule. A canonical form for a search key is the single representation for
that key that conforms to the rule.
As a simple example, we could state that the canonical form for a key
requires that the key consist solely of uppercase letters and have no extra
blanks at the end. So, if someone enters “Ames,” we would convert the key to
the canonical form “AMES” before searching for it.
It is often desirable to have distinct keys, or keys that uniquely identify a
single record. If there is not a one-to-one relationship between the key and a
single record, the program has to provide additional mechanisms to allow the
user to resolve the confusion that can result when more than one record fits a
particular key. Suppose, for example, that we are looking for Mary Ames’s
address. If there are several records in the file for several different people
named Mary Ames, how should the program respond? Certainly it should not
just give the address of the first Mar)' Ames it finds. Should it give all the
addresses at once? Should it provide a way of scrolling through the records?
The simplest solution is to prevent such confusion. The prevention takes
place as new records are added to the file. When the user enters a new record,
we form a unique canonical key for that record and then search the file for that
key. This concern about uniqueness applies only to primary keys. A primary
key is, by definition, the key that is used to identify a record uniquely.
It is also possible, as we see later, to search on secondary keys. An
example of a secondary key might be the city field in our name and address file.
If we wanted to find all the records in the file for people who live in towns
named Stillwater, we would use some canonical form of “Stillwater” as a
secondary key. Typically, secondary keys do not uniquely identify a record.
Although a person’s name might at first seem to be a good choice for a
primary key, a person’s name runs a high risk of failing the test for uniqueness.
A name is a perfectly fine secondary key and in fact is often an important
secondary key in a retrieval system, but there is too great a likelihood that two
names in the same file will be identical.
156 Chapter 5 Managing Files of Records

The reason a name is a risky choice for a primary key is that it contains a
real data value. In general, primary keys should be dataless. Even when we
think we are choosing a unique key, if it contains data, there is a danger that
unforeseen identical values could occur. Sweet (1985) cites an example of a
file system that used a person’s social security number as a primary key for
personnel records. It turned out that, in the particular population that was
represented in the file, there was a large number of people who were not United
States citizens, and in a different part of the organization, all of these people
had been assigned the social security number 999-99-9999!
Another reason, other than uniqueness, that a primary key should be
dataless is that a primary key should be unchanging. If information that
corresponds to a certain record changes and that information is contained in a
primary key, what do you do about the primary key? You probably cannot
change the primary key, in most cases, because there are likely to be reports,
memos, indexes, or other sources of information that refer to the record by its
primary key. As soon as you change the key, those references become useless.
A good rule of thumb is to avoid putting data into primary keys. If we
want to access records according to data content, we should assign this content
to secondary keys. We give a more detailed look at record access by primary
and secondary keys in Chapter 6. For the rest of this chapter, we suspend our
concern about whether a key is primary or secondary and concentrate on
finding things by key.

5.1.2 A Sequential Search


Now that you know about keys, you should be able to write a program that
reads through the file, record by record, looking for a record with a particular
key. Such sequential searchingis just a simple extension of our read- var
program—adding a comparison operation to the main loop to see if the key for
the record matches the key we . are seeking. We leave the program as an
exercise.

Evaluating Performance of Sequential Search


In the chapters that follow, we find ways to search for records that are faster
than the sequential search mechanism. We can use sequential searching as a
kind of baseline against which to measure the improvements we make. It is
important, therefore, to find some way of expressing the amount of time and
work expended in a sequential search.
■ Developing a performance measure requires that we decide on a unit of
work that usefully represents the constraints on the performance of the whole
process. When we describe the performance of searches that take place in
electronic memory, where comparison operations are more expensive than
fetch operations to bring data in from memory, we usually use the number of
comparisons required for the search as the measure of work. But, given that
the cost of a comparison in memory is so small compared with the cost of a
disk access, comparisons do not fairly represent. the performance constraints
for a search through a file on secondary storage. Instead, we count low-level
Read calls. We assume that each Read call requires a seek and that any one
Read call is as costly as any other. We know from the discussions of matters,
such as system buffering in Chapter 3, that these assumptions are not strictly
accurate. But in a multiuser environment where many processes are using the
disk at once, they are close enough to correct to be useful.
Suppose we have a file with a thousand records, and we want to use a.
sequential search to find A1 Smith’s record. How many Read calls are
required? If A1 Smith’s record is the first one in the file, the program has to
read in only a single record. If it is the last record in the file, the program makes
a thousand Read calls before concluding the search. For an average search, 500
calls are needed. •
If we double the number of records in a file, we also double both the
average and the maximum number of Read calls required. Using a sequential
search to find A1 Smith’s record in a file of two thousand records requires, on
the average, a thousand calls. In other words, the amount of work required for
a sequential search is directly proportional to the number of records in the file.
In general, the work required to search sequentially for a record in a file
with n records is proportional to n: it takes, at most n comparisons; on average
it takes approximately nil comparisons. A sequential search is said to be of the
order O(n) because the time it takes is proportional to rO

Improving Sequential Search Performance with Record Blocking

It is interesting and useful to apply some of the information from Chapter 3


about disk performance to the problem of improving sequential search
performance. We learned in Chapter 3 that the major cost associated with a
disk access is the time required to perform a seek to the right location on 1

1. If you are not familiar with this “big-oh” notation, you should look it up. Knuth (1997) is a good source.
Chapter 5 Managing Files of Records

the disk. Once data transfer begins, it is relatively fast, although still much
slower than a data transfer within memory. Consequently, the cost of seeking
and reading a record, then seeking and reading another record, is greater than
the cost of seeking just once then reading two successive records. (Once again,
we are assuming a multiuser environment in which a seek is required for each
separate Read call.) It follows that we should be able to improve the
performance of sequential searching by reading in a block of several records all
at once and then processing that block of records in memory.
We began the previous chapter with a stream of bytes. We grouped the .
bytes into fields, then grouped the fields into records. Now we are considering a
yet higher level of organization—-grouping records into blocks. This new level
of grouping, however, differs from the others. Whereas fields and records are
ways of maintaining the logical organization within' the file, blocking is done
strictly as a performance measure. As such, the block size is usually related
more to the physical properties of the disk drive than to the content of the data.
For instance, on sector-oriented disks, the block size is almost always some
multiple of the sector size.
Suppose that we have a file of four thousand records and that the average
length of a record is 512 bytes. If our operating system uses sector- . sized
buffers of 512 bytes, then an unblocked sequential search requires, on the
average, 2,000 Read calls before it can retrieve a particular record. By blocking
the records in groups of sixteen per block so each Read call brings in 8
kilobytes worth of records, the number of reads required for an average search
comes down to 125. Each Read requires slightly more time, since more data is
transferred from the disk, but this is a cost that is usually well worth paying for
such a large reduction in the number of reads.
There are several things to note from this analysis and discussion of record
blocking:
■ Although blocking can result in substantial performance improvements, it
does not change the order of the sequential search operation. The cost of
searching is still O(n), increasing in direct proportion to increases in the
size of the file.
■ Blocking clearly reflects the differences between memory access speed
and the cost of accessing secondary storage.
■ Blocking does not change the number of comparisons that must be done in
memory, and it probably increases the amount of data transferred between
disk and memory. (We always read a whole block, even if the record we
are seeking is the first one in the block.)
■ Blocking saves time because it decreases the amount of seeking. We find,
again and again, that this differential between the cost of seeking and the
cost of other operations, such as data transfer or memory access, is the
force that drives file structure design.

When Sequential Searching is Good


Much of the remainder of this text is devoted to identifying better ways to
access individual records; sequential searching is just too expensive for most
serious retrieval situations. This is unfortunate because sequential access has
two major practical advantages over other types of access: it is extremely easy
to program, and it requires the simplest of file structures.
Whether sequential searching is advisable depends largely on how the file
is to be used, how fast the computer system is that is performing the search, and
how the file is structured. There are many situations in which a sequential
search is reasonable. Here are some examples:
■ ASCII files in which you are searching for some pattern (see grep in the
next section);
■ Files with few records (for example, ten records);
■ Files that hardly ever need to be searched (for example, tape files usually
used for other kinds of processing); and
■ Files in which you want all records with a certain secondary key value,
where a large number of matches is expected.
Fortunately, these sorts of applications do occur often in day-to-day
computing—so often, in fact, that operating systems provide many utilities for
performing sequential processing. Unix is one of the best examples of this, as
we see in the next section.

5.1.2 Unix Tools for Sequential Processing


Recognizing the importance of having a standard file structure that is simple
and easy to program, the most common file structure that occurs in Unix is an
ASCII file with the new-line character as the record delimiter and, when
passible, white space as the field delimiter. Practically all files that we create
with Unix editors use this structure. And since most of the built-in C and C++
functions that perform I/O write to this kind of file, it is common to see data
files that consist of fields of numbers or words separated by blanks or tabs and
records separated by new-line characters. Such files are simple and easy to
process. We can, for instance, generate an ASCII file with a simple program
and then use an editor to browse through it or alter it.
Chapter 5 Managing Files of Records

Unix provides a rich array of tools for working with files in this form.
Since this kind of file structure is inherently sequential (records are variable in
length, so we have to pass from record to'record to find any particular field or
record), many of these tools process files sequentially.
Suppose, for instance, that we choose the white-space/new-line structure
for our address file, ending every field with a tab and ending every record with
a new line. While this causes some problems in distinguishing fields (a blank is
white space, but it doesn’t separate a field) and in that sense is not an ideal
structure, it buys us something very valuable: the full use of those Unix tools
that are built around the white-space/new-line structure. For example, we can
print the file on our console using any of a number of utilities, some of which
follow.

cat
% cat myfile
Ames Mary 123 Maple Stillwater OK 74075 MasonAlan
90 Eastgate Ada OK 74820

Or we can use tools like wc and grep for processing the files. . wc
The command wc (word count) reads through an ASCII file sequentially and
counts the number of lines (delimited by new lines), words (delimited by white
space), and characters in a file:
% wc myfile
2 14 76

grep
It is common to want to know if a text file has a certain word or character string
in it. For ASCII files that can reasonably be searched sequentially, Unix
provides an excellent filter for doing this called grep (and its variants egrep and f
grep). The word grep stands for generalized regular expression, which
describes the type of pattern that grep is able to recognize. In its simplest form,
grep searches sequentially through a file for a pattern. It then returns to standard
output (the console) all the lines in the file that contain the pattern.
% grep Ada myfile
MasonAlan 90 Eastgate Ada OK 74820
Record Access 161

We can also combine tools to create, on the fly, some very powerful file
processing software. For example, to find the number of lines containing the
word Ada and the number of words and bytes in those lines we use

As we move through the text, we will encounter a number of other


powerful Unix commands that sequentially process files with the basic
white-space/new-line structure. ,

5.1.4 Direct Access


The most radical alternative to searching sequentially through a file for a record
is a retrieval mechanism known as direct access. We have direct access to a
record when we can seek directly to the beginning of the record and read it in.
Whereas sequential searching is an O(n) operation, direct access is 0(1). No
matter how large the file is, we can still get to the record we want with a single
seek. Class IOBuffer includes direct read (DRead) and write (DWrite) operations
using the byte address of the record as the record reference: ,
int IOBuffer::DRead (istream & stream, int recref)
// read specified record;
(
stream . seekg (recref, ios::beg); if (stream . tellg
() != recref) return -1; return Read (stream);
}
The DRead function begins by seeking to the requested spot. If this does not
work, the function fails. Typically this happens when the request is beyond the
end-of-file. After the seek succeeds, the regular, sequential Read method of the
buffer object is called. Because Read is virtual, the system selects the correct
one.
Here we are able to write the direct read and write methods for the base
class IOBuffer, even though that class does not have sequential read and write
functions. In fact, even when we add new derived classes with their own different
Read and W r i t e methods, we still do not have to change Dread. Score another
one for inheritance and object-oriented design!
The major problem with direct access is knowing where the beginning of
the required record is. Sometimes this information about record location is
carried in a separate index file. But, for the moment, we assume that
162 Chapter 5 Managing Files of Records

we do not have an index. We assume that we know the relative record number
(RRN) of the record we want. RRN is an important concept that emerges from
viewing a file as a collection of records rather than as a collection of bytes. If a
file is a sequence of records, the RRN of a record gives its position relative to
the beginning of the file. The first record in a file has RRN 0, the next has RRN
1, and so forth.2
In our name and address file, we might tie a record to its RRN by assigning
membership numbers that are related to the order in which we enter the records
in the file. The person with the first record might have a membership number of
1001, the second a number of 1002, and so on. Given a membership number,
we can subtract 1001 to get the RRN of the record. •
What can we do with this RRN? Not much, given the file structures we
have been using so far, which consist of variable-length records. The RRN tells
us the relative position of the record we want in the sequence of records, but we
still have to read sequentially through the file, counting records as we go, to get
to the record we want. An. exercise at the end of this chapter explores a method
of moving through the file called skip sequential processing, which can
improve performance somewhat, but looking for a particular'RRN is still an
0(n) process.
To support direct access by RRN, we need to work with records of. fixed,
known length. If the records are all the same length, we can use a record’s RRN
to calculate the byte offset of the start of the record relative to the start of the
file. For instance, if we are interested in the record with an RRN of 546 and our
file has a fixed-length record size of 128 bytes per record, we can calculate the
byte offset as
Byte offset = 546 x 128 = 69 888
In general, given a fixed-length record file where the record size i$ r, the byte
offset of a record with an RRN of n is
Byte o,ffset = n x r
Programming languages and operating systems differ regarding where this
byte offset calculation is done and even whether byte offsets are used for
addressing within files. In C++ (and the .Unix and MS-DOS operating
systems), where a file is treated as just a sequence of bytes, the application
program does the calculation and uses the seekg and seekp methods to

2. In keeping with the conventions of C and C++, we assume that the RRN is a zero-based count. In some file
systems, the count starts at 1 rather than 0.
jump to the byte that begins the record. All movement within a file is in terms of
bytes. This is a very low-level view of files; the responsibility for translating an
RRN into a byte offset belongs wholly to the application program and not at all
to the. programming language or operating system.
Class FixedLengthBuf f er can be extended with its own methods DRead
and DWrite that interpret the recref argument as RRN instead of byte .address.
The methods are defined as virtual in class IOBuf f er to allow this. The code in
Appendix F does not include this extension; it is left as an exercise.
The Cobol language and the operating environments in which Cobol is
often used (OS/MVS, VMS) are examples of a much different, higher-level view
of files. The notion of a sequence of bytes is simply not present when you are
working with record-oriented files in this environment. Instead, files are viewed
as collections of records that are accessed by keys. The operating system takes
care of the translation between a key and a record’s location. In the simplest
case, the key isjust the record’s RRN, but the determination of location within the
file is still not the programmer’s concern.

5.2 More about Record Structures

5.2.1 Choosing a Record Structure and Record Length


Once we decide to fix the length of our records so we can use the RRN to give us
direct access to a record, we have to decide on a record length. Clearly, this
decision is related to the size of the fields we want to store in the record.
Sometimes the decision is easy. Suppose we are building a file of sales
transactions that contain the following information about each transaction:
■ A six-digit account number of the purchaser,
■ Six digits for the date field,
■ A five-character stock number for the item purchased,
H A three-digit field for quantity, and
■ A ten-position field for total cost.
These are all fixed-length fields; the sum of the field lengths is 30 bytes.
Normally we would stick with this record size, but if performance is so
important that we need to squeeze every bit of speed out of our retrieval system,
we might try to fit the record size to the block organization of our
16 4 Chapters Managing Files of Records

disk. For instance, if we intend to store the records on a typical sectored disk
(see Chapter 3) with a sec te r size of 512 bytes or some other power of 2, we
might decide to pad the record out to 32 bytes so we can place an integral
number of records in a sector. That way, records will never span sectors.
The choice of a record length is more complicated when the lengths of the
fields can vary, as in our name and address file. If we choose a record length
that is the sum of our estimates of the largest possible values for all the fields,
we can be reasonably sure that we have enough space for everything, but we
also waste a lot of space. If, on the other hand, we are conservative in our use of
space and fix the lengths of fields at smaller values, we may have to leave
information out of a field. Fortunately, we can avoid this problem to some
degree by appropriate design of the field structure within a record.
In our earlier discussion of record structures, we saw that there are two
general approaches we can take toward organizing fields within a fixed- length
record. The first, illustrated in Fig. 5.1(a) and implemented in class
FixedFieldBuf f er, uses fixed-length fields inside the fixed-length record. This
is the approach we took for the sales transaction file previously described. The
second approach, illustrated in Fig. 5.1(b), uses the fixed-length record as a
kind of standard-sized container for holding something that looks like a
variable-length record.
The first approach has the virtue of simplicity: it is very easy to “break
out” the fixed-length fields from within a fixed-length record. The second
approach lets us take advantage of an aver aging-out effect that usually occurs:
the longest names are not likely to appear in the same record as the longest
address field. By letting the field boundaries vary, we can make

Figure 5.1 Two fundamental approaches to field structure within a fixed-


length record, (a) Fixed-length records with fixed-length fields, (b) Fixed-
length records with variable-length fields.
More about Record Structures 165

more efficient use of a fixed amount of space. Also, note that the two
approaches are not mutually exclusive. Given a record that contains a number
of truly fixed-length fields and some fields that have variable- length
information, we might design a record structure that combines these two
approaches.
One interesting question that must be resolved in the design of this kind
of structure is that of distinguishing the real-data portion of the record from the
unused-space portion. The range of possible solutions parallels that of the
solutions for recognizing variable-length records in any other context: we can
place a record-length count at the beginning of the record, we can use a special
delimiter at the end of the record, we can count fields, and so on. As usual,
there is no single right way to implement this file structure; instead we seek the
solution that is most appropriate for our needs and situation.
Figure 5.2 shows the hex dump output from the two styles of repre-
senting variable-length fields in a fixed-length record. Each file has a header
record that contains three 2-byte values: the size of the header, the number of
records, and the size of each record. A full discussion of headers is deferred to
the next section. For now, however, just look at the structure of the data
records. We have italicized the length fields at the start of the records in the file
dump. Although we filled out the records in Fig. 5.2b with blanks to make the
output more readable, this blank fill is unnecessary. The length field at the start
of the record guarantees that we do not read past the end of the data in the
record.

5.2.2 Header Records


It is often necessary or useful to keep track of some general information about a
file to assist in future use of the file. A header record is often placed at the
beginning of the file to hold this kind of information. For example, in some
languages there is no easy way to jump to the end of a file, even though the
implementation supports direct access. One simple solution is to keep a count
of the number of records in the file and to store that count somewhere. We
might also find it useful to include information such as the length of the data
records, the date and time of the file’s most recent update, the name of the file,
and so on. Header records can help make a file a self-describing object, freeing
the software that accesses the file from having to know a priori everything
about its structure, hence making the file-access software able to deal with
more variation in file structures.
The header record usually has a different structure than the data records
in the file. The file of Fig. 5.2a, for instance, uses a 32-byte header
166 Chapter? Managing Files of Records

More about Record Structures 167

record, whereas the data records each contain 64 bytes. Furthermore, the data
records of this file contain only character data, whereas the header record
contains integer fields that record the header record size, the number of data
records, and the data record size.
Header records are a widely used, important file design tool. For example,
when we reach the point at which we are discussing the construction of
tree-structured indexes for files, we will see that header records are often
placed at the beginning of the index to keep track of such matters as the RRN
of the record that is the root of the index.

5.2.3 Adding Headers to C++ Buffer Classes


This section is an example of how to add header processing to the IOBuf f er
class hierarchy. It is not intended to show an optimal strategy for headers.
However, these headers are used in all further examples in the book. The Open
methods of new classes take advantage of this header strategy to verify that the
file being opened is appropriate for its use. The important principle is that each
file contains a header that incorporates information about the type of objects
stored in the file.
The full definition of our buffer class hierarchy, as given in Appendix F,
has been extended' to include methods that support header records. Class
IOBuf fer includes the following methods:
virtual int ReadHeader ();
virtual int WriteHeader () ,-
Most of the classes in the hierarchy include their own versions of these
methods. The write methods add a header to a file and return the number of
bytes in the header. The read methods read the header and check for
consistency. If the header at the beginning of a file is not the proper header for
the buffer object, a FALSE value is returned; if it is the correct header, TRUE
is returned.
To illustrate the use of headers, we look at fixed-length record files as
defined in classes IOBuf fer and FixedLengthBuf fer. These classes were
introduced in Chapter 4 and now include methods ReadHeader and
WriteHeader. Appendix F contains the implementation of these methods of all
of the buffer classes. The WriteHeader method for IOBuf fer writes the string
IOBuf fer at the beginning of the file. The header for FixedLengthBuf fer adds
the string Fixed and the record size.
■ The ReadHeader method of FixedLengthBuf fer reads the record size from
the header and checks that its value is the same as that of the Buf f e r S i z e
member of the buffer object. That is, ReadHeader
168 Chapter 5 Managing Files of Records

verifies that the file was created using fixed-size records that are the right size
for using the buffer object for reading and writing.
Another aspect of using headers in these classes is that the header can be
used to initialize the buffer. - At the end of FixedLengthBuf f er : :ReadHeader
(see Appendix F), after the buffer has been found to be uninitialized, the record
size of the buffer is set to the record size that was read from the header.
You will recall that in Section 4.5, “Ah Object-Oriented Class for Record
Files,” we introduced class B u f f e r F i l e as a way to guarantee the proper
interaction between buffers and files. Now that the buffer classes support
headers, Buf f e r F i l e : : Create puts the correct header in every file, and
B u f f e r : : Open either checks for consistency or initializes the buffer, as
appropriate. Buf f e r F i l e : : ReadHeader is called by Open and does all of its
work in a single virtual function call. Appendix F has the details of the
implementation of these methods.
Buf f e r F i l e : : Rewind repositions the get and put file pointers to the
beginning of the first data record—that is, after the header record. This method
is required because the Headersize member is protected. Without this method,
it would be impossible to initiate a sequential read of the file.

5.3 Encapsulating Record I/O Operations in a Single Class

A good object-oriented design for making objects persistent should provide


operations to read and write objects directly. So far, the write operation
requires two separate operations: pack into a buffer and write the buffer to a
file. In this section, we introduce class RecordFile which supports a read
operation that takes an object of some class and writes it to a file. The use of
buffers is hidden inside the class.
The major problem with defining class RecordFile is howto make it
possible to support files for different object types without needing different
versions of the class. Consider the following code that appears to read a Person
from one file and a Recording (a class defined in Chapter 7) from another file:
Person p; RecordFile pFile; pFile . Read (p);
Recording r; RecordFile rFile; rFile'. Read (r);
Is it possible that class RecordFile can support read and unpack for a Person
and a Recording without change? Certainly the objects are different—they
have different unpacking methods. Virtual function calls
Encapsulating Record I/O Operations in a Single Glass 169

do not help because Person and Recording do not have a common base type."
It is the C++ template feature that solves our problem by supporting
parameterized function and class definitions. Figure 5.3 gives the definition
of the template class RecordFile.

#include "buffile.h"
If include "iobuffer.h”
// template class to support direct read and write of records // The template
parameter RecType must.support the following //. int Pack (BufferType &); pack
record into buffer .
// int Unpack {BufferType S c ) ; unpack record from buffer

template cclass RecType>


class RecordFile: public BufferFile
{public:
int Read (RecType & record, int recaddr = -1);
int Write (const RecType & record, int recaddr = -1);
RecordFile (IOBuffer & buffer) : Buf ferFile (buffer) {}'
};

// template method bodies template cclass RecType>


int RecordFile<RecType>: :Read (RecType & record/ irit recaddr = -1)
{.
int writeAddr, result;
writeAddr = BufferFile::Read (recaddr); if (!writeAddr) return
-1; result = record . Unpack (Buffer)? if (Iresult) return -1;
return writeAddr;
}'
template cclass RecType>
int RecordFilecRecType>::Write (const RecType & record, int recaddr = -1)
' #
{ '.
int result;
result = record . Pack (Buffer) ?
if (iresult) return -1;
return Buf ferFile: .-Write (recaddr);
}

Figure 5.3 Template class RecordFile.


170 Chapter5 Managing Files of Records

The definition of class RecordFile is a template in the usual sense of the


word: a pattern that is used as a guide to make something accurately. The
definition does not define a specific class but rather shows how particular
record file classes can be constructed. When a template class is supplied with
values for its parameters, it becomes a real class. For instance, the following
defines an object called PersonFile:
RecordFile<Person> PersonFile (Buffer);

The object Personf ile is a RecordFile that operates on Person objects. All
of the operations of RecordFile<Person> are available, including those from
the parent class Buf ferFile. The following code includes legitimate uses of
PersonFile:
Person person;
PersonFile.CreateCperson.dat", ios::in); // create a file
PersonFile.Read(person); // read a record into person
PersonFile.Append(person); // write person at end of file
PersonFile.Open("person.dat", ios-.:in) ; // open and check header
Template definitions in C++ support the reuse of code. We can write a
single class and use it in multiple contexts. The same RecordFile class
declared here and used for files of Person objects will be used in subsequent
chapters for quite different objects. No changes need be made to RecordFile to
support these different uses.
Program test file . cpp, in Appendix F, uses RecordFile to test all of the
buffer I/O classes. It also includes a template function, TestBuffer, which is
used for all of the buffer tests.

5.4 File Access and File Organization

In the course of our discussions in this and the previous chapter, we have
looked at
■ Variable-length records,
■ Fixed-length records,
* Sequential access, and
■ Direct access.
The first two of these relate to aspects of file organization; the last two have to
do with file access. The interaction between file organization and file access is
a useful one; we need to look at it more closely before continuing.
Most of what we have considered so far falls into the category of file
organization:
■ Can the file be divided into fields?
■ Is there a higher level of organization to the file that combines the fields into
records?
■ Do all the records have the same number of bytes or fields?
■ How do we distinguish one record from another?
■ How do we organize the internal structure of a fixed-length record so we can
distinguish between data and extra space?
We have seen that there are many possible answers to these questions and that
the choice of a particular, file organization depends on many things, including
the file-handling facilities of the language you are using and the use you want
to make of the file.
Using a file implies access. We looked first at sequential access, ulti-
mately developing a sequential search. As long as we did not know where
individual records began, sequential access was the only option open to us.
When we wanted direct access, we fixed the length of our records, and this
allowed us to calculate precisely where each record began and to seek directly
to it.
In other words, our desire for direct access caused us to choose a fixed-
length record file organization. Does this mean that we can equate fixed-
length records with direct access? Definitely not. There is nothing about our
having fixed the length of the records in a file that precludes sequential access;
we certainly could write a program that reads sequentially through a fixed-
length record file.
Not only can we elect to read through the fixed-length records sequen-
tially but we can also provide direct access to variable-length records simply
by keeping a list of the byte offsets from the start of the file for the placement
of each record. We chose a fixed-length record structure for the files of Fig.
5.2 because it is simple and adequate for the data we wanted to store.
Although the lengths of our names and addresses vary, the variation is not so
great that we cannot accommodate it in a fixed-length record.
Consider, however, the effects of using a fixed-length record organization
to provide direct access to documents ranging in length from a few hundred
bytes to more than a hundred kilobytes. Using fixed-length
17 2 Chapter 5 Managing Files of Records
t

records to store these documents would be disastrously wasteful of space, so some


form of variable-length record structure would have to be found. Developing
file structures to handle such situations requires that you clearly distinguish
between the matter of access and your options regarding organization.
The restrictions imposed by the language and file system used to develop
your applications impose limits on your ability to take advantage of this distinction
between access method and organization. For example, the C++ language provides
the programmer with the ability to implement direct access to variable-length
records, since it allows access to any byte in the file. On the other hand, Pascal, even
when seeking is supported, imposes limitations related to the language’s definition
of a file as a collection of elements that are all of the same type and, consequently,
size. Since the elements must all be of the same size, direct access to variable-
length records is difficult, at best, in Pascal. . •
CHAPTER 6

Organizing Files
for Performance

6.1 Data Compression

In this section we look at some ways to make files smaller. There are many
reasons for making files smaller. Smaller files
■ Use less storage, resulting in cost savings;
B • Can be transmitted faster, decreasing access time or, alternatively, allowing the
same access time with a lower and cheaper bandwidth; and B Can be processed
faster sequentially.
Data compression involves encoding the information in a file in such a
way that it takes up less space. Many different techniques are available for
compressing data. Some are very general, and some are designed for specific
kinds of data, such as speech, pictures, text, or instrument data. The variety of
data compression techniques is so large that we can only touch on the topic
here, with a few examples.

6.1.1 Using a Different Notation


Remember our Person file from Chapter 4? It had several fixed-length fields,
including LastName, State, arid ZipCode. Fixed-length fields such as these are
good candidates for compression. For instance, the State field in the Person file
required 2 ASCII bytes, 16 bits. How many bits are really needed for this field?
Since there are only fifty states, we could represent all possible states with only
6 bits. Thus, we could encode all state names in a single 1-byte field, resulting in
a space savings of 1 byte, or 50 percent, per occurrence of the state field.
This type of compression technique, in which we decrease the number of
bits by finding a more compact notationis one of many compression
techniques classified as redundancy reduction.-The 10 bits that we were
able to throw away were redundant in the sense that having 16 bits
instead, of 6 provided no extra information.
What ar.e the costs of this compression scheme?- In this case, there are
many:
■ By using a pure binary encoding, we have made the file unreadable by
humans.
■ We incur some cost in encoding time whenever we add a new state- name field
to our file and a similar cost for decoding when we need to get a readable
version of state name from the file.
■ We must also now incorporate the encoding and/or decoding modules in all
software that will process our address file, increasing the complexity of the
software.
With so many costs, is this kind of compression worth it? We can answer
this only in the context of a particular application. If the file is already fairly small,
if the file is often accessed by many different pieces of software, and if some of the
software that will access the file cannot deal with binary data (for example, an
editor), then this form of compression is a bad idea. On the other hand, if the file
contains several million records and is generally processed by one program,
compression is probably a very good idea. Because the encoding and decoding
algorithms for this kind of compression are extremely simple, the savings in access
time is likely to exceed any processing time required for encoding or decoding.

6.1.1 Assigning Variable-Length Codes


Suppose you have two different symbols to use in an encoding scheme: a dot (•)
and a dash (-). You have to assign combinations of dots and dashes. to letters of
the alphabet. If you are very clever, you might determine the most frequently
occurring letters of the alphabet (e and t) and use a single dot for one and a
single dash for the other. Other letters of the alphabet will be assigned two or
more symbols, with the more frequentiy occurring letters getting fewer
symbols.
Sound familiar? You may recognize this scheme as the oldest and most
common of the variable-length codes, the Morse code. Variable-length codes,
in general, are based on the principle that some values occur more frequently
than others, so the codes for those values should take the least amount of space.
Variable-length codes are another form of redundancy reduction.
A variation on the compact notation technique, the Morse code can be
implemented using a table lookup, where.the table never changes. In contrast,
since many sets of data values do not exhibit a predictable frequency
distribution, more modern variable-length coding techniques dynamically
build the tables that describe the encoding scheme. One of the most successful
of these is the Huffman code, which determines the probabilities of each value
occurring in the data set and then builds a binary tree in which the search path
for each value represents the code for that •value. More frequently occurring
values are given shorter search paths in the tree. This tree is then turned into a
table, much like a Morse code table, that can be used to encode and decode the
data.
For example, suppose we have a data set containing only the seven letters
shown in Fig. 6.2, and each letter occurs with the probability indicated. The
third row in the figure shows the Huffman codes that would be assigned to the
letters. Based on Fig. 6.2, the string “abde” would be encod- ed as
“101000000001.”
In the example, the letter a occurs much more often than any of the others,
so it is assigned the 1-bit code 1. Notice that the minimum number of bits
needed to represent these seven letters is 3, yet in this case as many as 4 bits are
required. This is a necessary trade-off to ensure that the
Data Compression 207

Letter: a b c d . e f g
Probability: 0.4 0.1 0.1 0.1 0.1 0.1 0.1
Code 1 010 Oil 0000 0001 0010 0011

Figure 6.2 Example showing the Huffman encoding for a set of seven letters, assuming
certain probabilities (from Lynch, 1985).

distinct codes can be stored together, without delimiters between them, and still be recognized.

6.1.4 Irreversible Compression Techniques


The techniques we have discussed so far preserve all information in the original data. In effect,
they take advantage of the fact that the data, in its original form, contains redundant information
that can be removed and then reinserted at a later time. Another type of compression, irreversible
compression, is based on the assumption that some information can be sacrificed.2
An example of irreversible compression would be shrinking a raster image from, say, 400-
b'y-400 pixels to 100-by-100 pixels. The new image contains 1 pixel for every 16 pixels in the
original image, and there is no way, in general, to determine what the original pixels were from'the
one new pixel.
Irreversible compression is less common in data files than reversible compression, but there
are times when the information that is lost is of little or no value. For example, speech compression
is often done by voice coding, a technique that transmits a paramaterized description of speech,
which can be synthesized at the receiving end with varying amounts of distortion.

6.2 Reclaiming Space in Files

Suppose a record in a variable-length record file is modified in such a way


that the new record is longer than the original record. What do you do with
the extra data? You could append it to the end of the file and put a pointer
from the original record space to the extension of the record. Or you could
rewrite the whole record at the end of the file (unless the file needs to be
sorted), leaving a hole at the original location of the record. Each solution has
a drawback: in ,the former case, the job of processing the record is more
awkward and slower than it was originally; in the latter case, the file contains
wasted space.
In this section we take a close look at the way file organization deteri-
orates as a file is modified. In general, modifications can take any one of
three forms:
■ Record addition,
■ Record updating, and
H Record deletion.
If the only kind of change to a file is record addition, there is
no dete- rioration of the kind we cover in this chapter. It is only when
variable- length records are updated, or when either fixed- or
variable-length records are deleted, that maintenance issues become
complicated and interesting. Since record updating can always be treated
as a record deletion followed by a record addition, our focus is on
the effects of record deletion. When a record has been deleted, we
want to reuse the space.

6.2.1 Record Deletion and Storage Compaction


Storage compaction makes files smaller by looking for places in a file where
there is no data at all and recovering this space. Since empty spaces occur in
files when we delete records, we begin our discussion of compaction with a
look at record deletion.
Any record-deletion strategy.must provide some way for us to recognize
records as deleted. A simple and usually workable approach is to place a
special mark in each deleted record. For example, in the file of Person objects
with delimited fields developed in Chapter 4, we might place an asterisk as the
first field in a deleted record. Figures 6.3(a) and 6.3(b) show a name and
address file similar to the one in Chapter 4 before and after the second record
is marked as deleted. (The dots at the ends of records 0 and 2 represent padding
between the last field and the end of each record.)
Once we are able to recognize a record as deleted, the next question is
how to reuse the space from the record. Approaches to this problem that rely
on storage compaction do not reuse the space for a while. The records are
simply marked as deleted and left in the file for. a period of time. Programs
using the file must include logic that causes them to ignore records that are
marked as deleted. One benefit to this approach is that it is usually possible to
allow the user to undelete a record with very little

Figure 6.3 Storage requirements of sample,file using 64-byte fixed-length records,


(a) Before deleting the second record, (b) After deleting the second record.(c) After
compaction—the second record is gone.
210 Chapter 6 Organizing Files for Performance

effort. This is particularly easy'if you keep the deleted mark in a special field
rather than destroy some of the original data, as in our example.
The reclamation of space from the deleted records happens all at once.
After deleted records have accumulated for some time, a special program is
used to reconstruct the file with all the deleted records squeezed out as shown
in Fig. 6.3(c). If there is enough space, the simplest way to d'o this
compaction is through a file copy program that skips over the deleted
records. It is also possible, though more complicated and time-consuming, to
do the compaction in place. Either of these approaches can be used with both
fixed-and variable-length records.
The decision about how often to run the storage compaction program can
be based on either the number of deleted records or the calendar. In
accounting programs, for example, it often makes sense to run a compaction
procedure on certain files at the end of the fiscal year or at some other point
associated with closing the books.

6.2.1 Deleting Fixed-Length Records for Reclaiming Space


Dynamically
Storage compaction is the simplest and most widely used of the storage
reclamation methods we discuss. There are some applications, however, that
are too volatile and interactive for storage compaction to be useful. In these
situations we want to reuse the space from deleted records as soon as
possible. We begin our discussion of such dynamic storage reclamation with
a second look at fixed-length record deletion, since fixed-length records
make the reclamation problem much simpler.
In general, to provide a mechanism for record deletion with subse- ■
quent reutilization of the freed space, we need to be able to guarantee two
things:
B That deleted records are marked in some special way, and
B That we can find the space that deleted records once occupied so we can reuse
that space when we add records.
We have already identified a method of meeting the first requirement: we
mark records as deleted by putting a field containing an asterisk at the
beginning of deleted records.'
If you are working with fixed-length records and are willing to search
sequentially through a file before adding a record, you can always provide
the second guarantee if you have provided the first. Space reutilization can
take the form of looking through the file, record by record, until a deleted
Reclaiming Space in Files 211

record is found. If the program reaches the end of the file without finding a
deleted record, the new record can be appended at the end.
Unfortunately, this approach makes adding records an intolerably slow
process, if the program is an interactive one and the user has to sit at the
terminal and wait as the record addition takes place. To make record reuse
happen more quickly, we need
■ A way to know immediately if there are empty slots in the file, and
■ A way to jump directly to one of those slots if they exist.

Linked Lists
The use of a linked list for stringing together all of the available records can
meet both of these needs. A linked list is a data structure in which each element
or node contains some kind of reference to its successor in the list. (See Fig.
6.4.)
If you have a head reference to the first node in the list, you can move
through the list by looking at each node and then at the node’s pointer field, so
you know where the next node is located. When you finally encounter a pointer
field with some special, predetermined end-of-list value, you stop the traversal
of the list. In Fig. 6.4 we use a - I in the pointer field to mark the end of the list.
When a list is made up of deleted records that have become available
space within the file, the list is usually called an avail list. When inserting a
new record into a fixed-length record file, any one available record is just as
good as any other. There is no reason to prefer one open slot over another since
all the slots are the same size. It follows that there is no reason to order the
avail list in any particular way. (As we see later, this situation changes for
variable-length records.)

Stacks
The simplest way to handle a list is as a stack. A stack is a list in which all
insertions and removals of nodes take place at one end of the list. So, if we

Figure 6.4 A linked list.


Chapter 6 Organizing Files for Performance

have an avail list managed as a stack that contains relative record numbers
(RRN) 5 and 2, and then add RRN 3, it looks like this before and after the
addition of the new node:

When a new node is added to the top or front of a stack, we say that it is
pushed onto the stack. If the next thing that happens is a request for some
available space, the request is filled by taking RRN 3 from the avail list. This
is called popping the stack. The list returns to a state in which it contains only
records 5 and 2.

Linking and Stacking Deleted Records


Now we can meet the two criteria for rapid access to reusable space from
deleted records. We need
■ A way to know immediately if there are empty slots in the file, and
■ A way to jump directly to one of those slots if it exists.
Placing the deleted records on a stack meets both criteria. If the pointer to
the top of the stack contains the end-of-list value, then we know that there are
no empty slots and that we have to add new records by appending them to the
end of the file. If the pointer to the stack top contains a valid node reference,
then we know not only that a reusable slot is available, but also exactly where
to find it.
Where do we keep the stack? Is- it a separate list, perhaps maintained in a
separate file, or is it somehow embedded within the data file? Once again, we
need to be careful to distinguish between physical.and conceptual structures.
The deleted, available records are not moved anywhere when they are pushed
onto the stack. They stay right where we need them, located in the file. The
stacking and linking are done by arranging and rearranging the links used to make
one available record slot point to the next. Since we are working with fixed-length
records in a disk file rather than with memory addresses, the pointing is not
done with pointer variables in the formal sense but through relative record
numbers (RRNs).
Reclaiming Space in Files 21 3

Suppose we are working with a fixed-length record file that once


contained seven records (RRNs 0-6). Furthermore, suppose that records 3
and 5 have been deleted, in that order, and that deleted records are marked by
replacing the first field with an asterisk. We can then use the second field of a
deleted record to hold the link to the next record on the avail list. Leaving out
the details of the valid, in-use records, Fig. 6.5(a) shows how the file might
look.
Record 5 is the first record on the avail list (top of the stack) as it is the
record'that is most recently deleted. Following the linked list, we see that
record 5 points to record 3. Since the link field for record 3 contains -1, which
is our end-of-list marker, we know that record 3 is the last slot available for
reuse.
Figure 6.5(b) shows the same file after record 1 is also deleted. Note that
the contents of all the other records on the avail list remain unchanged.
Treating the list as a stack results in a minimal amount of list reorganization
when we push and pop records to and from' the list.
If we now add a new name to the file, it is placed in record 1, since RRN
1 is the first available record. The avail list would return to the

List head (first available record) - + - 5

Figure 6.5 Sample fiie showing linked lists of deleted records, (a) After deletion of records
3 and 5, in that order, (b) After deletion of records 3,5, and 1, in that order, (c) After
insertion of three new records.
214 Chapter 6 Organizing Files for Performance

configuration shown in Fig. 6.5(a). Since there are still-two record slots on the
avail list, we could add two more names to the file without increasing the size
of the file. After that, however, the avail list would be empty as shown in Fig.
6.5(c). If yet another name is added to the file, the program knows that the
avail list is empty and that the name requires the addition • of a new record at
the end of the file.

Implementing Fixed-Length Record Deletion

Implementing mechanisms that place deleted records on a linked avail list and
that treat the avail list as a stack is relatively straightforward. We need a
suitable place to keep the RRN of the first available record on the avail list.
Since this is information that is specific to the data file, it can be carried in a
header record at the start of the file.
When we delete a record, we must be able to mark the record as deleted
and then place it on the avail list. A simple way to doithis is to place an * (or
some other special mark) at the beginning of the record as a deletion mark,
followed by the RRN of the next record on the avail list.
Once we have a list of available records within a file, we can reuse the
space previously occupied by deleted records. For this we would write a single
function thatreturns either (I) the RRN of a reusable record slot or (2) the RRN
of the.next record to be appended if no reusable slots are available.

6.2.2 Deleting Variable-Length Records


Now that we have-a mechanism for handling an avail list of available space
once records are deleted, let’s apply this mechanism to the more complex
problem of reusing space from deleted variable-length records. We have seen
that to support record reuse through an avail list, we need
■ A way to link the deleted records together into a list- (that is, a place to
put a link field); ;
■ An algorithm for adding newly deleted records to the avail list; and
■ An algorithm for finding and removing records from the avail list when
we are ready to use them.

An Avail List of Variable-Length Records


What kind of file structure do we need to support an avail list of variable-
length records? Since we will want to delete whole records and then' place
records on an avail list, we need a structure in which the record is a clearly
defined entity. The file structure o f V a r i a b l e L e n g t h B u f f e r , in which we
define the length of each record by placing a byte count at the beginning o f
each record, will serve us well in this regard.
We can handle the contents of a deleted variable-length record just as we
did with fixed-length records. That is, we can place a single asterisk in the first
field, followed by a binary link field pointing to the next deleted record on the
avail list. The avail list can be organized just as it was with fixed-length records,
but with one difference: we cannot use relative record numbers for links. Since
we cannot compute the byte offset of variable- length records from their RRNs,
the links must contain the byte offsets themselves.
To illustrate, suppose we begin with a variable-length record file containing
the three records for Ames, Morrison, and Brown introduced earlier. Figure
6.6(a) shows what the file looks like (minus the header) before any deletions,
and Fig. 6.6(b) shows what it looks like after the deletion of the second record.
The periods in the deleted record signify discarded characters.

Adding and Removing Records

Let’s address the questions of adding and removing records to and from the list
together, since they are clearly related. With fixed-length records we

HE AD. F IRST _A VA IL : -1

Figure 6.6 A sample file for illustrating variable-length record deletion, (a) Original
sample file stored in variable-length format with byte count (header record not
included), (b) Sample file after deletion of the second record (periods show discarded
characters).
216 Chapter 6 Organizing Files for Performance

could access the avail list as a stack because one member of the avail list is just
as usable as any other. That is not true when, the record slots on the avail list
differ in size, as they do in a variable-length record file. We now have an extra
condition that must be met before we can reuse a record: the record must be the
right size. For the moment we define right size as “big enough.” Later we find
that it is sometimes useful to be more particular about the meaning of right
size.
It is possible, even likely,-that we need to search through the avail list for
a record slot that is the right size. We can’t just pop the stack and expect the
first available record to be big enough. Finding a proper slot on the avail list
now means traversing the list until a record slot that is big enough to hold the
new record is found.
For example, suppose the avail list contains the deleted record slots shown
in Fig. 6.7(a), and a record that requires 55 bytes is to be added. Since the avail
list is not empty, we traverse the records whose sizes are 47 (too small), 38
(too small), and 72 (big enough). Having found a slot big enough to hold our
record, we remove it from the avail list by creating a new link that jumps over
the record as shown in Fig. 6.7(b). If we had reached the end of the avail list
before finding a record' that was large enough, we would have appended the
new record at the end of the file.
Because this procedure for Finding a reusable record looks through the
entire avail list if necessary, we do not need a sophisticated method for putting
newly deleted records onto the list. If a record of the right size is

Figure 6.7 Removal of a record from an avail list with variable-length records,
(a) Before removal, (b) After removal.
somewhere on this list, our get-available-record procedure eventually finds it.
It follows that we can continue to push new members onto the front of the list,
just as we do with fixed-length records.
Development of algorithms for adding and removing avail list records is
left to you as part of the exercises found at the end of this chapter.

6.2.3 Storage Fragmentation


Let’s look again at the fixed-length record version of our three-record file
(Fig. 6.8). The dots at the ends of the records represent characters we use as
padding between the last field and the end of the records. The padding is
wasted space; it is part of the cost of using fixed-length records. Wasted space
within a record is called internal fragmentation.
Clearly, we want to minimize internal fragmentation. If we are working
with fixed-length records, we attempt this by choosing a record length that is
as close as possible to what we need for each record. But unless the. actual
data is fixed in length, we have to put up with a certain amount of internal
fragmentation in a'fixed-length record file. •
One of the attractions of variable-length records is that they minimize
wasted space by doing away with internal fragmentation. The space set aside
for each record is exactly as long as it needs to be. Compare the fixed- length
example with the one in Fig. 6.9, which uses the variable-length record
structure—a byte count followed by delimited data fields. The only space
(other than the delimiters) that is not used for holding data in each record is
the count field. If we assume that this field uses 2 bytes, this amounts to only
6 bytes for the three-record file. The fixed-length record file wastes 24 bytes
in the very first record.

AmesiM ary1123 M apl e I Stillwater!OK 174075 I ........................ .................M


orrison I Sebastian I 9035 Sout h HillcrestI F orest Village I0KI74820I Brown I M
art ha!6 25 K i m bar kl D es M oi n es 1I A 15 0311 1 ................. ............................ .

Figure 6.8 Storage requirements of sample file using 64-byte fixed-length records.

40 Ames I M ary1123 M a ple! Still water! OK 17407 5! 64 M orris on I Sebas t i a n 1 90 35


Sout h Hillcrest 1'F orest Village I OK 174820 145 Brown I Mart ha I 62 5
Ki m barklD es M oi nes !I Ai503 11!

Figure 6.9 Storage requirements of sample file using variable-length records with a
count field.
218 Chapter 6 Organizing Fifes for Performance

But before we start congratulating ourselves for solving the problem of


wasted space due to internal fragmentation, we should consider what happens
in a variable-length record file after a record is deleted and replaced with a
shorter record. If the shorter record takes less space than the original record,
internal fragmentation results. Figure 6.10 shows how the problem could
occur with our sample file when the second record in the file is deleted and the
following record is added:
Ham|Al1 2 8 Elm I Ada|O K I 7 0 3 3 2 I
It appears that escaping internal fragmentation is not so easy. The slot
vacated by the deleted record is 37 bytes larger than is needed for the new-
record. Since we treat the extra 37 bytes as part of the new record, they are
not on the avail list and are therefore unusable. But instead of keeping the
64-byte record slot intact, suppose we break it into two parts: one part to hold
the new Ham record, and the other to be placed back on the avail list. Since
we would take only as much space as necessary for the Ham record, there
would be no internal fragmentation.
Figure 6.11 shows what our file looks like if we use this approach to
insert the record for Al Ham. We steal the space for the Ham record from the e
n d of the 64-byte slot and leave the first 35 bytes of the slot on the avail list.
(The available space is 3 5 rather than 37 bytes because we need 2. bytes to
form a new size field for the Ham record.) The 35 bytes still on the avail list
can be used to hold yet another record. Figure 6.12 shows the effect of
inserting the following 25-byte record:.

Lee I Ed IRt 2 I Ada I OK 1 7 4 8 2 0 I '

Figure 6.10 Illustration of fragmentation with variable-length records, (a) After deletion of
the second record (unused characters in the deleted record are replaced by periods), (b)
After the subsequent addition of the record for Al Ham.
Reclaiming Space in Files

Figure 6.11 Combating internal fragmentation by putting the unused part of the
deleted slot back on the avail list.

As we would expect, the new record is carved out of the 35-byte record that is
on the avail list. The data portion of the new record requires 25 bytes, and we
need 2 more bytes for another size field. This leaves 8 bytes in the record still
on the avail list.
What are the chances of finding a record that can make use of these 8
bytes? Our guess would be that the probability is close to zero. These 8 bytes
are not usable, even though they are not trapped inside any other record. This
is an example of external fragmentation. The space is actually on the avail list
rather than being locked inside some other record but is too fragmented to be
reused.
There are some interesting ways to combat external fragmentation. One
way, which we discussed at the beginning of this chapter, is storage
compaction. We could simply regenerate the file when external fragmentation
becomes intolerable. Two other approaches'are as follows:
■ If two record slots on the avail list are physically adjacent, combine them
to make a single, larger record slot. This is called coalescing the holes in
the storage space.
■ Try to minimize fragmentation before it happens by adopting a placement
strategy that the program can use as it selects a record slot from the avail
list.

Fi g ure 6 .12 Addition of the second record into the slot originally occupied by a single
deleted record.
Chapter 6 Organizing Files for Performance

Coalescing holes presents some interesting problems. The avail list is not
kept in physical record qrder; if there are two deleted records that are
physically adjacent, there is no reason to presume that they are linked adjacent
to each other on the avail list. Exercise 15 at the.end of this chapter provides a
discussion of this problem along with a framework for developing a solution.
The development of better placement strategies, however, is a different
matter. It is a topic that warrants a separate discussion, since the choice among
alternative strategies is not as obvious as it might seem at first glance.

6.2.5 Placement Strategies


Earlier we discussed ways to add and remove variable-length records from an
avail list. We add records by treating the avail list as a stack and putting
deleted records at the front. When we need to remove a record slot from the
avail list (to add a record to the file), we look through the list, starting at the
beginning, until we either find a record slot that is big enough or reach the end
of the list.
This is called a first-fit placement strategy. The least possible amount of
work is expended when we place newly available space on the list, and we are
not very particular about the closeness of fit as we look for a record slot to hold
a new record. We accept the first available record slot that will do the job,
regardless of whether the slot.is ten times bigger than what is needed or
whether it is a perfect fit.
We could, of course, develop a more orderly approach for placing records
on the avail list by keeping them in either ascending or descending sequence
by size. Rather than always putting the newly deleted records at the front of
the list, these approaches involve moving through the list, looking for the place
to insert the record to maintain the desired sequence.
If we order the avail list in ascending order by size, what is the effect on
the closeness of fit of the records that are retrieved from the list? Since the
retrieval procedure searches sequentially through the avail list until it
encounters a record that is big enough to hold the new record, the first record
encountered is the smallest record that will do the job. The fit between the
available slot and the new record’s needs would be as close as we can make it.
This is called a best-fit placement strategy.
A best-fit strategy is intuitively appealing. There is, of course, a price to
be paid for obtaining this fit. We end up having to search through at
Reclaiming Space in Files 221

least a part of the list—not only when we get records from the list, but also
when we put newly deleted records on the list. In a real-time environment,
the extra processing time could be significant.
A less obvious disadvantage of the best-fit strategy is related to the idea
of finding the best possible fit and ensuring that the free area left over after
inserting a new record into a slot is as small as possible. Often this remaining
space is too small to be useful, resulting in external fragmentation.
Furthermore, the slots that are least likely to be useful are the ones that will
be placed toward the beginning of the list, making first-fit searches longer as
time go ?s on.
These problems suggest an alternative strategy. What if we arrange the
avail list so it is in descending order by size? Then the largest record slot on
the avail list would always be at the head of the list. Since the procedure that
retrieves records starts its search at the beginning of the avail list, it always
returns the l irgest available record slot if it returns any slot at all. This is
known as a v^orst-fit placement strategy. The amount of space in the record
slot, beyond jwhat is actually needed, is as large as possible.
A worst-fit strategy does not, at least initially, sound very appealing. But
consider the fol owing:
■ The procedure for removing records can be simplified so it looks only at
the first element of the avail list. If the first record slot is not large enough
to do tne job, none of the others will be.
■ By extracting the space we need from the largest available slot, we are
assured that th e unused portion of the slot is as large as possible,
decreasing the li kelihood of external fragmentation.

What can you cbnclude from all of this? It should be clear that no one
placement strategy ;.s superior under all. circumstances. The best you can do
is formulate a series of general observations, and then, given a particular
design situation, try to select the strategy that seems most appropriate. Here
are some suggestions. The judgment will have to be yours.
■ Placement strategies make sense only with regard to volatile, variable-
length record files. With fixed-length records, placement is simply not
an issue.
■ If space is lost djie to internal fragmentation, the choice is between first
fit and best fit. A worst-fit strategy truly makes internal fragmentation
worse.
■ If the space is lost due to external fragmentation, one should give careful
consideration to a worst-fit strategy.
222 Chapters Organizing Files for Performance

6.3 Finding Things Quickly: An Introduction to


Internal Sorting and Binary Searching

This text begins with a discussion of the cost of accessing secondary stor age.
You may remember that the magnitude of the difference between accessing
memory and seeking information on a fixed disk is such that, if we magnify
the time for a memory.access to twenty seconds, a similarly magnified disk
access would take fifty-eight days.
So far we have not had to pay much attention to this cost. This section;
then, marks a kind of turning point. Once we move from fundamental
organizational issues to the matter of searching a file for a particular piece of
information, the cost of a seek becomes a major factor in determining our
approach. And what is true for searching is all the more true for sorting. If you
have studied sorting algorithms, you know that even a good sort involves
making many comparisons. If each of these comparisons involves a seek, the
sort is agonizingly slow.
Our discussion of sorting and searching, then, goes beyond simply getting
the job done. We develop approaches that minimize the number of disk
accesses and therefore minimize the amount of time expended. This concern
with minimizing the number of seeks continues to be a major focus throughout
the rest of this text. This is just the beginning of a quest for ways to order and
find things quickly.

6.3.1 Finding Things in Simple Field and Record Files


All of the programs we have written up to this point, despite any other
strengths they offer, share a major failing: the only way to retrieve or find a
record with any degree of rapidity is to look for it by relative record number. If
the file has fixed-length records, knowing the RRN lets us compute the
record’s byte offset and jump to it using direct access.
But what if we do not know the byte offset or RRN of the record we want?
How likely is it that a question about this file would take the form, “What is
the record stored in RRN 23?” Not very likely, of course. We are much more
likely to know the identity of a record by its key, and the question is more
likely to take the form, “What is the record for jane Kelly?”
Given the methods of organization developed so far, access by key
implies a sequential search. What if there is no record containing the requested
key? Then we would have to look through the entire file.'What if we suspect
that there might be more than one record that contains the key,
Finding Things Quickly: An Introduction to internal Sorting and Binary Searching 223

and we want to find them all? Once again, we would be doomed to looking at
every record in the file. Clearly, we need to find a better way to handle keyed
access. Fortunately, there are many better ways.

6.3.2 Search by Guessing: Binary Search


Suppose we are looking for a record for Jane Kelly in a file of one thousand
fixed-length records, and suppose the file is sorted so the records appear in
ascending order by key. We start by comparing KELLY JANE (the canonical
form of the search key) with the middle key in the file, which is the key whose
RRN is 500. The result of the comparison tells us which half of the file
contains Jane Kelly’s record. Next, we compare KELLY JANE with the
middle key among records in the selected half of the file to find out which
quarter of the file Jane Kelly’s record is in. This process is repeated until either
Jane Kelly’s record is found or we have narrowed the number of potential
records to zero.
This kind of searching is called binary searching. An algorithm for binary
searching on a file of fixed-sized records is shown in Fig. 6.13. Binary
searching takes at most ten comparisons—to find Jane Kelly’s record if it is in
the file, or to determine that it is not in the file. Compare this with a sequential
search for the record. If there are one thousand records, then it takes at most
one thousand comparisons to find a given record (or establish that it is not
present); on the average, five hundred comparisons are needed.
We refer to the code in Fig. 6.13 as an algorithm, not a function, even
though it is given in the form of a C++ function. This is because this is not a
full implementation of binary search. Details of the implementation of the
method are not given. From the code, we can infer that there must be a class
FixedRecordFile that has methods NumRecs and ReadByRRN and that those
methods have certain specific meaning. In particular, NumRecs must return the
number of records in the FixedRecordFile, and ReadByRRN must read the
record at a specific RRN and unpack it into a RecType object.
It is reasonable to suppose that a full implementation of binary search
would be a template function with parameters for the type of the data record
and the type of the key. It might also be a method of a fixed-record file class.
Changing these details will not affect the algorithm and might not even
require changes in the code. We do know, however, that in order to perform
binary search, we must be able to read the file by relative record number, we
must have assignment and key extraction methods on the data record type,
and we must have relational operations on the key type.
224 Chapter 6 Organizing Files for Performance

int BinarySearch
(FixedRecordFile & file, RecType & obj, KeyType & key)
// binary search for key
// if key found, obj contains corresponding record', 1 returned // if key
not found, 0 returned
{ ' ■ • int low = 0; int high = file.NumRecs()-1; while (low <= high)
{
int guess = (high - low) / 2; file.ReadByRRN (obj, guess);
if (obj.Key() == key) return 1; // record found
if (obj'. Key () < key) high = guess - 1;// search before guess
else low = guess + 1;// search after guess
}
return 0; // loop ended without finding key

Figure 6.13 A binary search algorithm.

Figure 6.14 gives the minimum definitions that must be present to allow a
successful compilation of BinarySearch. This includes a class RecType with
a Key method that returns the key value of an object and class KeyType with
equality and less-than operators. No further details of any of these classes
need be given.

class KeyType {public:


int operator == (KeyType &) ,- // equality operator int operator
< (KeyType &); // less than operator
};
class RecType. (public:. KeyType Key {) ;} ;

class FixedRecordFile {public:


int NumRecs();
int ReadByRRN (RecType & record, int RRN);
};

Figure 6.14 Classes and methods that must be implemented to support the
binary search algorithm.
Finding Things Quickly: An Introduction to Internal Sorting and Binary Searching 225

This style of algorithm presentation is the object-oriented replacement


for the pseudocode approach, which has been widely used to describe
algorithms. Pseudocode is typically used to describe an algorithm without
including all of the details of implementation. In Fig. 6.13, we have been able
to present the algorithm without all of the details but in a form that can be
passed through a compiler to verify that it is syntactically correct and conforms
in its use of its related objects. The contrast between object-oriented design and
pseudocode is that the object-oriented approach uses a specific syntax and a
specific interface. The object- oriented approach is no harder to write but has
significantly more.detail.

6.3.3 Binary Search versus Sequential Search


In general, a binary search of a file with n records takes at most
l_log2 nj + 1 comparisons and on average approximately
Llog2 nj + 1/2 comparisons.
A binary search is therefore said to be 0(log2 n). In contrast, you may recall
that a sequential search of the same file requires at most n comparisons, and on
average ?(tk?) n, which is to say that a sequential search is O(n).
The difference between a binary search and a sequential search becomes
even more dramatic as we increase the size of the file to be searched. If we
double the number of records in the file, we double the number of comparisons
required for sequential search; when binary search is used, doubling the file
size adds only one more guess to our worst case. This makes sense, since we
know that each guess eliminates half of the possible choices. So, if we tried to
find Jane Kelly’s record in a file of two thousand records, it would take at most
1 + |_log2 2000J = 11 comparisons
whereas a sequential search would average
1/2 n = 1000 comparisons
and could take up to two thousand comparisons.
Binary searching is clearly .a more attractive way to find things than
sequential searching. But, as you might expect, there is a price to be paid before we can
use binary searching: it works only when the list of records is ordered in terms
of the key we are using in the search. So, to make use of binary searching, we
have to be able to sort a list on the basis of a key.
226 Chapter s Organizing Files for Performance

Sorting is a very important part of file processing. Next, we will look at


some simple approaches to sorting files in memory, at the same time
introducing some important new concepts in file structure design. We take a
second look at sorting in Chapter 8, when we deal with some tough problems
that occur when files are too large to sort in memory.

6.3.4 Sorting a Disk File in Memory


Consider the operation of any internal sorting algorithm with which you are
familiar. The algorithm requires multiple passes over the list that is to- be
sorted, comparing and reorganizing the elements. Some of the items in the list
are moved a long distance from their original positions in the list. If such an
algorithm were applied directly to data stored on a disk, it is clear that there
would be a lot of jumping around, seeking, and rereading of data. This would
be a very slow operation—-unthinkably slow.
If the entire contents of the file can be held in memory, a very attractive
alternative is to read the entire file from the disk into memory and then do the
sorting there, using an internal sort. We still have to-access the data on the
disk, but this way we can access it sequentially, sector after sector, without
having to incur the costs of a lot of seeking and of multiple, passes over the
disk.
This is one instance of a general class of solutions to the problem of
minimizing disk usage: force your disk access into a sequential mode,
performing the more complex, direct accesses in memory.
Unfortunately, it is often not possible to use this simple kind of solution,
but when you can, you should take advantage of it. In the case of sorting,
internal sorts are increasingly viable'as the amount of memory space grows. A
good illustration of an internal sort is the Unix sort utility, which sorts files in
memory if it can find enough space. This utility is described in Chapter 8.

6.3.5 The Limitations of Binary Searching and


Internal Sorting
Let’s look at three problems associated with our “sort, then binary search”
approach to finding things.

Problem 1: Binary Searching Requires More Than One or Two Accesses


In the average case, a binary search requires approximately |_log 2 n\ + 1/2
comparisons. If each comparison requires a disk access, a'series of binary
Finding Things Quickly: An Introduction to Internal Sorting and Binary Searching 227

searches on a list of one thousand items requires, on the average, 9.5 accesses
per request. If the list is expanded to one hundred thousand items, the average
search length extends to 16.5 accesses. Although this is a tremendous
improvement over the cost of a sequential search for the key, it is also true that
16 accesses, or even 9 or 10 accesses, is not a negligible cost. The cost of this
searching is particularly noticeable and objectionable, if we are doing a large
enough number of repeated accesses by key.
When we access records by relative record number rather than by key, we
are able to retrieve a record with a single access. That is an order of magnitude
of improvement over the ten or more accesses that binary, searching requires
with even a moderately large file. Ideally, we would like to approach RRN
retrieval performance while still maintaining the advantages of access by key.
In the following chapter, on the use of index structures, we begin to look at
ways to move toward this ideal.

Problem 2: Keeping a File Sorted Is Very Expensive


Our ability to use a binary search has a price attached to it: we must keep the
file in sorted order by key. Suppose we are working with a file to which . we
add records as often as we search for existing records. If we leave the file in
unsorted order, conducting sequential searches for records, then on average
each search requires reading through half the file. Each record addition,
however, is very fast, since it involves nothing more than jumping to the end of
the file and writing a record.
If, as an alternative, we keep the file in sorted order, we can cut down
substantially on the cost of searching, reducing it to a handful of accesses. But
we encounter difficulty when we add a record, since we want to keep all the
records in sorted order. Inserting a new record into the file requires, on average,
that we not only read through half the records, but that we also shift the records
to open up the space required for the insertion. We are actually doing more
work than if we simply do sequential searches on an unsorted file.
The costs of maintaining a file that can be accessed through binary
searching are not always as large as in this example involving frequent record
addition. For example, it is often the case that searching is required much more
frequently than record addition. In such a circumstance, the benefits of faster
retrieval can more than offset the costs of keeping the file sorted. As another
example, there are many applications in which record additions can be
accumulated in a transaction file and made in a batch mode. By sorting the list
of new records before adding them to the main file, it is possible to merge them
with the existing records. As we see in
228 Chapter 6 Organizing Files for Performance

Chapter 8, such merging is a sequential process, passing only once over each
record in the file. This can be an efficient, attractive approach to maintaining
the file.
So, despite its problems, there are situations in which binary searching
appears to be a useful strategy. However, knowing the costs of binary searching
also lets us see better solutions to the problem of finding things by key. Better
solutions will have to meet at least one of the following conditions:
■ They will not involve reordering of the records in the file when a new
record is added, and
■ They will be associated with data structures that allow for substantially
more rapid, efficient reordering of the file.
In the chapters that follow we develop approaches that fall into each of
these categories. Solutions of the first type can involve the use of simple
indexes. They can also involve hashing. Solutions of the second type can
involve the use of tree structures, such as a B-tree, to keep the file in order.

Problem 3: An Internal Sort Works Only on Small Files


Our ability to use binary searching is limited by our ability to sort the file. An
internal sort works only if we can read the entire contents of a file into the
computer’s electronic memory. If the file is so large that we cannot do that, we
need a different kind of sort. ■
In the following section we develop a variation on internal sorting called a
keysort. Like internal sorting, keysort is limited, in terms of how large a file it
can sort, but its limit is larger. More important, our work on keysort begins to
illuminate a new approach to the problem of finding things that will allow us to
avoid the sorting of records in a file.

6.4 Keysorting

Keysort, sometimes referred to as tag sort, is based on the idea that when we
sort a file in memory the only things that we really need to sort are the record
keys; therefore, we do not need to read the whole file into memory during the
sorting process. Instead, we read the keys from the file into memory, sort them,
and then rearrange the records in the file according to the new ordering of the
keys.
Keysorting 229

Since keysort never reads the complete set of records into memory, it can
sort larger files than a.regular internal sort, given the same amount of memory.

6.4.1 Description of the Method


To keep things simple, we assume that we are dealing, with a fixed-length
record file of the kind developed in Chapter 4, with a count of the number of
records stored in a header record.
We present the algorithm in an object-oriented pseudocode. As in Section
6.3.3, we need to identify the supporting object classes. The file class.
(FixedRecordFile) must support methods NurhRecs and ReadByRRN. In
order to store the key RRN pairs from the file, we need a class Key RRN that
has two data members, KEY and RRN. Figure 6.15 gives the minimal
functionality required by these classes.
The algorithm begins by reading the key RRN pairs into an array of
KeyRRN objects. We call this array KEYNODES [ ]. Figure 6.16 illustrates
the relationship between the array KEYNODES [ ] and the actual file at the

class FixedRecordFile {public:


int NumRecs();
int ReadByRRN (RecTvpe & record, int RRN);
// additional methods required for keysort int
Create (char * fileName); int Append (RecType &
record);
};

class KeyRRN
// contains a pair (KEY, RRN)
(public:
KeyType KEY; int RRN;
KeyRRN () ;
KeyRRN (KeyType key, int rrn);
};
int Sort (KeyRRN [], int numKeys); // sort array by key

Figure 6.15 Minimal functionality required for classes used by the keysort algorithm.
230 Chapter 6 Organizing Files for Performance

Figure 6.16 Conceptual view of KEYNODES array to be used in memory by internal sort
routine and record array on secondary store.

time the keysort procedure begins. The RRN field of each array element
contains the RRN of the record associated with the corresponding key.
The actual sorting process simply sorts the KEYNODES [ ] array according to
the KEYfield. This produces an arrangement like that shown in Fig. 6.17. The
elements of KEYNODES [ ] are now sequenced in such a way that the first
element has the RRN of the record that should be moved to the first position
.in the file, the second element identifies the record that should be second, and
so forth.
Once KEYNODES [ ] is sorted, we are ready to reorganize the file according
to this new ordering by reading the records from the input file and writing to a
new file in the order of the KEYNODES []'array.
Figure 6.18 gives an algorithm for keysort. This algorithm works much
the same way that a normal internal sort would work, but with two important
differences:
■ Rather than read an entire record into a memory array, we simply read each
record into a temporary buffer, extract the key, then discard it; and
IS When we are writing the records out in sorted order, we have to read them in
a second time, since they are not all stored in memory.
Keysorting 231

int KeySort (FixedRecordFile & 'inFile, char * outFileName)


{
RecType obj; •
KeyRRN * KEYNODES '= new KeyRRN [inFile . NumRecsO];
// read file and load Keys
for (int i = 0; i < inFile NumRecs () ; i++)
{
inFile . ReadByRRN (obj, i);// read record i
KEYNODES[i] = KeyRRN(obj.Key(),i);//put key and RRN into Keys
}
Sort (KEYNODES, inFile . NumRecs());//-sort Keys
FixedRecordFile outFile;// file to hold records in key .order
outFile . Create (outFileName);// create a new file
// write new file in key order •
for (int j = 0; j < inFile . NumRecsO; j++)
{
inFile . ReadByRRN (obj, KEYNODES[j].RRN);//read in key order outFile .
Append (obj);// write in key order } :
return 1; . ■ '

Figure 6.18 Algorithm for keysort


232 Chapter 6 Organizing Files for Performance

6.4.2 Limitations of the Keysort Method


At first glance, keysorting appears to be an obvious improvement over sorting
performed entirely in memory; it might even appear to be a case of getting
something for nothing. We know that sorting is an expensive operation and
that we want to do it in memory. Keysorting allows us to achieve this
objective without having to hold the entire file in memory at once.
But, while reading about the operation of writing the records out in sorted
order, even a casual reader probably senses a cloud on this apparently bright
horizon. In keysort we need to read in the recdrds a second time before we can
write out the new sorted file. Doing something twice is never desirable. But
the problem is worse than that.
Look carefully at the for loop that reads in the records before writing them
out to the new file. You can see that we are not reading through the input file
sequentially. Instead, we are working in sorted order, moving from the sorted
KEYNODES[] to the RRNs of the records. Since we have to seek to each
record and read it in before writing it back out, creating the sorted file requires
as many random seeks into the input file as there are records. As we have
noted a number of times, there is an enormous difference between the time
required to read all the records in a file sequentially and the time required to
read those same records if we must seek to each record separately. What, is
worse, we are performing all of these accesses in alternation with write
statements to the output file. So, even the writing of the output file, which
would otherwise appear to be sequential, involves seeking in most cases. The
disk drive must move the head back and forth between the two files as it reads
and writes.
The getting-something-for-nothing aspect of keysort has suddenly
evaporated. Even though keysort does the hard work of sorting in memo ry, it
turns out that creating a sorted version of the file from the map supplied by the
KEYNODES[] array is not at all a trivial matter when the only copies of the
records are kept on secondary store.

6.4.3 Another Solution: Why Bother to Write the File Back?


The idea behind keysort is an attractive one: why work with an entire record
when the only parts of interest, as far as sorting and searching are concerned,
are the fields used to form the key? There is a compelling parsimony behind
this idea, and it makes keysorting look promising. The promise fades only
when we run into the problem of rearranging all the records in the file so they
!
reflect the new, sorted order.
Keysorting 233

It is interesting to ask whether we can avoid this problem by simply not


bothering with the task that is giving us trouble. What if we just skip the
time-consuming business of writing out a sorted version of the file? What if,
instead, we simply write out a copy of the array of canonical key nodes? If we
do without writing the records back in sorted order, writing out the contents of
our KEYNODES[] array instead, we will have written a program that outputs
an index to the original file. -The relationship between the two files is
illustrated in Fig. 6.19.
This is an instance of one of our favorite categories of solutions to
computer science problems: if some part of a process begins to look like a
bottleneck, consider skipping it altogether. Ask if you'can do without it.
Instead of creating a new, sorted copy of the file to use for searching, we have
created a second kind of file, an index file, that is to be used in conjunction
with the original file. If we are looking for a particular record, we do our binary
search on the index file and then use the RRN stored in the index file record to
find the corresponding record in the original file.
There is much to say about the use of index files, enough to fill several
chapters. The next chapter is about the various ways we can use a simple index,
which is the kind of index we illustrate here. In later chapters we talk about
different ways of organizing the index to provide more flexible access and
easier maintenance.

Index file Original file

Figure 6.19 Relationship between the index file and the data file.
Chapter 7

Indexing
7 .1 What Is an Index?

The last few pages of many books contain an index. Such an index is a table
containing a list of topics (keys) and numbers of pages where the topics can be
found (reference fields).
All indexes are based on the same basic concept—keys and reference
fields. The types of indexes we examine in this chapter are called simple
indexes because they are represented using simple arrays of structures that
contain the keys and reference fields. In later chapters we look at indexing
schemes that use more complex data structures, especially trees. In this
chapter, however, we want to emphasize that indexes can be very simple and
still provide powerful tools for file processing.
The index to a book provides a way to find a topic quickly. If you have
ever had to use a book that doesn’t have a good index, you already know, that
an index is a desirable alternative to scanning through the book, sequentially to
find a topic. In general, indexing is another way to handle the problem we
explored in Chapter 6: an index is a way to find things.
Consider what would happen if we tried to apply the previous chapter’s
methods, sorting and binary searching, to the problem of finding things in a
book. Rearranging all the wcyds in the book so they were in
alphabetical order certainly would make finding any particular term easier but would
obviously have disastrous effects on the meaning of the book. In a sense, the terms in the
book are pinned records. This is an absurd example, but it clearly underscores the power and
importance of the index as a conceptual tool. Since it works by indirection, an index lets
you impose order on a file, without rearranging thefde. This not only keeps us from
disturbing pinned records, but also makes matters such as record addition much less
expensive than they are with a sorted file.
Take, as another example, the problem of finding books in a library. We
want to be able to locate books by a specific author, title, or subject area. One
way of achieving this is to have three copies of each book and three separate
library buildings. All of the books in one building would be sorted by author’s
name, another building would contain books arranged by title, and the third
would have them ordered by subject. Again, this is an absurd example, but
one that underscores another important advantage of indexing. Instead of
using multiple arrangements, a library uses a card catalog. The card catalog is
actually a set of three indexes, each using a different key field, and all of them
using the same catalog number as a reference field. Another use of indexing,
then, is to provide multiple access paths to a file.
We also find that indexing gives us keyed access to variable-length
record files. Let’s begin our discussion of indexing by exploring this problem
of access to variable-length records and the simple solution that indexing
provides.
One final note: the example data objects used in the following sections
are musical recordings. This may cause some fconfusion as we use the term
record to refer to an object in a file, and recording to refer to a data object. We
will see how to get information about recordings by finding records in files.
We’ve tried hard to make a distinction between these two terms: The
distinction is between the file system view of the elements that make up files
(records), and the user’s or application’s view of the objects that are being
manipulated (recordings).

7.2 A Simple Index for Entry-Sequenced Files

Suppose we own an extensive collection of musical recordings, and we want


to keep track of the collection through the use of computer files. For each
recording, we keep the information shown in Fig. 7.1. Appendix G includes
files recording. h and recording. cpp that define class
250 Chapter 7 Indexing

Identification number
Title
Composer or composers Artist
or artists Label (publisher)

Figure 7.1 Contents of a data


record.

Recording. Program makerec.cpp in Appendix G uses classes.


DelimFieldBuffer and BufferFile to create the file of Recording objects
displayed in Fig. 7.2. The first column of the table contains the record
addresses associated with each record in the file.
_ Suppose we formed a primary key for these recordings consisting of the
initials for the company label combined with the recording’s ID number. This
will make a good primary key as it should provide a unique key for each entry
in the file. We call this key the Label ID. The canonical form for the Label ID
consists of the uppercase form of the Label field followed immediately by the
ASCII-representation of the ID number. For example,
LON2312

Record ID
address number

Label Title Composer(s) Artist(s)

17 LON 2312 . Romeo and Juliet Prokofiev Maazel


RCA Beethoven Julliard
62 2626 • Quartet in C Sharp
Minor
117 WAR 23699 Touchstone Corea Corea
152 ANG ' 3795 Symphony No. 9 Beethoven Giulini
196 COL 38358 Nebraska Springsteen Springsteen
241 DG 18807 Symphony No. 9 Beethoven Karajan
285 MER 75016 Coq d’Or Suite Rimsky-Korsakov Leinsdorf
33 8 COL 31809 Symphony No. 9 Dvorak Bernstein

382 DG 139201 Violin Concerto Beethoven Ferras


427 FF 245 Good News Sweet Honey in the Sweet Honey • in
Rock the Rock

Figure 7.2 Contents of sample recording file.


A Simple Index for Entry-Sequenced Files 251

How could we organize the file to provide rapid keyed access to indi-
vidual records? Could we sort the file and then use binary searching?
Unfortunately, binary searching depends on being able to jump to the middle
record in the file. This is not possible in a variable-length record file because
direct access by relative record number is not possible; there is no way to
know where the middle record is in any group of records.
An alternative to sorting is to construct an index for the file. Figure 7.3
illustrates such an index. On the right is the data file containing information
about our collection of recordings, with one variable-length data record per
recording. Only four fields are shown (Label, ID number, Title, and
Composer), but it is easy to imagine the other information filling out each
record.
On the left is the index, each entry of which contains a key corresponding
to a certain Label ID in the data file. Each key is associated with a reference
field giving the address of the first byte of the corresponding data record.
ANG3795, for example, corresponds to the reference field containing the
number 152, meaning that the record containing full information on the
recording with Label ID ANG3795 can be found starting at byte number 152
in the record file.

Index Recording file


Referen Address of
ce
Key record Actual data record
field
LON 1 2312.1 Romeo and Juliet 1 Prokofiev 1...
ANG3795 152 17
RCA 1 26261 Quartet in C Sharp Minor 1 Beethoven 1...
COB 1809 338 62

WAR 1 23699 1 Touchstone ! Corea 1...


COL38358 196 117
ANG 1 3795 1 Symphony No. 9 1 Beethoven 1...
DG139201 382 152
COL 1 38358 1 Nebraska 1 Springsteen 1...
DG18807 241 196

241 DG 1 18807 1 Symphony No. 9 1 Beethoven 1...


FF245 427
285 MER 1 75016 1 Coq d’Or Suite 1 Rimsky-Korsakov 1...
LON2312 17
338 COL 1 31809 1 Symphony No. 9 1 Dvorak 1...
MER75016 285 -
■ 382 DG 1 139201 1 Violin Concerto 1 Beethoven 1...
RCA2626 62

427 FF 1 245 1 Good News ISweet Honey in the Rock 1...


WAR23699 117
Figure 7.3 Index of the sample recording file.
The structure of the index object is very simple. It is a list of pairs of fields:
a key field and a byte-offset field. There is one entry in the index for each
record in the data file. Class Text Index of Fig. 7.4 encapsulates the index data
and index operations. The full implementation of class Text Index is given in
files textind.h and textind.cpp of Appendix G. An index is implemented with
arrays to hold the keys and record references. Each object is declared with a
maximum number of • entries and can be used for unique keys (no duplicates)
and for nonunique keys (duplicates allowed). The methods Insert and Search
do most of the work of indexing. The protected method Find locates the element
key and returns its index. If the key is not in the index, Find returns -1. This
method is used by Insert, Remove, and Search.
A C++ feature used in this class is the destructor, method -TextIndex. This
method is automatically called whenever a Text Index object is deleted, either
because of the return from a function that includes the declaration of a
TextIndex object or because of explicit deletion of an object created
dynamically with new. The role of the destructor is to clean up the object,
especially when it has dynamically created data members. In the case of class
Text Index, the protected members Keys and RecAddrs are created
dynamically by the constructor and should be deleted by the destructor to avoid
an obvious memory leak:
Text Index::-Text Index () (delete Keys.; delete RecAddrs;}

class Textlndex (public:


Textlndex (int maxKeys = 100, int unique = 1); int Insert (const char * key,
int recAddr); // add to index int Remove (const char * key); // remove key
from index int Search (const char * key) const;
// search for key, return recaddr void
Print (ostream &) const; protected:
int MaxKeys; // maximum number of entries int NumKeys; // actual number of
entries char * * Keys; / / array, of key values int.* RecAddrs; // array
of record references int Find (const char * key) const; int Init (int maxKeys,
int unique);
int Unique; // if true, each key must be unique in the index

Figure 7.4 Class Textlndex.


Note also that' the index is sorted, whereas the data file is not.
Consequently, although Label ID ANG3795 is the first entry in the index, it is
not necessarily the first entry in the data file. In fact, the data file is entry
sequenced, which means that' the records occur in the order they are entered
into the file. As we will see, the use of an entry-sequenced file can make
record addition and file maintenance much simpler than the case with a data
file that is kept sorted by some key.
Using the index to provide access to the data file by Label ID is a simple
matter. The code to use our classes to retrieve a single record by key from a
recording file is shown in the function RetrieveRecording:
int RetrieveRecording (Recording & recording, char * key,
Textlndex & Recordinglndex, BufferFile.k RecordingFile)
// read and unpack the recording, return TRUE if succeeds ( int result;
result = RecordingFile .,Read (Recordinglndex.Searchfkey) ) ;
if (result == -1) return FALSE;
result = recording.Unpack (RecordingFile.GetBuffer());
. return 'result;

With an open file and an index to the file in memory, RetrieveRecording puts
together the index search, file read, and buffer unpack operations into a single
function.
Keeping the index in memory as the program runs also lets us find
records by key more quickly with an indexed file than with a sorted one since
the binary searching can be performed entirely in memory. Once the byte
offset for the data record is found, a single seek is all that is required to
retrieve the record. The use of a sorted data file, on the other hand, requires a
seek for each step of the binary search.

7.3 Using Template Classes in C++ for Object I/O

A good object-oriented design for a file of objects should provide operations


to read and write data objects without having to go through the intermediate
step of packing and unpacking buffers. In Chapter 4, we supported I/O for
data with the buffer classes and class Buf ferFile. In order to provide I/O for
objects, we added Pack and Unpack methods to our Person object class. This
approach gives us the required functionality but <
Chapter 7 Indexing
254

stops short of providing a read operation whose arguments are a file and a data
object. We want a class RecordFile that makes the following code possible:
Person p; RecordFile pFile; pFile . Read (p);
Recording r; RecordFile rFile; rFile .Read (r);

The major difficulty with defining class RecordFile is making it possible


to support files for different record types without having to modify the class.
Is it possible that class RecordFile can support read and unpack for a Person
and a Recording without change? Certainly the objects are different; they
have different unpacking methods. Virtual function calls do not help because
Person and Recording do not have a common base type. It seems that class
RecordFile needs to be parameterized so different versions of the class can be
constructed for different types of data objects.
It i$ the C++ template feature that supports parameterized function and
class definitions, and RecordFile is a template class. As shown in Fig. 7.5,
class RecordFile includes the parameter RecType, which is used as the
argument type for the read and write methods of the class. Class RecordFile is
derived from Buff erFile, which provides most of the functionality. The
constructor for RecordFile is given inline and' simply calls the Buf f erFile
constructor.
The definitions of pFile and rFile just given are not consistent with use of
a template class. The actual declarations and calls are:
RecordFile <Person> pFile; pFile . Read (p);
RecordFile <Recording> rFile; rFile . Read (p);

template <class RecTvpe>


class RecordFile: public BufferFile
{public: i
int Read (RecType! & record, int recaddr = -1);
int Write (const RecType & record, int recaddr = -1);
int Append (const RecType & record);
RecordFile (IOBuffer & buffer): BufferFile (buffer) {}
};
// The template parameter RecType must have the following methods // int Pack
(IOBuffer &); pack record into buffer // int Unpack (IOBuffer &); unpack record
from buffer ■

Figure 7.5 Template Class RecordFile.


Object-Oriented Support for Indexed, Entry-Sequenced Files of Data Objects 255

Object rFile is of type RecordFile<Recording>, which is an instance of


class RecordFile. The call to rFile . Read looks the same as the call
to pFile . Read, and the two methods share the same source code, but
the implementations of the classes are somewhat different. In
particular, the Pack and Unpack methods of class Recording are used
for methods of object rFile, but Person methods are used for pFile.
The implementation of method Read of class RecordFile is given in Fig.
7.6; the implementation of all the methods are in file reef ile . h in Appendix
G. The method makes use of the Read method of Buf ferFile and the Unpack
method of the parameter RecType. A new version of RecordFile : : Read is
created by the C++ compiler for each instance of RecordFile. The call rFile
. Read (r) calls Recording :: Unpack, and the call pFile . Read (p) calls
Person: : Unpack.
Class RecordFile accomplishes the goal of providing object- oriented I/O
for data. Adding I/O to an existing class (class Recording,, for example)
requires three steps:
1. Add methods Pack.and Unpack to class Recording.
2. Create a buffer object to use in the I/O:
DelimFieldBuffer Buffer;
3. Declare an object of type RecordFile<Recording>:
RecordFile<Recording> rFile (Buffer);
Now we can directly open a file and read and write objects of class
Recording:
Recording rl, r2; rFile . Open
("myfile"); rFile . Read
,(rl) ; rFile . Write (r2);

7.4 Object-Oriented Support for Indexed,


Entry-Sequenced Files of Data Objects

Continuing with our object-oriented approach to I/O, we will add indexed


access to the sequential access provided by class RecordFile. A new class,
IndexedF.ile, extends RecordFile with Update and
256 Chapter 7 Indexing

template <class RecType> ,


int RecordFile<RecType>::Read (RecType & record, int recaddr) { ' int
writeAddr, result;
writeAddr = BufferFile::Read (recaddr); if
(IwriteAddr) return -1;
result = record . Unpack (Buffer)-; //RecType: :Unpack if
(Iresult) return -1; return writeAddr;
}

Figure 7.6 implementation of RecordFile::Read.

Append methods that maintain a primary key index of the data file and a Read
method that supports access to object by key.
So far, we have classes Text Index, which supports maintenance and
search by primary key, and RecordFile, which supports create, open, and close
for files as well as read and write for data objects. We have already seen how
to create a primary key index for a data file as a memory object. There are still
two issues to address:
■ How to make a persistent index of a file. That is, how to store the index in
a file when it is not in memory.
■ How to guarantee that the index is an accurate reflection of the contents
of the data file.

7.4.1 Operations Required to Maintain an Indexed File


The support and maintenance of an entry-sequenced file coupled with a
simple index requires the operations to handle a number of different tasks.
Besides the RetrieveRecording function described previously, other
operations used to find things by means of the index include the following:
■ Create the original empty index and data files,
■ Load the index file into memory before using it,
■ Rewrite the index file from memory after using it,
■ Add data records to the data file,
■ Delete records from the data file,
Object-Oriented Support for Indexed, Entry-Sequenced Files of Data Objects 257

■ Update records in the data file, and


■ Update the index to reflect changes in the data file.
A great benefit of our object-oriented approach is that’everything we need to
implement these operations is already available in the methods of our
classes. We just need to glue them together. We begin by identifying the
methods required for each of these operations. We continue to use class
Recording as our example data class.

Creating the Files


Two files must be created: a data file to hold the data objects and an index
file to hold the primary key index. Both the index file and the data file are
created as empty files, with header records and nothing else. This can be
accomplished quite easily using the Create method implemented in class
BufferFile. The data file is represented by an object of class
RecordFile<Recording>. The index file is a BufferFile of
fixed-size records, as described below. As an example of the manipulation of
index files, program makeind. cpp of Appendix G creates an index file
from a file of recordings. •

Loading the Index into Memory


Both loading (reading) and storing (writing) objects is supported in the IOBuf
f er classes. With these buffers, we can make files of index objects. For this
example, we are storing the full index in a single object, so our index file needs
only one record. As our use of indexes develops in the rest of the book, we
will make extensive use of multiple record index files. .
We need to choose a particular buffer class to use for our index file. We
define class TextlndexBuffer as a derived class of FixedFieldBuffer to
support reading and writing of index objects. TextlndexBuf fer. includes pack
and unpack methods for index objects. This style is an alternative 1 to adding
these methods to the data class, which in this case is Text IndexBuf f er. The
full implementation of class TextlndexBuf fer is in files tindbuff.h and
tindbuff. cpp in Appendix G.

Rewriting the Index File from Memory


As part of the Close operation on an IndexedFile, the index in memory needs
to be written to the index file. This is accomplished using the Rewind and
Write operations of class BufferFile.
258 Chapter 7 Indexing

It is important to consider what happens if this rewriting of the index


does not take place or if it takes place incompletely. Programs do not always
run to completion. A program designer needs to guard against power failures,
the operator turning the machine off at the wrong time, and other such
disasters. One of the dangers associated with reading an index into memory
and then writing it out when the program is over is that the copy of the index
on disk will be out of date and incorrect if the program is interrupted. It is
imperative that a program contain at least the following two safeguards to
protect against this kind of error:
■ There should be a mechanism that permits the program to know' when
the index is out of date. One possibility involves setting a status flag as
soon as the copy of the index in memory is changed. This status flag
could be written into the header record of the index file on disk as soon as
the index is read into memory and subsequently cleared when the index
is rewritten. All programs could check the status flag before using an
index. If the flag is found to be set, the program would know that the
index is out of date.
■ If a program detects that an index is out of date, the program must
have access to a procedure that reconstructs the index from the data file.
This should happen automatically and take place before any attempt is
made to use the index. j

Record Addition

Adding a new record to the data file requires that we also add an entry to the
index. Adding to the data file itself uses RecordFile<Recording> : : Write.
The record key and the result-' ing record reference are then inserted into the
index record using Text Index. Insert.-
Since the index is kept in sorted order by key, insertion of the new index
entry probably requires some rearrangement of the index. In a way, the
situation is similar to the one we face as we add records to a sorted data file.
We have to shift or slide all the entries with keys that come in order after the
key of the record we are inserting. The shifting opens up a space for the new
entry. The big difference between the work we have to do on the index entries
and the work required for a sorted data file is that the index is contained
wholly in memory. All of the index rearrangement can be done without any
file access.. The implementation of Text Index: : Insert is given in file text
ind. cpp of Appendix G.
Object-Oriented Support for Indexed, Entry-Sequenced Files of Data Objects 259

Record Deletion
In Chapter 6 we described a number of approaches to deleting records in
variable-length record files that allow for the'reuse of the space occupied by
these records. These approaches are completely viable for our data file
because, unlike a sorted data file, the records in this file need not be moved
around to maintain an ordering on the file. This is one of the great advantages
of an indexed file organization: we have rapid access to individual records by
key without disturbing pinned records. In fact, the indexing itself pins all the
records. The implementation of data record deletion is not included in this
text but has been left as exercises.
Of course, when we delete a record from the data file, we must also delete
the corresponding entry from our index, using Text Index: : Delete. Since the
index is in memory during program execution, deleting the index entry and
shifting the other entries to close up the space may not be an overly expensive
operation. Alternatively, we could simply mark the index entry as deleted,
just as we might mark the corresponding data record. Again, see text ind. cpp
for the implementation of Text Index: .-Delete.

Record Updating
Record updating falls into two categories:
■ The update changes the value of the key field. This kind of update can
bring about a reordering of the index file as well as the data file.
Conceptually, the easiest way to think of this kind of change is as a
deletion followed by an insertion. This delete/insert approach can be
implemented while still providing the program user with the view that he
or she is merely changing a record..
■ The update does not affect the key field. This second kind, of update does
not require rearrangement of the index file but may well involve
reordering of the data file. If the record size is unchanged or decreased by
the update, the record can be written directly into its old space. But if the
record size is increased by the update, a new slot for the record will have
to be found. In the latter case the starting address of the rewritten record
must replace the old address in the corresponding RecAddrs element.
Again, the delete/insert approach to maintaining the index can be used. It
is also possible to implement an operation simply to change the
RecAddrs member.
260 Chapter 7 Indexing

268 Chapter 7 Indexing

Figure 7.10
Secondary key index
organized by recording
title. -

Record Deletion
Deleting a record usually implies removing all references to that record in the
file system. So removing a record from the data file would mean removing not
only the corresponding entry in the primary index but also all of the entries in
the secondary indexes that refer to this primary index entry. The problem with
this is that secondary indexes, like the primary index, are maintained in sorted
order by key. Consequently, deleting an entry would involve rearranging the
remaining entries to close up the space left open by deletion.
This delete-ali-references approach would indeed be advisable if the
secondary index referenced the data file directly. If we did not delete the
secondary key references and if the secondary keys were associated with
actual byte offsets in the data file, it could be difficult to tell when these
references were no longer valid. This is another instance of the pinned- record
problem. The reference fields associated with the secondary keys would be
pointing to byte offsets that could, after deletion and subsequent space reuse in
the data file, be associated with different data records.
Indexing to Provide Access to Multiple Keys 269

But we have carefully avoided referencing actual addresses in the


secondary key index. After we search to find the secondary key we do another
search, this time on primary key. Since the primary index does reflect changes
due to record deletion, a search for the primary key of a record that has been
deleted will fail, returning a record-not-found condition. In a sense, the
updated primary key index acts as a kind of final check, protecting us from
trying to retrieve records that no longer exist.
Consequently, one option that is open to us when we delete a record from
the data file is to modify and rearrange only the primary key index. We ' could
safely leave intact the references to the deleted record that exist in the
secondary key indexes. Searches starting from a secondary key index that lead
to a deleted record are caught when we consult the primary key index.
If there are a number of secondary key indexes, the savings that results
from not having to rearrange all of these indexes when a record is deleted can
be substantial. This is especially important when the secondary key indexes are
kept on secondary storage. It is also important with an interactive system in
which the user is waiting at a terminal for the deletion operation to complete.
There is, of course, a cost associated with this shortcut: deleted records
take up space in the secondary index files. In a file system that undergoes few
deletions, this is not usually a problem. In a somewhat more volatile file
structure, it is possible to address the problem by periodically removing from
the secondary index files all entries that contain references that are no longer in
the primary index. If a file system is so volatile that even periodic purging is
not adequate, it is probably time to consider another index structure, such as
a.B-tree, that allows for deletion without having to rearrange a lot of records.

Record Updating
In our discussion of record deletion, we find that the primary key index serves
as a kind of protective buffer, insulating the secondary indexes from changes in
the data file. This insulation extends to record updating as well. If our
secondary indexes contain references directly to byte offsets in the data file,
then updates to the data file that result in changing a record’s physical location
in the file also require updating the secondary indexes. But, since we are
confining such detailed information to the primary index, data file updates
affect the secondary index only when they change either the primary or the
secondary key. There are three possible situations:
270 Chapter? Indexing

H Update changes the secondary key: if the secondary key is changed, we may
have to rearrange the secondary key index so it stays in sorted order. This
can be a relatively expensive operation.
■ Update changes the primary key: this kind of change has a large impact on
the primary key index but often requires that we update only the affected
reference field (Label ID in our example) in all the secondary indexes.
This involves searching the secondary indexes (on the unchanged
secondary keys) and rewriting the affected fixed-length field. It does not
require reordering of the secondary indexes unless the 1 corresponding
secondary key occurs more than once in the index; If a secondary key does
occur more than once, there may be some local reordering, since records
having the same secondary key are ordered by the reference field (primary
key).
■ Update confined to other fields: all updates that do not affect either the
primary or secondary key fields do not affect the secondary key index,
even if the update is substantial. Note that if there are several secondary
key indexes associated with a file, updates to records often affect only a
subset of the secondary indexes.

7.7 Retrieval Using Combinations of Secondary Keys \

One of the most important applications of secondary keys involves using two
or more of them in combination to retrieve special subsets of records from the
data file. To provide an example of how this can be done, we will extract another
secondary key ind'ex from our file of recordings. This one uses the recording’s
title as the key, as illustrated in Fig. 7.10. Now we can • respond to requests such
as
■ Find the recording with Label ID COL38358 (primary key access);
■ Find all the recordings of Beethoven’s work (secondary keyncompos- er);
and
■ Find all the recordings titled “Violin Concerto” (secondary keyntitle).
What is more interesting, however, is that we can also respond to a request
that combines retrieval on the composer index with retrieval on the title index,
such as: Find all recordings of Beethoven’s Symphony No. 9. Without the use
of secondary indexes, this kind of request requires a sequential search through
the entire file. Given a file containing thousands,
271
Retrieval Using Combinations of Secondary Keys

or even hundreds, of records, this is a very expensive process. But, with the
aid of seCondary.indexes, responding to this request is simple and quick.
We begin by recognizing that this request can be rephrased as a Boolean
and operation, specifying the intersection of two subsets of the data file:
Find all .data records with:
composer = 'BEETHOVEN' and title = 'SYMPHONY NO. 9'

We begin our response to this request by searching the composer • index for
the list of Label IDs that identify recordings with Beethoven as the composer. This
yields the following list of Label IDs:
ANG3795
DG139201
DG18807
RCA2626
Next we search the title index for the Label IDs associated with records
that have SYMPHONY NO. 9 as the title key:
ANG3795
COL31809
DG18807
Now we perform the Boolean and, which is a match operation, combining
the lists so only the members that appear in both lists are placed in the output
list.

We give careful attention to algorithms for performing this kind of match


operation in Chapter 8. Note that this kind of matching is much easier if the
lists that are being combined are in sorted order. That is the reason that, when
we have more than one entry for a given secondary key, the records are
ordered by the primary key reference fields.
Finally* once we have the list of primary keys occurring in both lists, we
can proceed-to the primary key index to look up the addresses of the data file
records. Then we can retrieve the records:
272 Chapter 7 Indexing

This is the kind of operation that makes computer-indexed file systems


useful in a way that far exceeds the capabilities of manual systems. We have
only one copy of each data file record, and yet, working through the secondary
indexes, we have multiple views of these records: we can look at them in order
by title, by composer, or by any other field that interests us. Using the
computer’s ability to combine sorted lists rapidly, we can even combine
different views, retrieving intersections (Beethoven and Symphony No. 9) dr
unions (Beethoven or Prokofiev or Symphony No. 9) of these views. And
since our data file is entry sequenced, we can do all of this without having to
sort data file records and can confine our sorting to the smaller index records
that can often be held in memory.
Now that we have a general idea of the design and uses of secondary
indexes, we can look at ways to improve these indexes so they take less space
and require less sorting.

7.8 Improving the Secondary Index Structure:


Inverted Lists

The secondary index structures that we have developed so far result in two
distinct difficulties:
■ We have to rearrange the index file every time a new record is added to the
file, even if the new record is for an existing secondary key. For example,
if we add another recording of Beethoven’s Symphony No. 9 to our
collection, both the composer and title indexes would have to be
rearranged, even though both indexes already contain entries for
secondary keys (but not the Label IDs) that are being added.
a If there are duplicate secondary keys, the secondary key field is repeated for
each entry. This wastes space because it makes the files larger than
necessary. Larger index files are less likely to fit in memory.

7.8.1 A First Attempt at a Solution


One simple response to these difficulties is to change the secondary index
structure so it associates an array of references with each secondary key.
Improving the Secondary Index Structure: Inverted Lists 273

For example, we might use a record structure that allows us to associate up to


four Label ID reference fields with a single secondary key, as in

Figure 7.11 provides a schematic example of how such an index would look if used with
our sample data file.
The major contribution of this revised index structure is, its help in solving our first
difficulty: the need to rearrange the secondary index file every time a new record is added
to the data file. Looking at Fig. 7.11, we can see that the addition of another recording of
a work by Prokofiev does not require the addition of another record to the index. For
example, if we add the recording

ANG. 36193 Piano Concertos 3 and 5 Prokofiev Francois


we need to modify only the corresponding secondary index record by
inserting a second Label ID:

Since we are not adding another record to the secondary index, there is no
need to rearrange any records. All that is required is a rearrangement of the
fields in the existing record for- Prokofiev.
Although this new structure helps avoid the need to rearrange the
secondary index file so often, it does have some problems. For one thing, it
provides space for only four Label IDs to be associated with a given key. In
the very likely case that more than four Label IDs will go with some key, we
need a mechanism for keeping track of the extra Label IDs.
A second problem has to do with space usage. Although the structure
does help avoid, the waste of space due to the repetition of identical keys, this
space savings comes at a potentially high cost. By extending the fixed length
of each of the secondary index records to hold more reference fields, we
might easily lose more space to internal fragmentation than we gained by not
repeating identical keys.
Since we don’t want to waste any more space than we have to, we need
to ask whether we can improve on this record structure. Ideally, what we
would like to do is develop a new design, a revision of our revision, that
274 Chapter 7 Indexing

Revised composer index


Secondary key Set of primary key references

BEETHOVEN ANG3795 DG139201 DG18807 RCA2626

COREA . WAR23699

DVORAK COO 1809

PROKOFIEV LON2312

RIMSKY-KORSAKOV MER75016

SPRINGSTEEN COL38358

SWEET HONEY IN THE R FF245

Figure 7.11 Secondary key index containing space for multiple references for each
secondary key.

■ Retains the attractive feature of not requiring reorganization of the


secondary indexes for every new entry to the data file;
■ Allows more than four Label IDs to be associated with each secondary .
key; and
■ Eliminates the waste of space due to internal fragmentation.

7.8.1 A Better Solution: Linking the List of References


Files such as our secondary indexes, in which a secondary key leads to a set of
one or more primary keys, are called inverted lists. The sense in which the list
is inverted should be clear if you consider that we are working our way
backward from a secondary key to the primary key to the record itself.
The second word in the term “inverted list” also tells us something
important: we are, in fact, dealing with a list of primary key references. Our
revised secondary index, which collects a number of Label IDs for each
secondary key, reflects- this list aspect of the data more directly than our initial
secondary index. Another way of conceiving of this list aspect of our inverted
list is illustrated in Fig. 7.12.
As Fig. 7.12 shows, an ideal situation would be to have each secondary
key point to a different list of primary key references. Each of these lists
Improving the Secondary Index Structure: Inverted Lists 275

Lists of primary

Figure 7.12 Conceptual view of the primary key reference fields as a series of lists.

could grow to be just as long as it needs to be. If we add the new Prokofiev
record, the .list of Prokofiev references becomes

Similarly, adding two new Beethoven recordings adds just two additional
elements to the list of references associated with the Beethoven key. Unlike our
record structure which allocates enough space for four Label IDs for each
secondary key, the lists could contain hundreds of references, if needed, while
still requiring only one instance of a secondary key. On the other, hand, if a list
requires only one element, then no space is lost to internal fragmentation..Most
important, we need to rearrange only the file of secondary keys if a new
composer is added to the file.
276 Chapter 7 Indexing

How can we set up an unbounded number of different lists, each of


varying length, without creating a large number of small files? The simplest
way is through the use of linked lists. We could redefine our secondary index
so it consists of records with two fields—a secondary key field and a field
containing the relative record number of the first corresponding pritnary key
reference (Label ID) in the inverted list. The actual primary key references
associated with each secondary key would be stored in a separate, entry-
sequenced file.
Given the sample data we have been working with, this new design would
result in a secondary key file for composers and an associated Label ID file
that are organized as illustrated in Fig. 7.13. Following the links for the list of
references associated with Beethoven helps us see how the Label ID List file
is organized. We begin, of course,, by searching the secondary key index of
composers for Beethoven. The record that we find points us to relative record
number (RRN) 3 in the Label ID List file. Since this is a fixed-length file, it is
easy to jump to RRN 3 and read in its Label ID

Figure 7.13 Secondary key index referencing linked


lists of primary key references.
(ANG3795). Associated with this Label ID is a link to a record with RRN
8. We read in the Label ID for that record, adding it to our list (ANG379
DG139201). We continue following links and collecting Label IDs until the
list looks like this:
ANG3795 DG139201 DG18807 RCA2626
The link field in the last record read from thi Label ID List file contains a
value of-1. As in our earlier programs, this indicates end-of-list, so we know
that we now have all the Label ID references for Beethoven records.
To illustrate how record addition affects the Secondary Index and Label
ID List files, we add the Prokofiev recording mentioned earlier:
ANG 36193 Piano Concertos 3 and 5 Prokofiev Francois

You can see (Fig. 7.13) that the Label ID for this new recording is the last
one in the Label ID List file, since this file is entry sequenced. Before this
record is added, there is only one Prokofiev recording. It has a Label ID of
LON2312. Since we want to keep the Label ID Lists in order by ASCII
character values, the new recording is inserted in the list for Prokofiev so it
logically precedes the LON2312 recording. .
Associating the Secondary Index file with a new file containing linked
lists of references provides some advantages over any of the structures
considered up to this point:
■ The only time we need to rearrange the Secondary Index file is when a
new composer’s name is added or an existing composer’s name is
changed (for example, it was misspelled.on input). Deleting or adding
recordings for a composer who is already in the index involves changing
only the Label ID List file. Deleting all the recordings for a composer
could be handled by modifying the Label ID List file while leaving the
entry in the Secondary Index file in place, using a value of -1 in its
reference field to indicate that the list of entries for this composer is empty.
■ In the event that we need to rearrange the Secondary Index file, the task is
quicker now since there are fewer records and each record is smaller.
■ Because there is less need for sorting, it follows that there is less of a /
penalty associated with keeping the Secondary Index files off on
secondary storage, leaving more room in memory for other data
structures.

■ The Label ID List file is entry sequenced. That means that it never needs
to be sorted.
■ Since the Label ID List file is a fixed-length record file, it would be very
easy to implement a mechanism for reusing the space from deleted
records, as described in Chapter 6.
There is also at least one potentially significant disadvantage to this kind of
file organization: the Label IDs associated with a given composer are no longer
guaranteed to be grouped together physically. The technical term for such
“togetherness” is locality. With a linked, entry-sequenced structure such as this, it is
less likely that there will be locality associated with the logical groupings of
reference fields for a given secondary key. Note, for example, that our list of Label
IDs for Prokofiev consists of the very last and the very first records in the file. This
lack of locality means that picking up the references for a composer with a long list
of references could involve a large amount of seeking back and forth on the disk.
Note that this kind of seeking would not be required for our original Secondary j
Index file structure. ... j
One obvious antidote to this seeking problem is to keep the Label ID i
List file in memory. This could be expensive and impractical, given many j
secondary indexes, except for the interesting possibility of using the same. |
Label ID List file to hold the lists for a number of Secondary Index files. j
Even if the file of reference lists were too large to hold in memory, it might j
be possible to obtain a performance improvement by holding only a part j
of the file in memory at a time, paging sections of the file in and out of
memory as they are needed.
Several exercises at the end of the chapter explore these possibilities
more thoroughly. These are very important problems, as the notion of dividing
the index into pages is fundamental to the design of B-trees and other methods
for handling large indexes on secondary storage.

7.9 Selective Indexes

Another interesting feature of secondary indexes is that they can be used to


divide a file into parts and provide a selective view. For example, it is possible
to build a selective index that contains only the titles of classical recordings in
the record collection. If we have additional information about the recordings
in the data file, such'as the date the recording was released, we could build
selective indexes such as “recordings released prior
280 Chapter 7 Indexing

up, associating the secondary keys with reference fields consisting of primary
keys allows the primary key index to act as a kind of final check of whether a
record is really in the file. The secondary indexes can afford to be wrong. This
situation is very different if the secondary index keys contain addresses. We
would then be jumping directly from the secondary key into the data file; the
address would need to be right.
This brings up a related safety,aspect: it is always more desirable to make
important changes in one place rather than in many places. With a
bind-at-retrieval-time scheme such as we developed, we need to remember to
make a change in only one place, the primary key index, if we move a data
record. With a more tightly bound system, we have to make many changes
successfully to keep the system internally consistent, braving power failures,
user interruptions, and so on.
When designing a new file system, it is better to deal with this question of
binding intentionally and early m the design process rather than letting the
binding just happen. In general, tight, in-the-data binding is most attractive
when
■ The data file is static or nearly so, requiring little or no adding, deleting, or
updating of records; and .
■ Rapid performance during actual retrieval is a high priority.
For example, tight binding is desirable for file organization on a mass-
produced, read-only optical disk. The addresses will never change because no
new records can ever be added; consequently, there is ho reason not to obtain
the extra performance associated with tight binding.
For file applications in which record addition, deletion, and updating do
occur, however, binding at retrieval time is usually the more desirable option.
Postponing binding as long as possible usually makes these operations simpler
and safer. If the file structures are carefully designed, and, in particular, if the
indexes use more sophisticated organizations such as 13- trees, retrieval
performance is usually quite acceptable, even given the additional work
required by a bind-at-retrieval system.

You might also like