Expert C Programming - Deep C Secrets

Expert C Programming: Deep C Secrets By Peter van der Linden

Introduction C code. C code run. Run code run…please! —Barbara Ling All C programs do the same thing: look at a character and do nothing with it. —Peter Weinberger Have you ever noticed that there are plenty of C books with suggestive names like C Traps and Pitfalls, or The C Puzzle Book, or Obfuscated C and Other Mysteries, but other programming languages don't have books like that? There's a very good reason for this! C programming is a craft that takes years to perfect. A reasonably sharp person can learn the basics of C quite quickly. But it takes much longer to master the nuances of the language and to write enough programs, and enough different programs, to become an expert. In natural language terms, this is the difference between being able to order a cup of coffee in Paris, and (on the Metro) being able to tell a native Parisienne where to get off. This book is an advanced text on the ANSI C programming language. It is intended for people who are already writing C programs, and who want to quickly pick up some of the insights and techniques of experts. Expert programmers build up a tool kit of techniques over the years; a grab-bag of idioms, code fragments, and deft skills. These are acquired slowly over time, learned from looking over the shoulders of more experienced colleagues, either directly or while maintaining code written by others. Other lessons in C are self-taught. Almost every beginning C programmer independently rediscovers the mistake of writing:

if (i=3) instead of:

if (i==3) Once experienced, this painful error (doing an assignment where comparison was intended) is rarely repeated. Some programmers have developed the habit of writing the literal first, like this: if (3==i). Then, if an equal sign is accidentally left out, the compiler will complain about an "attempted assignment to literal." This won't protect you when comparing two variables, but every little bit helps.

The $20 Million Bug In Spring 1993, in the Operating System development group at SunSoft, we had a "priority one" bug report come in describing a problem in the asynchronous I/O library. The bug was holding up the sale of $20 million worth of hardware to a customer who specifically needed the library functionality, so we were extremely motivated to find it. After some intensive debugging sessions, the problem was finally traced to a statement that read :

x==2; It was a typo for what was intended to be an assignment statement. The programmer 's finger had bounced on the "equals" key, accidentally pressing it twice instead of once. The statement as written compared x to 2, generated true or false, and discarded the result . C is enough of an expression language that the compiler did not complain about a statement which evaluated an expression, had no side-effects, and simply threw away the result. We didn't know whether to bless our good fortune at locating the problem, or cry with frustration at such a common typing error causing such an expensive problem. Some versions of the lint program would have detected this problem, but it's all too easy to avoid the automatic use of this essential tool. This book gathers together many other salutary stories. It records the wisdom of many experienced programmers, to save the reader from having to rediscover everything independently. It acts as a guide for territory that, while broadly familiar, still has some unexplored corners. There are extended discussions of major topics like declarations and arrays/pointers, along with a great many hints and mnemonics. The terminology of ANSI C is used throughout, along with translations into ordinary English where needed.

Programming Challenge

OR

Handy Heuristic

Sample Box Along the way, we have Programming Challenges outlined in boxes like this one. These are suggestions for programs that you should write. There are also Handy Heuristics in boxes of their own. These are ideas, rules-of-thumb, or guidelines that work in practice. You can adopt them as your own. Or you can ignore them if you already have your own guidelines that you like better.

Convention One convention that we have is to use the names of fruits and vegetables for variables (only in small code fragments, not in any real program, of course):

char pear[40]; double peach; int mango = 13; long melon = 2001; This makes it easy to tell what's a C reserved word, and what's a name the programmer supplied. Some people say that you can't compare apples and oranges, but why not—they are both hand-held round edible things that grow on trees. Once you get used to it, the fruit loops really seem to help. There is one other convention—sometimes we repeat a key point to emphasize it. In addition, we sometimes repeat a key point to emphasize it. Like a gourmet recipe book, Expert C Programming has a collection of tasty morsels ready for the reader to sample. Each chapter is divided into related but self-contained sections; it's equally easy to read the book serially from start to finish, or to dip into it at random and review an individual topic at length. The technical details are sprinkled with many true stories of how C programming works in practice. Humor is an important technique for mastering new material, so each chapter ends with a "light relief" section containing an amusing C story or piece of software folklore to give the reader a change of pace. Readers can use this book as a source of ideas, as a collection of C tips and idioms, or simply to learn more about ANSI C, from an experienced compiler writer. In sum, this book has a collection of useful ideas to help you master the fine art of ANSI C. It gathers all the information, hints, and guidelines together in one place and presents them for your enjoyment. So grab the back of an envelope, pull out your lucky coding pencil, settle back at a comfy terminal, and let the fun begin!

Some Light Relief—Tuning File Systems Some aspects of C and UNIX are occasionally quite lighthearted. There's nothing wrong with wellplaced whimsy. The IBM/Motorola/Apple PowerPC architecture has an E.I.E.I.O. instruction [1] that stands for "Enforce In-order Execution of I/O". In a similar spirit, there is a UNIX command,

tunefs, that sophisticated system administrators use to change the dynamic parameters of a filesystem and improve the block layout on disk. [1]

Probably designed by some old farmer named McDonald.

The on-line manual pages of the original tunefs, like all Berkeley commands, ended with a "Bugs" section. In this case, it read:

Bugs: This program should work on mounted and active file systems, but it doesn't. Because the superblock is not kept in the buffer cache, the program will only take effect if it is run on dismounted file systems; if run on the root file system, the system must be rebooted. You can tune a file system, but you can't tune a fish. Even better, the word-processor source had a comment in it, threatening anyone who removed that last phrase! It said:

Take this out and a UNIX Demon will dog your steps from now until the time_t's wrap around. When Sun, along with the rest of the world, changed to SVr4 UNIX, we lost this gem. The SVr4 manpages don't have a "Bugs" section—they renamed it "Notes" (does that fool anyone?). The "tuna fish" phrase disappeared, and the guilty party is probably being dogged by a UNIX demon to this day. Preferably lpd.

Programming Challenge

Computer Dating When will the time_t's wrap around? Write a program to find out. 1. Look at the definition of time_t. This is in file /usr/include/time.h. 2. Code a program to place the highest value into a variable of type time_t, then pass it to ctime() to convert it into an ASCII string. Print the string. Note that ctime has nothing to do with the language C, it just means "convert time."

For how many years into the future does the anonymous technical writer who removed the comment have to worry about being dogged by a UNIX daemon? Amend your program to find out. 1. Obtain the current time by calling time(). 2. Call difftime() to obtain the number of seconds between now and the highest value of time_t. 3. Format that value into years, months, weeks, days, hours, and minutes. Print it. Is it longer than your expected lifetime?

Programming Solution

Computer Dating The results of this exercise will vary between PCs and UNIX systems, and will depend on the way time_t is stored. On Sun systems, this is just a typedef for long. Our first attempted solution is

#include <stdio.h> #include int main() { time_t biggest = 0x7FFFFFFF; printf("biggest = %s \n", ctime(&biggest) ); return 0; } This gives a result of:

biggest = Mon Jan 18 19:14:07 2038 However, this is not the correct answer! The function ctime() converts its argument into local time, which will vary from Coordinated Universal Time (also known as Greenwich Mean Time), depending on where you are on the globe. California, where this book was written, is eight hours behind London, and several years ahead. We should really use the gmtime() function to obtain the largest UTC time value. This function doesn't return a printable string, so we call asctime()to get this. Putting it all

together, our revised program is

#include <stdio.h> #include int main() { time_t biggest = 0x7FFFFFFF;

}

printf("biggest = %s \n", asctime(gmtime(&biggest)) ); return 0;

This gives a result of:

biggest = Tue Jan 19 03:14:07 2038 There! Squeezed another eight hours out of it! But we're still not done. If you use the locale for New Zealand, you can get 13 more hours, assuming they use daylight savings time in the year 2038. They are on DST in January because they are in the southern hemisphere. New Zealand, because of its easternmost position with respect to time zones, holds the unhappy distinction of being the first country to encounter bugs triggered by particular dates. Even simple-looking things can sometimes have a surprising twist in software. And anyone who thinks programming dates is easy to get right the first time probably hasn't done much of it.

Chapter 1. C Through the Mists of Time C is quirky, flawed, and an enormous success. —Dennis Ritchie the prehistory of C…the golden rule for compiler-writers… early experiences with C…the standard I/O library and C preprocessor…K&R C…the present day: ANSI C…it's nice, but is it standard?…the structure of the ANSI C standard…reading the ANSI C standard for fun, pleasure, and profit…how quiet is a "quiet change"?…some light relief—the implementation-defined effects of pragmas

The Prehistory of C The story of C begins, paradoxically, with a failure. In 1969 the great Multics project—a joint venture between General Electric, MIT, and Bell Laboratories to build an operating system—was clearly in trouble. It was not only failing to deliver the promised fast and convenient on-line system, it was failing to deliver anything usable at all. Though the development team eventually got Multics creaking into action, they had fallen into the same tarpit that caught IBM with OS/360. They were trying to create an operating system that was much too big and to do it on hardware that was much too small.

Multics is a treasure house of solved engineering problems, but it also paved the way for C to show that small is beautiful. As the disenchanted Bell Labs staff withdrew from the Multics project, they looked around for other tasks. One researcher, Ken Thompson, was keen to work on another operating system, and made several proposals (all declined) to Bell management. While waiting on official approval, Thompson and co-worker Dennis Ritchie amused themselves porting Thompson's "Space Travel" software to a little-used PDP-7. Space Travel simulated the major bodies of the solar system, and displayed them on a graphics screen along with a space craft that could be piloted and landed on the various planets. At the same time, Thompson worked intensively on providing the PDP-7 with the rudiments of a new operating system, much simpler and lighter-weight than Multics. Everything was written in assembler language. Brian Kernighan coined the name "UNIX" in 1970, parodying the lessons now learned from Multics on what not to do. Figure 1-1 charts early C, UNIX, and associated hardware. Figure 1-1. Early C, UNIX, and Associated Hardware

In this potential chicken-and-egg situation, UNIX definitely came well before C (and it's also why UNIX system time is measured in seconds since January 1, 1970—that's when time began). However, this is the story not of poultry, but of programming. Writing in assembler proved awkward; it took longer to code data structures, and it was harder to debug and understand. Thompson wanted the advantages of a high-level implementation language, but without the PL/I [1] performance and complexity problems that he had seen on Multics. After a brief and unsuccessful flirtation with Fortran, Thompson created the language B by simplifying the research language BCPL [2] so its interpreter would fit in the PDP-7's 8K word memory. B was never really successful; the hardware memory limits only provided room for an interpreter, not a compiler. The resulting slow performance prevented B from being used for systems programming of UNIX itself. [1] The difficulties involved in learning, using, and implementing PL/I led one programmer to pen this verse: IBM had a PL/I / Its syntax worse than JOSS / And everywhere this language went / It was a total loss. JOSS was an earlier language, also not noted for simplicity. [2]

"BCPL: A Tool for Compiler Writing and System Programming," Martin Richards, Proc. AFIPS Spring Joint Computer Conference, 34 (1969), pp. 557-566. BCPL is not an acronym for the "Before C Programming

Language", though the name is a happy coincidence. It is the "Basic Combined Programming Language"— "basic" in the sense of "no frills"—and it was developed by a combined effort of researchers at London University and Cambridge University in England. A BCPL implementation was available on Multics.

Software Dogma

The Golden Rule of Compiler-Writers: Performance Is (almost) Everything. Performance is almost everything in a compiler. There are other concerns: meaningful error messages, good documentation, and product support. These factors pale in comparison with the importance users place on raw speed. Compiler performance has two aspects: runtime performance (how fast the code runs) and compile time performance (how long it takes to generate code). Runtime performance usually dominates, except in development and student environments. Many compiler optimizations cause longer compilation times but make run times much shorter. Other optimizations (such as dead code elimination, or omitting runtime checks) speed up both compile time and run time, as well as reducing memory use. The downside of aggressive optimization is the risk that invalid results may not be flagged. Optimizers are very careful only to do safe transformations, but programmers can trigger bad results by writing invalid code (e.g., referencing outside an array's bounds because they "know" that the desired variable is adjacent). This is why performance is almost but not quite everything—if you don't get accurate results, then it's immaterial how fast you get them. Compiler-writers usually provide compiler options so each programmer can choose the desired optimizations. B's lack of success, until Dennis Ritchie created a high-performance compiled version called "New B," illustrates the golden rule for compiler-writers. B simplified BCPL by omitting some features (such as nested procedures and some looping constructs) and carried forward the idea that array references should "decompose" into pointer-plusoffset references. B also retained the typelessness of BCPL; the only operand was a machine word. Thompson conceived the ++ and -- operators and added them to the B compiler on the PDP-7. The popular and captivating belief that they're in C because the PDP-11 featured corresponding autoincrement/decrement addressing modes is wrong! Auto increment and decrement predate the PDP-11 hardware, though it is true that the C statement to copy a character in a string:

*p++ = *s++; can be compiled particularly efficiently into the PDP-11 code:

movb (r0)+,(r1)+ leading some people to wrongly conclude that the former was created especially for the latter. A typeless language proved to be unworkable when development switched in 1970 to the newly introduced PDP-11. This processor featured hardware support for datatypes of several different sizes, and the B language had no way to express this. Performance was also a problem, leading Thompson to reimplement the OS in PDP-11 assembler rather than B. Dennis Ritchie capitalized on the more powerful PDP-11 to create "New B," which solved both problems, multiple datatypes, and performance. "New B"—the name quickly evolved to "C"—was compiled rather than interpreted, and it introduced a type system, with each variable described in advance of use.

Early Experiences with C The type system was added primarily to help the compiler-writer distinguish floats, doubles, and characters from words on the new PDP-11 hardware. This contrasts with languages like Pascal, where the purpose of the type system is to protect the programmer by restricting the valid operations on a data item. With its different philosophy, C rejects strong typing and permits the programmer to make assignments between objects of different types if desired. The type system was almost an afterthought, never rigorously evaluated or extensively tested for usability. To this day, many C programmers believe that "strong typing" just means pounding extra hard on the keyboard. Many other features, besides the type system, were put in C for the C compiler-writer's benefit (and why not, since C compiler-writers were the chief customers for the first few years). Features of C that seem to have evolved with the compiler-writer in mind are: •

•

•

•

•

Arrays start at 0 rather than 1. Most people start counting at 1, rather than zero. Compilerwriters start with zero because we're used to thinking in terms of offsets. This is sometimes tough on non-compiler-writers; although a[100] appears in the definition of an array, you'd better not store any data at a[100], since a[0] to a[99] is the extent of the array. The fundamental C types map directly onto underlying hardware. There is no built-in complex-number type, as in Fortran, for example. The compiler-writer does not have to invest any effort in supporting semantics that are not directly provided by the hardware. C didn't support floating-point numbers until the underlying hardware provided it. The auto keyword is apparently useless. It is only meaningful to a compiler-writer making an entry in a symbol table—it says this storage is automatically allocated on entering the block (as opposed to global static allocation, or dynamic allocation on the heap). Auto is irrelevant to other programmers, since you get it by default. Array names in expressions "decay" into pointers. It simplifies things to treat arrays as pointers. We don't need a complicated mechanism to treat them as a composite object, or suffer the inefficiency of copying everything when passing them to a function. But don't make the mistake of thinking arrays and pointers are always equivalent; more about this in Chapter 4. Floating-point expressions were expanded to double-length-precision everywhere. Although this is no longer true in ANSI C, originally real number constants were always doubles, and float variables were always converted to double in all expressions. The reason, though we've never seen it appear in print, had to do with PDP-11 floating-point hardware. First, conversion from float to double on a PDP-11 or a VAX is really cheap: just append an extra word of zeros. To convert back, just ignore the second word. Then understand that some PDP-11 floating-point hardware had a mode bit, so it would do either all single-precision or all double-precision arithmetic, but to switch between the two you had to change modes.

Since most early UNIX programs weren't floating-point-intensive, it was easier to put the box in double-precision mode and leave it there than for the compiler-writer to try to keep track of it! • •

No nested functions (functions contained inside other functions). This simplifies the compiler and slightly speeds up the runtime organization of C programs. The exact mechanism is described in Chapter 6, "Poetry in Motion: Runtime Data Structures." The register keyword. This keyword gave the compiler-writer a clue about what variables the programmer thought were "hot" (frequently referenced), and hence could usefully be kept in registers. It turns out to be a mistake. You get better code if the compiler does the work of allocating registers for individual uses of a variable, rather than reserving them for its entire lifetime at declaration. Having a register keyword simplifies the compiler by transferring this burden to the programmer.

There were plenty of other C features invented for the convenience of the C compiler-writer, too. Of itself this is not necessarily a bad thing; it greatly simplified the language, and by shunning complicated semantics (e.g., generics or tasking in Ada; string handling in PL/I; templates or multiple inheritance in C++) it made C much easier to learn and to implement, and gave faster performance. Unlike most other programming languages, C had a lengthy evolution and grew through many intermediate shapes before reaching its present form. It has evolved through years of practical use into a language that is tried and tested. The first C compiler appeared circa 1972, over 20 years ago now. As the underlying UNIX system grew in popularity, so C was carried with it. Its emphasis on lowlevel operations that were directly supported by the hardware brought speed and portability, in turn helping to spread UNIX in a benign cycle.

The Standard I/O Library and C Preprocessor The functionality left out of the C compiler had to show up somewhere; in C's case it appears at runtime, either in application code or in the runtime library. In many other languages, the compiler plants code to call runtime support implicitly, so the programmer does not need to worry about it, but almost all the routines in the C library must be explicitly called. In C (when needed) the programmer must, for example, manage dynamic memory use, program variable-size arrays, test array bounds, and carry out range checks for him or herself. Similarly, I/O was originally not defined within C; instead it was provided by library routines, which in practice have become a standardized facility. The portable I/O library was written by Mike Lesk [3] and first appeared around 1972 on all three existing hardware platforms. Practical experience showed that performance wasn't up to expectations, so the library was tuned and slimmed down to become the standard I/O library. [3]

It was Michael who later expressed the hilariously ironic rule of thumb that "designing the system so that the manual will be as short as possible minimizes learning effort." (Datamation, November 1981, p.146). Several comments come to mind, of which "Bwaa ha ha!" is probably the one that minimizes learning effort.

The C preprocessor, also added about this time at the suggestion of Alan Snyder, fulfilled three main purposes: •

String replacement, of the form "change all foo to baz", often to provide a symbolic name for a constant.

•

•

Source file inclusion (as pioneered in BCPL). Common declarations could be separated out into a header file, and made available to a range of source files. Though the ".h" convention was adopted for the extension of header files, unhappily no convention arose for relating the header file to the object library that contained the corresponding code. Expansion of general code templates. Unlike a function, the same macro argument can take different types on successive calls (macro actual arguments are just slotted unchanged into the output). This feature was added later than the first two, and sits a little awkwardly on C. White space makes a big difference to this kind of macro expansion.

#define a(y) a_expanded(y) a(x); expands into:

a_expanded(x); However,

#define a (y) a_expanded (y) a(x); is transformed into: (y) a_expanded (y)(x); Not even close to being the same thing. The macro processor could conceivably use curly braces like the rest of C to indicate tokens grouped in a block, but it does not. There's no extensive discussion of the C preprocessor here; this reflects the view that the only appropriate use of the preprocessor is for macros that don't require extensive discussion. C++ takes this a step further, introducing several conventions designed to make the preprocessor completely unnecessary.

Software Dogma

C Is Not Algol Writing the UNIX Version 7 shell (command interpreter) at Bell Labs in the late 1970's, Steve Bourne decided to use the C preprocessor to make C a little more like Algol-68. Earlier at Cambridge University in England, Steve had written an Algol-68 compiler, and found it easier to debug code that had explicit "end statement" cues, such as if ... fi or case ... esac. Steve thought it wasn't easy enough to tell by looking at a " }"

what it matches. Accordingly, he set up many preprocessor definitions:

#define #define #define #define #define #define #define #define #define #define #define

STRING char * IF if( THEN ){ ELSE } else { FI ;} WHILE while ( DO ){ OD ;} INT int BEGIN { END }

This enabled him to code the shell using code like this:

INT compare(s1, s2) STRING s1; STRING s2; BEGIN WHILE *s1++ == *s2 DO IF *s2++ == 0 THEN return(0); FI OD return(*--s1 - *s2); END Now let's look at that again, in C this time:

int compare(s1, s2) char * s1, *s2; { while (*s1++ == *s2) { if (*s2++ == 0) return (0); } return (*--s1 - *s2); } This Algol-68 dialect achieved legendary status as the Bourne shell permeated far beyond Bell Labs, and it vexed some C programmers. They complained that the dialect made it much harder for other people to maintain the code. The BSD 4.3 Bourne shell (kept in /bin/sh) is written in the Algol subset to this day! I've got a special reason to grouse about the Bourne shell—it's my desk that the bugs reported against it land on! Then I assign them to Sam! And we do see our share of bugs:

the shell doesn't use malloc, but rather does its own heap storage management using sbrk. Maintenance on software like this too often introduces a new bug for every two it solves. Steve explained that the custom memory allocator was done for efficiency in stringhandling, and that he never expected anyone except himself to see the code. The Bournegol C dialect actually inspired The International Obfuscated C Code Competition, a whimsical contest in which programmers try to outdo each other in inventing mysterious and confusing programs (more about this competition later). Macro use is best confined to naming literal constants, and providing shorthand for a few well-chosen constructs. Define the macro name all in capitals so that, in use, it's instantly clear it's not a function call. Shun any use of the C preprocessor that modifies the underlying language so that it's no longer C.

K&R C By the mid 1970's the language was recognizably the C we know and love today. Further refinements took place, mostly tidying up details (like allowing functions to return structure values) or extending the basic types to match new hardware (like adding the keywords unsigned and long). In 1978 Steve Johnson wrote pcc, the portable C compiler. The source was made available outside Bell Labs, and it was very widely ported, forming a common basis for an entire generation of C compilers. The evolutionary path up to the present day is shown in Figure 1-2. Figure 1-2. Later C

Software Dogma

An Unusual Bug One feature C inherited from Algol-68 was the assignment operator. This allows a repeated operand to be written once only instead of twice, giving a clue to the code generator that operand addressing can be similarly thrifty. An example of this is writing b+=3 as an abbreviation for b=b+3. Assignment operators were originally written with assignment first, not the operator, like this: b=+3. A quirk in B's lexical analyzer made it simpler to

implement as =op rather than op= as it is today. This form was confusing, as it was too easy to mix up

b=-3; /* subtract 3 from b */ and b= -3; /* assign -3 to b */ The feature was therefore changed to its present ordering. As part of the change, the code formatter indent was modified to recognize the obsolete form of assignment operator and swap it round to operator assignment. This was very bad judgement indeed; no formatter should ever change anything except the white space in a program. Unhappily, two things happened. The programmer introduced a bug, in that almost anything (that wasn't a variable) that appeared after an assignment was swapped in position. If you were "lucky" it would be something that would cause a syntax error, like

epsilon=.0001; being swapped into epsilon.=0001; But a source statement like

valve=!open; /* valve is set to logical negation of open */ would be silently transmogrified into valve!=open; /* valve is compared for inequality to open */ which compiled fine, but did not change the value of valve. The second thing that happened was that the bug lurked undetected. It was easy to work around by inserting a space after the assignment, so as the obsolete form of assignment operator declined in use, people just forgot that indent had been kludged up to "improve" it. The indent bug persisted in some implementations up until the mid-1980's. Highly pernicious! In 1978 the classic C bible, The C Programming Language, was published. By popular acclamation, honoring authors Brian Kernighan and Dennis Ritchie, the name "K&R C" was applied to this version of the language. The publisher estimated that about a thousand copies would be sold; to date (1994) the figure is over one and a half million (see Figure 1-3). C is one of the most successful programming languages of the last two decades, perhaps the most successful. But as the language spread, the temptation to diverge into dialects grew. Figure 1-3. Like Elvis, C is Everywhere

The Present Day: ANSI C By the early 1980's, C had become widely used throughout the industry, but with many different implementations and changes. The discovery by PC implementors of C's advantages over BASIC provided a fresh boost. Microsoft had an implementation for the IBM PC which introduced new keywords (far, near, etc.) to help pointers to cope with the irregular architecture of the Intel 80x86 chip. As many other non-pcc-based implementations arose, C threatened to go the way of BASIC and evolve into an ever-diverging family of loosely related languages. It was clear that a formal language standard was needed. Fortunately, there was much precedent in this area—all successful programming languages are eventually standardized. However, the problem with standards manuals is that they only make sense if you already know what they mean. If people write them in English, the more precise they try to be, the longer, duller and more obscure they become. If they write them using mathematical notation to define the language, the manuals become inaccessible to too many people. Over the years, the manuals that define programming language standards have become longer, but no easier to understand. The Algol-60 Reference Definition was only 18 pages long for a language of comparable complexity to C; Pascal was described in 35 pages. Kernighan and Ritchie took 40 pages for their original report on C; while this left several holes, it was adequate for many implementors. ANSI C is defined in a fat manual over 200 pages long. This book is, in part, a description of practical use that lightens and expands on the occasionally opaque text in the ANSI Standard document. In 1983 a C working group formed under the auspices of the American National Standards Institute. Most of the process revolved around identifying common features, but there were also changes and significant new features introduced. The far and near keywords were argued over at great length, but ultimately did not make it into the mildly UNIX-centric ANSI standard. Even though there are more than 50 million PC's out there, and it is by far the most widely used platform for C implementors, it was (rightly in our view) felt undesirable to mutate the language to cope with the limitations of one specific architecture.

Handy Heuristic

Which Version of C to Use? At this point, anyone learning or using C should be working with ANSI C, not K&R C. The language standard draft was finally adopted by ANSI in December 1989. The international standards organization ISO then adopted the ANSI C standard (unhappily removing the very useful "Rationale" section and making trivial—but very annoying—formatting and paragraph numbering changes). ISO, as an international body, is technically the senior organization, so early in 1990 ANSI readopted ISO C (again excluding the Rationale) back in place of its own version. In principle, therefore, we should say that the C standard adopted by ANSI is ISO C, and we should refer to the language as ISO C. The Rationale is a useful text that greatly helps in understanding the standard, and it's published as a separate document. [4] [4]

The ANSI C Rationale (only) is available for free by anonymous ftp from the site ftp.uu.net, in directory /doc/standards/ansi/X3.159-1989/. (If you're not familiar with anonymous ftp, run, don't walk, to your nearest bookstore and buy a book on Internet, before you become on the Information Highway.) The Rationale has also been published as a book, ANSI C Rationale, New Jersey, Silicon Press, 1990. The ANSI C standard itself is not available by ftp anywhere because ANSI derives an important part of its revenue from the sale of printed standards.

Handy Heuristic

Where to Get a Copy of the C Standard The official name of the standard for C is: ISO/IEC 9899-1990. ISO/IEC is the International Organization for Standardization International Electrotechnical Commission. The standards bodies sell it for around $130.00. In the U.S. you can get a copy of the standard by writing to: American National Standards Institute 11 West 42nd Street New York, NY 10036 Tel. (212) 642-4900

Outside the U.S. you can get a copy by writing to: ISO Sales Case postale 56 CH-1211 Genève 20 Switzerland Be sure to specify the English language edition. Another source is to purchase the book The Annotated ANSI C Standard by Herbert Schildt, (New York, Osborne McGraw-Hill, 1993). This contains a photographically reduced, but complete, copy of the standard. Two other advantages of the Schildt book are that at $39.95 it is less than one-third the price charged by the standards bodies, and it is available from your local bookstore which, unlike ANSI or ISO, has probably heard of the twentieth century, and will take phone orders using credit cards. In practice, the term "ANSI C" was widely used even before there was an ISO Working Group 14 dedicated to C. It is also appropriate, because the ISO working group left the technical development of the initial standard in the hands of ANSI committee X3J11. Toward the end, ISO WG14 and X3J11 collaborated to resolve technical issues and to ensure that the resulting standard was acceptable to both groups. In fact, there was a year's delay at the end, caused by amending the draft standard to cover international issues such as wide characters and locales. It remains ANSI C to anyone who has been following it for a few years. Having arrived at this good thing, everyone wanted to endorse the C standard. ANSI C is also a European standard (CEN 29899) and an X/Open standard. ANSI C was adopted as a Federal Information Processing Standard, FIPS 160, issued by the National Institute of Standards and Technology in March 1991, and updated on August 24, 1992. Work on C continues—there is talk of adding a complex number type to C.

It's Nice, but Is It Standard? Save a tree—disband an ISO working group today. —Anonymous The ANSI C standard is unique in several interesting ways. It defines the following terms, describing characteristics of an implementation. A knowledge of these terms will aid in understanding what is and isn't acceptable in the language. The first two are concerned with unportable code; the next two deal with bad code; and the last two are about portable code.

Unportable Code: implementation-defined— The compiler-writer chooses what happens, and has to document it. Example: whether the sign bit is propagated, when shifting an int right. unspecified— The behavior for something correct, on which the standard does not impose any requirements. Example: the order of argument evaluation.

Bad Code: undefined— The behavior for something incorrect, on which the standard does not impose any requirements. Anything is allowed to happen, from nothing, to a warning message to program termination, to CPU meltdown, to launching nuclear missiles (assuming you have the correct hardware option installed). Example: what happens when a signed integer overflows. a constraint— This is a restriction or requirement that must be obeyed. If you don't, your program behavior becomes undefined in the sense above. Now here's an amazing thing: it's easy to tell if something is a constraint or not, because each topic in the standard has a subparagraph labelled "Constraints" that lists them all. Now here's an even more amazing thing: the standard specifies [5] that compilers only have to produce error messages for violations of syntax and constraints! This means that any semantic rule that's not in a constraints subsection can be broken, and since the behavior is undefined, the compiler is free to do anything and doesn't even have to warn you about it! [5]

In paragraph 5.1.1.3, "Diagnostics", if you must know. Being a language standard, it doesn't say something simple like you've got to flag at least one error in an incorrect program. It says something grander that looks like it was drawn up by a team of corporate lawyers being paid by the word, namely, a conforming implementation shall [*] produce at least one diagnostic message (identified in an implementation-dependent manner) for every translation unit that contains a violation of any syntax rule or constraint. Diagnostic messages need not be produced in other circumstances. [*]

Useful rule from Brian Scearce [ standard. [

]

]

—if you hear a programmer say "shall" he or she is quoting from a

Inventor of the nested footnote.

Example: the operands of the % operator must have integral type. So using a non-integral type with % must cause a diagnostic. Example of a rule that is not a constraint: all identifiers declared in the C standard header files are reserved for the implementation, so you may not declare a function called malloc() because a standard header file already has a function of that name. But since this is not a constraint, the rule can be broken, and the compiler doesn't have to warn you! More about this in the section on "interpositioning" in Chapter 5.

Software Dogma

Undefined Behavior Causes CPU Meltdown in IBM PC's! The suggestion of undefined software behavior causing CPU meltdown isn't as farfetched as it first appears.

The original IBM PC monitor operated at a horizontal scan rate provided by the video controller chip. The flyback transformer (the gadget that produces the high voltage needed to accelerate the electrons to light up the phosphors on the monitor) relied on this being a reasonable frequency. However, it was possible, in software, to set the video chip scan rate to zero, thus feeding a constant voltage into the primary side of the transformer. It then acted as a resistor, and dissipated its power as heat rather than transforming it up onto the screen. This burned the monitor out in seconds. Voilà: undefined software behavior causes system meltdown!

Portable Code: strictly-conforming— A strictly-conforming program is one that: • • •

only uses specified features. doesn't exceed any implementation-defined limit. has no output that depends on implementation-defined, unspecified, or undefined features.

This was intended to describe maximally portable programs, which will always produce the identical output whatever they are run on. In fact, it is not a very interesting class because it is so small compared to the universe of conforming programs. For example, the following program is not strictly conforming:

#include #include <stdio.h> int main() { (void) printf("biggest int is %d", INT_MAX); return 0;} /* not strictly conforming: implementation-defined output! */ For the rest of this book, we usually don't try to make the example programs be strictly conforming. It clutters up the text, and makes it harder to see the specific point under discussion. Program portability is valuable, so you should always put the necessary casts, return values, and so on in your real-world code. conforming— A conforming program can depend on the nonportable features of an implementation. So a program is conforming with respect to a specific implementation, and the same program may be nonconforming using a different compiler. It can have extensions, but not extensions that alter the behavior of a strictly-conforming program. This rule is not a constraint, however, so don't expect the compiler to warn you about violations that render your program nonconforming! The program example above is conforming.

Translation Limits The ANSI C standard actually specifies lower limits on the sizes of programs that must successfully translate. These are specified in paragraph 5.2.4.1. Most languages say how many characters can be in a dataname, and some languages stipulate what limit is acceptable for the maximum number of array dimensions. But specifying lower bounds on the sizes of various other features is unusual, not to say

unique in a programming language standard. Members of the standardization committee have commented that it was meant to guide the choice of minimum acceptable sizes. Every ANSI C compiler is required to support at least: • • • • •

31 parameters in a function definition 31 arguments in a function call 509 characters in a source line 32 levels of nested parentheses in an expression The maximum value of long int can't be any less than 2,147,483,647, (i.e., long integers are at least 32 bits).

and so on. Furthermore, a conforming compiler must compile and execute a program in which all of the limits are tested at once. A surprising thing is that these "required" limits are not actually constraints—so a compiler can choke on them without issuing an error message. Compiler limits are usually a "quality of implementation" issue; their inclusion in ANSI C is an implicit acknowledgment that it will be easier to port code if definite expectations for some capacities are set for all implementations. Of course, a really good implementation won't have any preset limits, just those imposed by external factors like available memory or disk. This can be done by using linked lists, or dynamically expanding the size of tables when necessary (a technique explained in Chapter 10).

The Structure of the ANSI C Standard It's instructive to make a quick diversion into the provenance and content of the ANSI C standard. The ANSI C standard has four main sections: Section 4: An introduction and definition of terminology (5 pages). Section 5: Environment (13 pages). This covers the system that surrounds and supports C, including what happens on program start-up, on termination, and with signals and floating-point operations. Translator lower limits and character set information are also given. Section 6: The C language (78 pages) This part of the standard is based on Dennis Ritchie's classic "The C Reference Manual" which appeared in several publications, including Appendix A of The C Programming Language. If you compare the Standard and the Appendix, you can see most headings are the same, and in the same order. The topics in the standard have a more rigid format, however, that looks like Figure 1-4 (empty subparagraphs are simply omitted). Figure 1-4. How a Paragraph in the ANSI C Standard Looks

The original Appendix is only 40 pages, while this section of the standard is twice as long. Section 7: The C runtime library (81 pages). This is a list of the library calls that a conforming implementation must provide—the standard services and routines to carry out essential or helpful functions. The ANSI C standard's section 7 on the C runtime library is based on the /usr/group 1984 standard, with the UNIX-specific parts removed. "/usr/group" started life as an international user group for UNIX. In 1989 it was renamed "UniForum", and is now a nonprofit trade association dedicated to the promotion of the UNIX operating system. UniForum's success in defining UNIX from a behavioral perspective encouraged many related initiatives, including the X/Open portability guides (version 4, XPG/4 came out in October 1992), IEEE POSIX 1003, the System V Interface Definition, and the ANSI C libraries. Everyone coordinated with the ANSI C working group to ensure that all their draft standards were mutually consistent. Thank heaven. The ANSI C standard also features some useful appendices: Appendix F: Common warning messages. Some popular situations for which diagnostic messages are not required, but when it is usually helpful to generate them nonetheless. Appendix G: Portability issues. Some general advice on portability, collected into one place from throughout the standard. It includes information on behavior that is unspecified, undefined, and implementation-defined.

Software Dogma

Standards Are Set in Concrete, Even the Mistakes Just because it's written down in an international standard doesn't mean that it's complete, consistent, or even correct. The IEEE POSIX 1003.1-1988 standard (it's an OS standard that defines UNIX-like behavior) has this fun contradiction: "[A pathname] ... consists of at most PATH_MAX bytes, including the terminating null character."—section 2.3 "PATH_MAX is the maximum number of bytes in a pathname (not a string length; count excludes a terminating null)."—section 2.9.5 So PATH_MAX bytes both includes and does not include the terminating null! An interpretation was requested, and the answer came back (IEEE Std 1003.1-1988/INT, 1992 Edition, Interpretation number: 15, p. 36) that it was an inconsistency and both can be right (which is pretty strange, since the whole point is that both can't be right). The problem arose because a change at the draft stage wasn't propagated to all occurrences of the wording. The standards process is formal and rigid, so it cannot be fixed until an update is approved by a balloting group. This kind of error also appears in the C standard in the very first footnote, which refers to the accompanying Rationale document. In fact, the Rationale no longer accompanies the C Standard—it was deleted when ownership of the standard moved to ISO.

Handy Heuristic

Differences between K&R C and ANSI C Rest assured that if you know K&R C, then you already know 90% of ANSI C. The differences between ANSI C and K&R C fall into four broad categories, listed below in order of importance: 1. The first category contains things that are new, very different, and important. The only feature in this class is the prototype—writing the parameter types as part of

the function declaration. Prototypes make it easy for a compiler to check function use with definition. 2. The second category is new keywords. Several keywords were officially added: enum for enumerated types (first seen in late versions of pcc), const, volatile, signed, void, along with their associated semantics. The neverused entry keyword that found its way into C, apparently by oversight, has been retired. 3. The third category is that of "quiet changes"—some feature that still compiles, but now has a slightly different meaning. There are many of these, but they are mostly not very important, and can be ignored until you push the boundaries and actually stumble across one of them. For example, now that the preprocessing rules are more tightly defined, there's a new rule that adjacent string literals are concatenated. 4. The final category is everything else, including things that were argued over interminably while the language was being standardized, but that you will almost certainly never encounter in practice, for example, token-pasting or trigraphs. (Trigraphs are a way to use three characters to express a single character that a particularly inadequate computer might not have in its character set. Just as the digraph \t represents "tab", so the trigraph ??< represents "open curly brace".)

The most important new feature was "prototypes", adopted from C++. Prototypes are an extension of function declarations so that not just the name and return type are known, but also all the parameter types, allowing the compiler to check for consistency between parameter use and declaration. "Prototype" is not a very descriptive term for "a function name with all its arguments"; it would have been more meaningful to call it a "function signature", or a "function specification" as Ada does.

Software Dogma

The Protocol of Prototypes The purpose of prototypes is to include some information on parameter types (rather than merely giving the function name and return value type) when we make a forward declaration of a function. The compiler can thus check the types of arguments in a function call against the way the parameters were defined. In K&R C, this check was deferred till link time or, more usually, omitted entirely. Instead of

char * strcpy(); declarations in header files now look like this: char * strcpy(char *dst, const char *src); You can also omit the names of the parameters, leaving only the types:

char * strcpy(char * , const char * ); Don't omit the parameter names. Although the compiler doesn't check these, they often convey extra semantic information to the programmer. Similarly, the definition of the function has changed from

char * strcpy(dst, src) char *dst, *src; { ... } to char * strcpy(char *dst, const char *src) /* note no semi-colon! */ { ... } Instead of being ended with a semicolon, the function header is now directly followed by a single compound statement comprising the body of the function. Prototype everything new you write and ensure the prototype is in scope for every call. Don't go back to prototype your old K&R code, unless you take into account the default type promotions—more about this in Chapter 8. Having all these different terms for the same thing can be a little mystifying. It's rather like the way drugs have at least three names: the chemical name, the manufacturer 's brand name, and the street name.

Reading the ANSI C Standard for Fun, Pleasure, and Profit Sometimes it takes considerable concentration to read the ANSI C Standard and obtain an answer from it. A sales engineer sent the following piece of code into the compiler group at Sun as a test case.

1 foo(const char **p) { } 2 3 main(int argc, char **argv) 4{ 5 foo(argv); 6} If you try compiling it, you'll notice that the compiler issues a warning message, saying:

line 5: warning: argument is incompatible with prototype The submitter of the code wanted to know why the warning message was generated, and what part of the ANSI C Standard mandated this. After all, he reasoned,

argument char *s matches parameter const char *p This is seen throughout all library string functions. So doesn't argument char **argv match parameter const char **p ? The answer is no, it does not. It took a little while to answer this question, and it's educational in more than one sense, to see the process of obtaining the answer. The analysis was carried out by one of Sun's "language lawyers," [6] and it runs like this: [6]

The New Hacker's Dictionary defines a language lawyer as "a person who will show you the five sentences scattered through a 200-plus-page manual that together imply the answer to your question 'if only you had thought to look there.'" Yep! That's exactly what happened in this case.

The Constraints portion of Section 6.3.2.2 of the ANSI C Standard includes the phrase: Each argument shall have a type such that its value may be assigned to an object with the unqualified version of the type of its corresponding parameter. This says that argument passing is supposed to behave like assignment. Thus, a diagnostic message must be produced unless an object of type const char ** may be assigned a value of type char **.To find out whether this assignment is legal, flip to the section on simple assignment, Section 6.3.16.1, which includes the following constraint: One of the following shall hold:… •

Both operands are pointers to qualified or unqualified versions of compatible types, and the type pointed to by the left has all the qualifiers of the type pointed to by the right.

It is this condition that makes a call with a char * argument corresponding to a const char * parameter legal (as seen throughout the string routines in the C library). This is legal because in the code

char * cp; const char *ccp; ccp = cp; • • •

The left operand is a pointer to "char qualified by const". The right operand is a pointer to "char" unqualified. The type char is a compatible type with char, and the type pointed to by the left operand has all the qualifiers of the type pointed to by the right operand (none), plus one of its own (const).

Note that the assignment cannot be made the other way around. Try it if you don't believe me.

cp = ccp; /* results in a compilation warning */

Does Section 6.3.16.1 also make a call with a char ** argument corresponding to a const

char ** parameter legal? It does not.

The Examples portion of Section 6.1.2.5 states: The type designated "const float *" is not a qualified type—its type is "pointer to const-qualified float" and is a pointer to a qualified type. Analogously, const char ** denotes a pointer to an unqualified type. Its type is a pointer to a pointer to a qualified type. Since the types char ** and const char ** are both pointers to unqualified types that are not the same type, they are not compatible types. Therefore, a call with an argument of type char ** corresponding to a parameter of type const char ** is not allowed. Therefore, the constraint given in Section 6.3.2.2 is violated, and a diagnostic message must be produced. This is a subtle point to grasp. Another way of looking at it is to note that: • •

the left operand has type FOO2—a pointer to FOO, where FOO is an unqualified pointer to a character qualified by the const qualifier, and the right operand has type BAZ2—a pointer to BAZ, where BAZ is an unqualified pointer to a character with no qualifiers.

FOO and BAZ are compatible types, but FOO2 and BAZ2 differ other than in qualifica-tion of the thing immediately pointed to and are therefore not compatible types; therefore the left and right operands are unqualified pointers to types that are not compatible. Compatibility of pointer types is not transitive. Therefore, the assignment or function call is not permitted. However, note that the restriction serves mainly to annoy and confuse users. The assignment is currently allowed in C++ translators based on cfront (though that might change).

Handy Heuristic

Const Isn't The keyword const doesn't turn a variable into a constant! A symbol with the const qualifier merely means that the symbol cannot be used for assignment. This makes the value re ad -onl y through that symbol; it does not prevent the value from being modified through some other means internal (or even external) to the program. It is pretty much useful only for qualifying a pointer parameter, to indicate that this function will not change the data that argument points to, but other functions may. This is perhaps the most common use of const in C and C++.

A const can be used for data, like so:

const int limit = 10; and it acts somewhat as in other languages. When you add pointers into the equation, things get a little rough:

const int * limitp = &limit; int i=27; limitp = &i; This says that limitp is a pointer to a constant integer. The pointer cannot be used to change the integer; however, the pointer itself can be given a different value at any time. It will then point to a different location and dereferencing it will yield a different value! The combination of const and * is usually only used to simulate call-by-value for array parameters. It says, "I am giving you a pointer to this thing, but you may not change it." This idiom is similar to the most frequent use of void *. Although that could theoretically be used in any number of circumstances, it's usually restricted to converting pointers from one type to another. Analogously, you can take the address of a constant variable, and, well, perhaps I had better not put ideas into people's heads. As Ken Thompson pointed out, "The const keyword only confuses library interfaces with the hope of catching some rare errors." In retrospect, the const keyword would have been better named readonly. True, this whole area in the standard appears to have been rendered into English from Urdu via Danish by translators who had only a passing familiarity with any of these tongues, but the standards committee was having such a good time that it seemed a pity to ruin their fun by asking for some simpler, clearer rules. We felt that a lot of people would have questions in the future, and not all of them would want to follow the process of reasoning shown above. So we changed the Sun ANSI C compiler to print out more information about what it found incompatible. The full message now says:

Line 6: warning: argument #1 is incompatible with prototype: prototype: pointer to pointer to const char : "barf.c", line 1 argument : pointer to pointer to char Even if a programmer doesn't understand why, he or she will now know what is incompatible.

How Quiet is a "Quiet Change"? Not all the changes in the standard stick out as much as prototypes. ANSI C made a number of other changes, usually aimed at making the language more reliable. For instance, the "usual arithmetic

conversions" changed between ye olde originale K&R C and ANSI C. Thus, where Kernighan and Ritchie say something like: Section 6.6 Arithmetic Conversions A great many operators cause conversions and yield result types in a similar way. This pattern will be called the "usual arithmetic conversions." First, any operands of type char or short are converted to int, and any of type float are converted to double. Then if either operand is double, the other is converted to double and that is the type of the result. Otherwise, if either operand is long, the other is converted to long and that is the type of the result. Otherwise, if either operand is unsigned, the other is converted to unsigned and that is the type of the result. Otherwise, both operands must be int, and that is the type of the result. The ANSI C manual has closed the loopholes by rewriting this as: Section 6.2.1.1 Characters and Integers (the integral promotions) A char, a short int, or an int bit-field, or their signed or unsigned varieties, or an enumeration type, may be used in an expression wherever an int or unsigned int may be used. If an int can represent all the values of the original type, the value is converted to an int; otherwise it is converted to an unsigned int. These are called the integral promotions. Section 6.2.1.5 Usual Arithmetic Conversions Many binary operators that expect operands of arithmetic type cause conversions and yield result types in a similar way. The purpose is to yield a common type, which is also the type of the result. This pattern is called the "usual arithmetic conversions." First, if either operand has type long double, the other operand is converted to long double. Otherwise, if either operand has type double, the other operand is converted to double. Otherwise, if either operand has type float, the other operand is converted to float. Otherwise the integral promotions [refer to section 6.2.1.1 for the integral promotions] are performed on both operands. Then the following rules are applied. If either operand has type unsigned long int, the other operand is converted to unsigned long int. Otherwise, if one operand has type long int and the other has type unsigned int, if a long int can represent all values of an unsigned int the operand of type unsigned int is converted to long int; if a long int cannot represent all the values of an unsigned int, both operands are converted to unsigned long int. Otherwise, if either operand has type long int, the other operand is converted to long int. Otherwise, if either operand has type unsigned int, the other operand is converted to unsigned int. Otherwise, both operands have type int. The values of floating operands and of the results of floating expressions may be represented in greater precision and range than that required by the type; the types are not changed thereby. In English (complete with loopholes and lack of precision), the ANSI C version would mean something like: Operands with different types get converted when you do arithmetic. Everything is converted to the type of the floatiest, longest operand, signed if possible without losing bits.

The unsigned preserving approach (K&R C) says that when an unsigned type mixes with an int or smaller signed type, the result is an unsigned type. This is a simple rule, independent of hardware, but, as in the example below, it does sometimes force a negative result to lose its sign! The value preserving approach (ANSI C) says that when you mix integral operand types like this, the result type is signed or unsigned depending on the relative sizes of the operand types. The following program fragment will print a different message under ANSI and pre-ANSI compilers:

main() { if ( -1 < (unsigned char) 1 ) printf("-1 is less than (unsigned char) 1: ANSI semantics "); else printf("-1 NOT less than (unsigned char) 1: K&R semantics "); } Depending on whether you compile it under K&R or ANSI C, the expression will be evaluated differently. The same bitpatterns are compared, but interpreted as either a negative number, or as an unsigned and hence positive number.

Software Dogma

A Subtle Bug Even though the rules were changed, subtle bugs can and do still occur. In this example, the variable d is one less than the index needed, so the code copes with it. But the if statement did not evaluate to true. Why, and what, is the bug?

int array[] = { 23, 34, 12, 17, 204, 99, 16 }; #define TOTAL_ELEMENTS (sizeof(array) / sizeof(array[0])) main() { int d= -1, x; /* ... */ if (d b ) /* swap a, b */ { int tmp = a; a = b; b = tmp; } C++ takes this a step further still, and allows arbitrary intermingling of statements and declarations, and even embedding declarations in the middle of "for" statements.

for (int i=0; i>4) and asked, "What does it mean? Is it saying 'If x is much greater than 4?'" The kind of place where overloading can be a problem is in statements like:

p = N * sizeof * q; Quickly now, are there two multiplications or only one? Here's a hint: the next statement is:

r = malloc( p ); The answer is that there's only one multiplication, because sizeof is an operator that here takes as its operand the thing pointed to by q (i.e., *q). It returns the size in bytes of the type of thing to which q points, convenient for the malloc function to allocate more memory. When sizeof 's operand is a type it has to be enclosed in parentheses, which makes people wrongly believe it is a function call, but for a variable this is not required. Here's a more complicated example:

apple = sizeof (int) * p; What does this mean? Is it the size of an int, multiplied by p? Or the size of whatever p points at, but cast to an int? Or something even weirder? The answer isn't given here, because part of being an expert programmer is learning to write little test programs to probe questions like this. Try it and see! The more work you make one symbol do, the harder it is for the compiler to detect anomalies in your use of it. It's not just the kind of people who sing along with the Tiki birds at Disneyland who have trouble here. C does seem to be a little further out on the ragged edge of token ambiguity than most other languages.

"Some of the Operators Have the Wrong Precedence" You know that you've definitely found a problem when the authors of the original report on C tell you that "some of the operators have the wrong precedence", as Kernighan and Ritchie mention on page 3 of The C Programming Language. Despite this admission, there were no changes in the precedence of operators for ANSI C. It's not surprising; any change in precedence would have imposed an intolerable burden on the existing source base. But which C operators specifically have the wrong precedence? The answer is "any that appear misleading when you apply them in the regular way." Some operators whose precedence has often caused trouble for the unwary are shown in Figure 2-1. Figure 2-1. Precedence Problems of C Operators

Most of these become more understandable if you sit down to consider them at length. The case involving the comma occasionally causes conniption fits in programmers, though. For example, when this line is executed:

i=1,2; what value does i end up with? Well, we know that the value of a comma operator is the value of the rightmost operand. But here, assignment has higher precedence, so you actually get:

(i=1), 2; /* i gets the value 1 */ i gets the value 1; then the literal 2 is evaluated and thrown away. i ends up being one, not two. In a posting on Usenet some years ago, Dennis Ritchie explained how some of these anomalies are historical accidents.

Software Dogma

'And' and 'AND' or 'Or' or 'OR' From decvax!harpo!npoiv!alice!research!dmr Date: Fri Oct 22 01:04:10 1982 Subject: Operator precedence Newsgroups: net.lang.c The priorities of && || vs. == etc. came about in the following way. Early C had no separate operators for & and && or | and ||. (Got that?) Instead it used the notion (inherited from B and BCPL) of "truth-value context": where a Boolean value was expected, after "if" and "while" and so forth, the & and | operators were interpreted as && and || are now; in ordinary expressions, the bitwise interpretations were used. It worked out pretty well, but was hard to explain. (There was the notion of "top-level operators" in a truth-value context.) The precedence of & and | were as they are now. Primarily at the urging of Alan Snyder, the && and || operators were added. This successfully separated the concepts of bitwise operations and short-circuit Boolean evaluation. However, I had cold feet about the precedence problems. For example, there were lots of programs with things like if (a==b & c==d) ... In retrospect it would have been better to go ahead and change the precedence of & to higher than ==, but it seemed safer just to split & and && without moving & past an existing operator. (After all, we had several hundred kilobytes of source code, and maybe 3 installations....) Dennis Ritchie

Handy Heuristic

Order of Evaluation The moral of all this is that you should always put parentheses around an expression that mixes Booleans, arithmetic, or bit-twiddling with anything else. And remember that while precedence and associativity tell you what is grouped with what,

the order in which these groupings will be evaluated is always undefined. In the expression:

x = f() + g() * h(); The values returned by g() and h() will be grouped together for multiplication, but g and h might be called in any order. Similarly, f might be called before or after the multiplication, or even between g and h. All we can know for sure is that the multiplication will occur before the addition (because the result of the multiplication is one of the operands in the addition). It would still be poor style to write a program that relied on that knowledge. Most programming languages don't specify the order of operand evaluation. It is left undefined so that compiler-writers can take advantage of any quirks in the architecture, or special knowledge of values that are already in registers. Pascal avoids all problems in this area by requiring explicit parentheses around expressions that mix Boolean operators and arithmetic operators. Some authorities recommend that there are only two precedence levels to remember in C: multiplication and division come before addition and subtraction. Everything else should be in parentheses. We think that's excellent advice.

Handy Heuristic

What "Associativity" Means While the precedence of operators can be perplexing, many people are equally puzzled about the associativity of operators. Operator associativity never seems to be explained very clearly in the standard C literature. This handy heuristic explains what it is, and when you need to know about it. The five-cent explanation is that it is a "tie-breaker" for operators with equal precedence. Every operator has a level of precedence and a "left" or "right" associativity assigned to it. The precedence indicates how "tightly" the operands in an unbracketed expression bind. For example, in the expression a * b + c,since multiplication has a higher precedence than addition, it will be done first, and the multiplicand will be b, not b + c. But many operators have the same precedence levels, and this is where associativity comes in. It is a protocol for explaining the real precedence among all operators that have the same apparent precedence level. If we have an expression like

int a, b=1, c=2; a = b = c;

we find that, since the expression only involves the assignment operator, precedence does not help us understand how the operands are grouped. So which happens first, the assignment of c to b, or the assignment of b to a? In the first case, a would be left with the value 2, and in the second case, a would end up as 1. All assignment-operators have right associativity. The associativity protocol says that this means the rightmost operation in the expression is evaluated first, and evaluation proceeds from right to left. Thus, the value of c is assigned to b. Then the value of b is stored in a. a gets the value 2. Similarly, for operators with left associativity (such as the bitwise and's and or 's), the operands are grouped from left to right. The only use of associativity is to disambiguate an expression of two or more equalprecedence operators. In fact, you'll note that all operators which share the same precedence level also share the same associativity. They have to, or else the expression evaluation would still be ambiguous. If you need to take associativity into account to figure out the value of an expression, it's usually better to rewrite the expression into two expressions, or to use parentheses. The order in which things happen in C is defined for some things and not for others. The order of precedence and association are well-defined. However, the order of expression evaluation is mostly unspecified (the special term defined in the previous chapter) to allow compiler-writers the maximum leeway to generate the fastest code. We say "mostly" because the order is defined for && and || and a couple of other operators. These two evaluate their operands in a strict left-to-right order, stopping when the result is known. However, the order of evaluation of the arguments in a function call is another unspecified order.

The Early Bug gets() the Internet Worm The problems in C are not confined to just the language. Some routines in the standard library have unsafe semantics. This was dramatically demonstrated in November 1988 by the worm program that wriggled through thousands of machines on the Internet network. When the smoke had cleared and the investigations were complete, it was determined that one way the worm had propagated was through a weakness in the finger daemon, which accepts queries over the network about who is currently logged in. The finger daemon, in.fingerd, used the standard I/O routine gets(). The nominal task of gets() is to read in a string from a stream. The caller tells it where to put the incoming characters. But gets() does not check the buffer space; in fact, it can't check the buffer space. If the caller provides a pointer to the stack, and more input than buffer space, gets() will happily overwrite the stack. The finger daemon contained the code:

main(argc, argv) char *argv[]; { char line[512]; ... gets(line);

Here, line is a 512-byte array allocated automatically on the stack. When a user provides more input than that to the finger daemon, the gets() routine will keep putting it on the stack. Most architectures are vulnerable to overwriting an existing entry in the middle of the stack with something bigger, that also overwrites neighboring entries. The cost of checking each stack access for size and permission would be prohibitive in software. A knowledgeable malefactor can amend the return address in the procedure activation record on the stack by stashing the right binary patterns in the argument string. This will divert the flow of execution not back to where it came from, but to a special instruction sequence (also carefully deposited on the stack) that calls execv() to replace the running image with a shell. Voilà, you are now talking to a shell on a remote machine instead of the finger daemon, and you can issue commands to drag across a copy of the virus to another machine. Repeat until sent to prison. Figure 2-2 shows the process. Figure 2-2. How the Internet Worm Gained Remote Execution Privileges

Ironically, the gets() routine is an obsolete function that provided compatibility with the very first version of the portable I/O library, and was replaced by standard I/O more than a decade ago. The manpage even strongly recommends that fgets() always be used instead. The fgets() routine sets a limit on the number of characters read, so it won't exceed the size of the buffer. The finger daemon was made secure with a two-line fix that replaced:

gets(line); by the lines:

if (fgets(line, sizeof(line), stdin) == NULL) exit(1); This swallows a limited amount of input, and thus can't be manipulated into overwriting important locations by someone running the program. However, the ANSI C Standard did not remove gets() from the language. Thus, while this particular program was made secure, the underlying defect in the C standard library was not removed.

Sins of Omission The "sins of omission" category covers things that the language doesn't do that it should. This includes missing features like standard argument processing and the mistake of extracting lint checking from the compiler.

Mail Won't Go to Users with an "f" in Their User names The bug report was very puzzling. It just said "mail isn't getting delivered to users who have an 'f' as the second character of their username." It sounded so unlikely. What could possibly cause mail to fail because of a character in the username? After all, there's no connection between the characters in a username and the mail delivery processing. Nonetheless, the problem was reported at multiple sites. After some urgent testing, we found that mail was indeed falling into the void when an addressee had an "f" as the second character of the username! Thus, mail would go to Fred and Muffy, but not to Effie. An examination of the source code quickly located the trouble. Many people are surprised to learn that ANSI C mandates the argc, argv convention of passing arguments to a C program, but it does. The UNIX convention has been elevated to the level of a standard, and it was partly to blame for the mail bug here. The mail program had been amended in the previous release to:

if ( argv[argc-1][0] == '-' || (argv[argc-2][1] == 'f' ) ) readmail(argc, argv); else sendmail(argc, argv); The "mail" program can be executed either to send mail, or to read your incoming mail. We won't enquire too closely into the merits of making one program responsible for two such different tasks. This code was supposed to look at the arguments and use the information to decide if we are reading mail or sending mail. The way to distinguish is somewhat heuristic: look for switches that are unique to either reading or sending. In this case, if the final argument is a switch (i.e., starts with a hyphen), we are definitely reading mail. We are also reading mail if the last argument is not an option but is a filename, that is, the next-to-last argument was "-f". And this is where the programmer went wrong, aided by lack of support in the language. The programmer merely looked at the second character of the next-to-last option. If it was an "f", he assumed that mail was invoked with a line like:

mail -h -d -f /usr/linden/mymailbox In most cases this was correct, and mail would be read from mymailbox. But it could also happen that the invocation was:

mail effie robert

In this case, the argument processing would make the mail program think it was being asked to read mail, not send it. Bingo! E-mail to users with an "f" as the second character of the name disappears! The fix was a one-liner: if you're looking at the next-to-last argument for a possible "f", make sure it is also preceded by a switch hyphen:

if ( argv[argc-1][0] == '-' || argv[argc-2][0] == '-' && (argv[argc-2][1] == 'f' ) ) readmail(argc, argv); The problem was caused by bad parsing of arguments, but it was facilitated by inadequate classification of arguments between switches and filenames. Many operating systems (e.g., VAX/VMS) distinguish between runtime options and other arguments (e.g., filenames) to programs, but UNIX does not; nor does ANSI C.

Software Dogma

Shell Fumbles on Argument Parsing The problem of inadequate argument parsing occurs in many places on UNIX. To find out which files in a directory are links, you might enter the command:

ls -l | grep -> This will yield the error message "Missing name for redirect", and most people will quickly figure out that the right chevron has been interpreted by the shell as a redirection, not as an argument to grep. They will then hide it from the shell by quotes, and try this:

ls -l | grep "->" Still no good! The grep program looks at the starting minus sign, interprets the argument as an unrecognized option of greater-than, and quits. The answer is to step back from "ls" and instead use:

file -h * | grep link Many people have been tormented by creating a file the name of which starts with a hyphen, and then being unable to get rm to remove it. One solution in this case is to give the entire pathname of the file, so that rm does not see a leading hyphen and try to interpret the name as an option.

Some C programmers have adopted the convention that an argument of " -- " means "from this point on, no arguments are switches, even if they start with a hyphen." A better solution would put the burden on the system, not the user, with an argument processor that divides arguments into options and non-options. The simple argv mechanism is now too well entrenched for any changes. Just don't send mail to Effie under pre-1990 versions of Berkeley UNIX.

Space—The Final Frontier A lot of people will tell you that white space isn't significant in C; that you can have as much or as little of it as you like. Not so! Here are some examples where white space radically changes the meaning or validity of a program. •

The backslash character can be used to "escape" several characters, including a newline. An escaped newline is treated as one logical line, and this can be used to continue long strings. A problem arises if you inadvertently slip a space or two in between the backslash and the carriage return, as \ whitespace newline is different than \newline. This error can be hard to find, as you are looking for something invisible (the presence of a space character where a newline was intended). A newline is typically escaped to continue a multiline macro definition. If your compiler doesn't have excellent error messages, you might as well give up now. Another reason to escape a newline is to continue a string literal, like this:

•

char a[]= "Hi! How are you? I am quite a \ long string, folded onto 2 lines"; The problem of multiline string literals was addressed by ANSI C introducing the convention that adjacent string literals are glued together. As we point out elsewhere in this chapter, that approach solved one potential problem at the expense of introducing another.

•

If you squeeze spaces out altogether, you can still run into trouble. For example, what do you think the following code means?

z = y+++x; The programmer might have meant z = y + ++x, or equally could have had z = y++ + x in mind. The ANSI standard specifies a convention that has come to be known as the maximal munch strategy. Maximal munch says that if there's more than one possibility for the next token, the compiler will prefer to bite off the one involving the longest sequence of characters. So the above example will be parsed as z = y++ + x. This can still lead to trouble, as the code

z = y+++++x; will therefore be parsed as z = y++ ++ + x, and cause a compilation error along the lines of "++ operator is floating loose in space". This will happen even though the compiler could, in theory, have deduced that the only valid arrangement of spaces is z = y++ + ++x.

• •

Yet a third white space problem occurred when a programmer had two pointers-to-int, and wanted to divide one int by the other. The code said

ratio = *x/*y; but the compiler issued an error message complaining of syntax error. The problem was the lack of space between the "/ " division operator and the "* " indirection operator. When put next to each other they opened a comment, and hid all the code up to the next closing comment! Related to opening a comment without intending to, is the case of accidentally not closing a comment when you did mean to. One release of an ANSI C compiler had an interesting bug. The symbol table was accessed by a hash function that computed a likely place from which to start a serial search. The computation was copiously commented, even describing the book the algorithm came from. Unfortunately, the programmer omitted to close the comment! The entire hash initial value calculation thus fell inside the continuing comment, resulting in the code shown below. Make sure you can identify the problem and try to predict what happened.

int hashval=0; /* PJW hash function from "Compilers: Principles, Techniques, and Tools" * by Aho, Sethi, and Ullman, Second Edition. while (cp < bound) { unsigned long overflow; hashval = ( hashval main.linker.map will give ld the "-m" option, telling it to produce a linker map. You should try this once or twice to see the kind of information that is produced. An object file isn't directly executable; it needs to be fed into a linker first. The linker identifies the main routine as the initial entry point (place to start executing), binds symbolic references to memory addresses, unites all the object files, and joins them with the libraries to produce an executable. There's a big difference between the linking facilities available on PC's and those on bigger systems. PC's typically provide only a small number of elementary I/O services, known as the BIOS routines. These exist in a fixed location in memory, and are not part of each executable. If a PC program or suite of programs requires more sophisticated services, they can be provided in a library, but the implementor must link the library into each executable. There's no provision in MS-DOS for "factoring out" a library common to several programs and installing it just once on the PC. UNIX systems used to be the same. When you linked a program, a copy of each library routine that you used went into the executable. In recent years, a more modern and superior paradigm known as dynamic linking has been adopted. Dynamic linking allows a system to provide a big collection of libraries with many useful services, but the program will look for these at runtime rather than having the library binaries bound in as part of the executable. IBM's OS/2 operating system has dynamic linking, as does Microsoft's new flagship NT operating system. In recent years, Microsoft Windows® has introduced this ability for the windowing part of PC applications. If a copy of the libraries is physically part of the executable, then we say the executable has been statically linked; if the executable merely contains filenames that enable the loader to find the program's library references at runtime, then we say it has been dynamically linked. The canonical names for the three phases of collecting modules together and preparing them for execution are linkediting, loading, and runtime linking. Statically linked modules are link edited and then loaded to run them. Dynamically linked modules are link-edited and then loaded and runtime-linked to run them. At execution, before main() is called, the runtime loader brings the shared data objects into the process address space. It doesn't resolve external function calls until the call is actually made, so there's no penalty to linking against a library that you may not call. The two linking methods are compared in Figure 5-2

. Figure 5-2. Static Linking versus Dynamic Linking

Even with static linking, the whole of libc. a is not brought into the executable, just the routines needed.

The Benefits of Dynamic Linking Dynamic linking is the more modern approach, and has the advantage of much smaller executable size. Dynamic linking trades off more efficient use of the disk and a quicker link-edit phase for a small runtime penalty (since some of the linker's work is deferred until loadtime).

Handy Heuristic

One Purpose of Dynamic Linking Is the ABI A major purpose of dynamic linking is to decouple programs from the particular library versions they use. Instead, we have the convention that the system provides an interface to programs, and that this interface is stable over time and successive OS releases. Programs can call services promised by the interface, and not worry about how they are provided or how the underlying implementation may change. Because this is an interface between application programs and the services provided by library binary executables, we call it an Application Binary Interface or ABI. A single ABI is the purpose of unifying the UNIX world around AT&T's SVr4. The ABI guarantees that the libraries exist on all compliant machines, and ensures the integrity of the interface. There are four specific libraries for which dynamic linking is mandatory: libc (C runtimes), libsys (other system runtimes), libX (X windowing), and libnsl (networking services). Other libraries can be statically linked, but dynamic linking is strongly preferred. In the past, application vendors had to relink their software with each new release of the OS or a library. It caused a huge amount of extra work for all concerned. The ABI does away with this, and guarantees that well-behaved applications will not be affected by wellbehaved upgrades in underlying system software. Although an individual executable has a slightly greater start-up cost, dynamic linking helps overall performance in two ways: 1. A dynamically linked executable is smaller than its statically linked counterpart. It saves disk and virtual memory, as libraries are only mapped in to the process when needed. Formerly, the only way to avoid binding a library copy into each executable was to put the service in the kernel instead of a library, contributing to the dreaded "kernel bloat." 2. All executables dynamically linked to a particular library share a single copy of the library at runtime. The kernel ensures that libraries mapped into memory are shared by all processes using them. This provides better I/O and swap space utilization and is sparing of physical memory, improving overall system throughput. If the executables were statically linked, each would wastefully contain its own complete duplicate copy of the library. For example, if you have eight XView™ applications running, only one copy of the XView library text segment has to be mapped into memory. The first process's mmap [1] call will result in the kernel mapping the shared object into memory. The next seven process mmaps will cause the kernel to share the existing mapping in each process. Each of the eight processes will share one copy of the XView library in memory. If the library were statically linked, there would be eight individual copies consuming more physical memory and causing more paging. [1]

The system call mmap() maps a file into a process address space. The contents of the file can then be obtained by reading successive memory locations. This is particularly appropriate when the file contains executable instructions. The file system is regarded as part of the virtual memory system in SVr4, and mmap is the mechanism for bringing a file into memory.

Dynamic linking permits easy versioning of libraries. New libraries can be shipped; once installed on the system, old programs automatically get the benefit of the new versions without needing to be relinked.

Finally (much less common, but still possible), dynamic linking allows users to select at runtime which library to execute against. It's possible to create library versions that are tuned for speed, or for memory efficiency, or that contain extra debugging information, and to allow the user to express a preference when execution takes place by substituting one library file for another. Dynamic linking is "just-in-time" linking. It does mean that programs need to be able to find their libraries at runtime. The linker accomplishes this by putting library filenames or pathnames into the executable; and this in turn, means that libraries cannot be moved completely arbitrarily. If you linked your program against library /usr/lib/libthread.so, you cannot move the library to a different directory unless you specified it to the linker. Otherwise, the program will fail at runtime when it calls a function in the library, with an error message like:

ld.so.1: main: fatal: libthread.so: can't open file: errno=2 This is also an issue when you are executing on a different machine than the one on which you compiled. The execution machine must have all the libraries that you linked with, and must have them in the directories where you told the linker they would be. For the standard system libraries, this isn't a problem. The main reason for using shared libraries is to get the benefit of the ABI—freeing your software from the need to recompile with each new release of a library or OS. As a side benefit, there are also overall system performance advantages. Anyone can create a static or dynamic library. You simply compile some code without a main routine, and process the resulting .o files with the correct utility—"ar" for static libraries, or "ld" for dynamic libraries.

Software Dogma

Only Use Dynamic Linking! Dynamic linking is now the default on computers running System V release 4 UNIX, and it should always be used. Static linking is now functionally obsolete, and should be allowed to rest in peace. The major risk you run with static linking is that future versions of the operating system will be incompatible with the system libraries bound in with your executable. If your application was statically linked on OS version N and you try to run it on version N+1, it may run, or it may fail with a core dump or a less obvious error. There is no guarantee that an earlier version of the system libraries will execute correctly on a later version of the system. Indeed, it is usually safer to assume the opposite. However, if an application is dynamically linked on version N of the system, it will correctly pick up the version N+1 libraries when run on version N+1 of the system. In contrast, statically linked

applications have to be regenerated for every new release of the operating system to ensure that they keep running. Furthermore, some libraries (such as libaio.so, libdl.so, libsys.so, libresolv.so, and librpcsvc.so) are only available in dynamic form. You are obliged to link dynamically if your application uses any of these libraries. The best policy is to avoid problems by making sure that all applications are dynamically linked. Static libraries are known as archives and they are created and updated by the ar—for archive— utility. The ar utility is misnamed; if truth in advertising applied to software, it would really be called something like glue_files_together or even static_library_updater. Convention dictates that static libraries have a ".a" extension on their filename. There isn't an example of creating a static library here, because they are obsolete now, and we don't want to encourage anyone to communicate with the spirit world. There was an interim kind of linking used in SVR3, midway between static linking and dynamic linking, known as "static shared libraries". Their addresses were fixed throughout their life, and thus could be bound to without the indirection required with dynamic linking. On the other hand, they were inflexible and required a lot of special support in the system. We won't consider them further. A dynamically linked library is created by the link editor, ld. The conventional file extension for a dynamic library is ".so" meaning "shared object"—every program linked against this library shares the same one copy, in contrast to static linking, in which everyone is (wastefully) given their own copy of the contents of the library. In its simplest form, a dynamic library can be created by using the -G option to cc, like this:

% cat tomato.c my_lib_function() {printf("library routine called\n"); } % cc -o libfruit.so -G tomato.c You can then write routines that use this library, and link with it in this manner:

% cat test.c main() { my_lib_function(); } % cc test.c -L/home/linden -R/home/linden -lfruit % a.out library routine called The -L/home/linden -R/home/linden options tell the linker in which directories to look for libraries at linktime and at runtime, respectively. You will probably also want to use the -K pic compiler option to produce position-independent code for your libraries. Position-independent code means that the generated code makes sure that every global data access is done through an extra indirection. This makes it easy to relocate the data simply by changing one value in the table of global offsets. Similarly, every function call is generated as a call through an indirect address in a procedure linkage table. The text can thus easily be relocated

to anywhere, simply by fixing up the offset tables. So when the code is mapped in at runtime, the runtime linker can directly put it wherever there is room, and the code itself doesn't have to be changed. By default, the compilers don't generate PICode as the additional pointer dereference is a fraction slower at runtime. However, if you don't use PICode, the generated code is tied to a fixed address— fine for an executable, but slower for a shared library, since every global reference now has to be fixed up at runtime by page modification, in turn making the page unshareable. The runtime linker will fix up the page references anyway, but the task is greatly simplified with position-independent code. It is a trade-off whether PICode is slower or faster than letting the runtime linker fix up the code. A rule of thumb is to always use PICode for libraries. Position-independent code is especially useful for shared libraries because each process that uses a shared library will generally map it at a different virtual address (though sharing one physical copy). A related term is "pure code." A pure executable is one that contains only code (no static or initialized data). It is "pure" in the sense that it doesn't have to be modified to be executed by any specific process. It references its data off the stack or from another (impure) segment. A pure code segment can be shared. If you are generating PIcode (indicating sharing) you usually want it to be pure, too.

Five Special Secrets of Linking with Libraries There are five essential, non-obvious conventions to master when using libraries. These aren't explained very clearly in most C books or manuals, probably because the language documenters consider linking part of the surrounding operating system, while the operating system people view linking as part of the language. As a result, no one makes much more than a passing reference to it unless someone from the linker team gets involved! Here are the essential UNIX linking facts of life: 1. Dynamic libraries are called lib something .so , and static libraries are called lib something .a By convention, all dynamic libraries have a filename of the form libname .so (version numbers may be appended to the name). Thus, the library of thread routines is called libthread .so. A static archive has a filename of the form libname.a. Shared archives, with names of the form libname .sa, were a transient phenomenon, helping in the transition from static to dynamic libraries. Shared archives are also obsolete now. 2. You tell the compiler to link with, for example, libthread.so by giving the option -lthread The command line argument to the C compiler doesn't mention the entire pathname to the library file. It doesn't even mention the full name of the file in the library directory! Instead, the compiler is told to link against a library with the command line option -lname where the library is called libname .so—in other words, the "lib" part and the file extension are dropped, and -l is jammed on the beginning instead. 3. The compiler expects to find the libraries in certain directories

At this point, you may be wondering how the compiler knows in which directory to look for the libraries. Just as there are special rules for where to find header files, so the compiler looks in a few special places such as /usr/lib/ for libraries. For instance, the threads library is in /usr/lib/libthread.so. The compiler option -Lpathname is used to tell the linker a list of other directories in which to search for libraries that have been specified with the -l option. There are a couple of environment variables, LD_LIBRARY_PATH and LD_RUN_PATH, that can also be used to provide this information. Using these environment variables is now officially frowned on, for reasons of security, performance, and build/execute independence. Use the -Lpathname -Rpathname options at linktime instead. 4. Identify your libraries by looking at the header files you have used Another key question that may have occurred to you is, "How do I know which libraries I have to link with?" The answer, as (roughly speaking) enunciated by ObiWan Kenobi in Star Wars, is, "Use the source, Luke!" If you look at the source of your program, you'll notice routines that you call, but which you didn't implement. For example, if your program does trigonometry, you've probably called routines with names like sin() or cos(), and these are found in the math library. The manpages show the exact argument types each routine expects, and should mention the library it's in. A good hint is to study the #includes that your program uses. Each header file that you include potentially represents a library against which you must link. This tip carries over into C++, too. A big problem of name inconsistency shows up here. Header files usually do not have a name that looks anything like the name of the corresponding library. Sorry! This is one of the things you "just have to know" to be a C wizard. Table 5-1 shows examples of some common ones.

Table 5-1. Library Conventions Under Solaris 2.x

#include Filename

Library Pathname

<math.h>

/usr/lib/libm.so

<math.h>

/usr/lib/libm.a

Compiler Option to Use

-lm -dn -lm

<stdio.h> /usr/lib/libc.so linked in automatically "/usr/openwin/include/X11.h" /usr/openwin/lib/libX11.so -



/usr/lib/libthread.so

<curses.h>

/usr/ccs/lib/libcurses.a

<sys/socket.h>

/usr/lib/libsocket.so

L/usr/openwin/lib -lX11 -lthread -lcurses -lsocket

Another inconsistency is that a single library may contain routines that satisfy the prototypes declared in multiple header files. For example, the functions declared in the header files <string.h>, <stdio.h>, and are all usually supplied in the single library libc.so. If you're in doubt, use the nm utility to list

the routines that a library contains. More about this in the next heuristic!

Handy Heuristic

How to Match a Symbol with its Library If you're trying to link a program and get this kind of error:

ld: Undefined symbol _xdr_reference *** Error code 2 make: Fatal error: Command failed for target 'prog' Here's how you can locate the libraries with which you need to link. The basic plan is to use nm to look through the symbols in every library in /usr/lib, grepping for the symbols you're missing. The linker looks in /usr/ccs/lib and /usr/lib by default, and so should you. If this doesn't get results, extend your search to all other library directories (such as /usr/openwin/lib), too.

% cd /usr/lib % foreach i (lib?*) ? echo $i ? nm $i | grep xdr_reference | grep -v UNDEF ? end libc.so libnsl.so [2491] | 217028| 196|FUNC |GLOB |0 |8 |xdr_reference libposix4.so ... This runs "nm" on each library in the directory, to list the symbols known in the library. Pipe it through grep to limit it to the symbol you are searching for, and filter out symbols marked as "UNDEF" (referenced, but not defined in this library). The result shows you that xdr_reference is in libnsl. You need to add -lnsl on the end of the compiler command line. 5. Symbols from static libraries are extracted in a more restricted way than symbols from dynamic libraries

Finally, there's an additional and big difference in link semantics between dynamic linking and static linking that often confuses the unwary. Archives (static libraries) are acted upon differently than are shared objects (dynamic libraries). With dynamic libraries, all the library symbols go into the virtual address space of the output file, and all the symbols are available to all the other files in the link. In contrast, static linking only looks through the archive for the undefined symbols presently known to the loader at the time the archive is processed. A simpler way of putting this is to say that the order of the statically linked libraries on the compiler command line is significant. The linker is fussy about where libraries are mentioned, and in what order, since symbols are resolved looking from left to right. This makes a difference if the same symbol is defined differently in two different libraries. If you're doing this deliberately, you probably know enough not to need to be reminded of the perils. Another problem occurs if you mention the static libraries before your own code. There won't be any undefined symbols yet, so nothing will be extracted. Then, when your object file is processed by the linker, all its library references will be unfulfilled! Although the convention has been the same since UNIX started, many people find it unexpected; very few commands demand their arguments in a particular order, and those that do usually complain about it directly if you get it wrong. All novices have trouble with this aspect of linking until the concept is explained. Then they just have trouble with the concept itself. The problem most frequently shows up when someone links with the math library. The math library is heavily used in many benchmarks and applications, so we want to squeeze the last nanosecond of runtime performance out of it. As a result, libm has often been a statically linked archive. So if you have a program that uses some math routines such as the sin() function, and you link statically like this:

cc -lm main.c you will get an error message like this:

Undefined first referenced symbol in file sin main.o ld: fatal: Symbol referencing errors. No output written to a.out In order for the symbols to get extracted from the math library, you need to put the file containing the unresolved references first, like so:

cc main.c -lm This causes no end of angst for the unwary. Everyone is used to the general command form of , so to have the linker adopt the different convention of is very confusing. It's exacerbated by the fact that it will silently accept the first

version and do the wrong thing. At one point, Sun's compiler group amended the compiler drivers so that they coped with the situation. We changed the SunOS 4.x unbundled compiler drivers from SC0.0 through SC2.0.1 so they would "do the right thing" if a user omitted -lm. Although it was the right thing, it was different from what AT&T did, and broke our compliance with the System V Interface Definition; so the former behavior had to be reinstated. In any case, from SunOS 5.2 onwards a dynamically linked version of the math library /usr/lib/libm.so is provided.

Handy Heuristic

Where to Put Library Options Always put the -l library options at the rightmost end of your compilation command line. Similar problems have been seen on PC's, where Borland compiler drivers tried to guess whether the floating-point libraries needed to be linked in. Unfortunately, they sometimes guessed wrongly, leading to the error:

scanf : floating point formats not linked Abnormal program termination They seem to guess wrongly when the program uses floating-point formats in scanf() or printf() but doesn't call any other floating-point routines. The workaround is to give the linker more of a clue, by declaring a function like this in a module that will be included in the link:

static void forcefloat(float *p) { float f = *p; forcefloat(&f); } Don't actually call the function, merely ensure that it is linked in. This provides a solid enough clue to the Borland PC linker that the floating-point library really is needed. NB: a similar message, saying "floating point not loaded" is printed by the Microsoft C runtime system when the software needs a numeric coprocessor but your computer doesn't have one installed. You fix it by relinking the program, using the floating-point emulation library.

Watch Out for Interpositioning Interpositioning (some people call it "interposing") is the practice of supplanting a library function by a user-written function of the same name. This is a technique only for people who enjoy a good walk on the wild side of the fast lane without a safety net. It enables a library function to be replaced in a

particular program, usually for debugging or performance reasons. But like a gun with no safety catch, while it lets experts get faster results, it also makes it very easy for novices to hurt themselves. Interpositioning requires great care. It's all too easy to do this accidentally and replace a symbol in a library by a different definition in your own code. Not only are all the calls that you make to the library routine replaced by calls to your version, but all calls from system routines now reference your routine instead. A compiler will typically not issue an error message when it notices a redefinition of a library routine. In keeping with C's philosophy that the programmer is always right, it assumes the programmer meant to do it. Over the years we have seen no convincing examples where interpositioning was essential but the effect could not be obtained in a different (perhaps less convenient) manner. We have seen many instances where a default global scope symbol combined with interpositioning to create a hard-to-find bug (see Figure 5-3). We have seen a dozen or so bug reports and emergency problem escalations from even the most knowledgeable software developers. Unhappily, it's not a bug; the implementation is supposed to work this way. Figure 5-3. Diagram of Interpositioning and Default Global Scope

Most programmers have not memorized all the names in the C library, and common names like index or mktemp tend to be chosen surprisingly often. Sometimes bugs of this kind even get into production code.

Software Dogma

An Interpositioning Bug in SunOS Under SunOS 4.0.3, the printing program /usr/ucb/lpr occasionally issued the error message

"out of memory" and refused to print. The fault appeared randomly, and it was very hard to track down. Finally, it was traced to an unintended interpositioning bug. The programmer had implemented lpr with a routine, global by default, called mktemp(), which expected three arguments. Unknown to the programmer, that duplicated a name already present in the (pre-ANSI) C library, which did a similar job but only took one argument. Unfortunately, lpr also invoked the library routine getwd(), which expects to use the library version of mktemp. Instead, it was bound to lpr 's special version! Thus, when getwd() called mktemp, it put one argument on the stack, but the lpr version of mktemp retrieved three. Depending on the random values it found, lpr failed with the "out of memory" error. Moral: Don't make any symbols in your program global, unless they are meant to be part of your interface! lpr was fixed by declaring its mktemp as static, making it invisible outside its own file (we could equally have given it a different name). Mktemp has now been replaced by the library routine tmpnam in ANSI C. However, the opportunity for the interpositioning problem still exists. If an identifier is shown in Table 5-2, never declare it in your own program. Some of these are always reserved, and others are only reserved if you include a specific header file. Some of these are reserved only in global scope, and others are reserved for both global and file scope. Also note that all keywords are reserved, but are left out of the table below for simplicity. The easiest way to stay out of trouble is to regard all these identifiers as belonging to the system at all times. Don't use them for your identifiers. Some entries look like is[a-z]anything . This means any identifier that begins with the string "is" followed by any other lowercase letter (but not, for example, a digit) followed by any characters. Other entries look like acos,-f,-l. This indicates that the three identifiers acos, acosf, and acosl are reserved. All routines in the math header file have a basic version that takes a double-precision argument. There can also be two extra versions: the basename with a l suffix is a version of the routine with quad precision arguments (type "long double"), and the f suffix is a version with single precision ("float").

Table 5-2. Names to Avoid Using as Identifiers (Reserved for the System in ANSI C) Don't Use These Names for Your Identifiers _anything

abort asin,-f,-l atexit

abs assert atof

acos,-f,-l atan,-f,-l atoi

asctime atan2,-f,-l atol

bsearch BUFSIZ calloc ceil,-f,-l CHAR_BIT CHAR_MAX CHAR_MIN clearerr clock clock_t CLOCKS_PER_SEC cos,-f,-l cosh,-f,-l ctime currency_symbol DBL_DIG DBL_EPSILON DBL_MANT_DIG DBL_MAX DBL_MAX_10_EXP DBL_MAX_EXP DBL_MIN DBL_MIN_10_EXP DBL_MIN_EXP decimal_point defined difftime div div_t E[0-9] E[A-Z]anything errno exit EXIT_FAILURE EXIT_SUCCESS exp,-f,-l fabs,-f,-l fclose feof ferror fflush fgetc fgetpos fgets FILE FILENAME_MAX floor,-f,-l FLT_DIG FLT_EPSILON FLT_MANT_DIG FLT_MAX FLT_MAX_10_EXP FLT_MAX_EXP FLT_MIN FLT_MIN_10_EXP FLT_MIN_EXP FLT_RADIX FLT_ROUNDS fmod,-f,-l fopen FOPEN_MAX fpos_t fprintf fputc fputs frac_digits fread free freopen frexp,-f,-l fscanf fseek fsetpos ftell fwrite getc getchar getenv gets gmtime grouping HUGE_VAL int_curr_symbol int_frac_digits INT_MAX INT_MIN is[a-z]anything jmp_buf L_tmpnam labs LC_[A-Z]anything lconv LDBL_DIG LDBL_EPSILON LDBL_MANT_DIG LDBL_MAX LDBL_MAX_10_EXP LDBL_MAX_EXP LDBL_MIN LDBL_MIN_10_EXP LDBL_MIN_EXP ldexp,-f,-l ldiv ldiv_t localeconv localtime log,-f,-l log10,-f,-l LONG_MAX LONG_MIN longjmp malloc MB_CUR_MAX MB_LEN_MAX mblen mbstowcs mbtowc mem[a-z]anything mktime modf,-f,-l mon_decimal_point mon_grouping mon_thousands_sep n_cs_precedes n_sep_by_space n_sign_posn NDEBUG negative_sign NULL offsetof p_cs_precedes p_sep_by_space p_sign_posn perror positive_sign pow,-f,-l printf ptrdiff_t putc putchar puts qsort raise rand RAND_MAX realloc remove rename rewind scanf SCHAR_MAX SCHAR_MIN SEEK_CUR SEEK_END SEEK_SET setbuf setjmp

setlocale SIG_[A-Z]anything SIG_IGN SIGILL SIGTERM sprintf srand stdout tanh,-f,-l tm tm_min tm_yday tmpnam ULONG_MAX va_end vprintf wcstombs

setvbuf sig_atomic_t SIG[A-Z]anything SIGINT sin,-f,-l sqrt,-f,-l sscanf str[a-z]anything thousands_sep tm_hour tm_mon tm_year to[a-z]anything ungetc va_list vsprintf wctomb

SHRT_MAX SIG_DFL SIGABRT signal sinh,-f,-l

SHRT_MIN SIG_ERR SIGFPE SIGSEGV size_t

stderr system time tm_isdst tm_sec TMP_MAX UCHAR_MAX USHRT_MAX va_start wchar_t

stdin tan,-f,-l time_t tm_mday tm_wday tmpfile UINT_MAX va_arg vfprintf wcs[a-z]anything

Remember that under ANSI section 6.1.2 (Identifiers), an implementation can define letter case not significant for external identifiers. Also, external identifiers only need be significant in the first six characters (ANSI section 5.2.4.1, Translation Limits). Both of these expand the number of identifiers that you should avoid. The list above consists of the C library symbols that you may not redefine. There will be additional symbols for each additional library you link against. Check the ABI document [2] for a list of these. [2]

The System V Application Binary Interface, AT&T, 1990.

The problem of name space pollution is only partially addressed in ANSI C. ANSI C outlaws a user redefining a system name (effectively outlawing interpositioning) in section 7.1.2.1: 7.1.2.1 Reserved Identifiers: All identifiers with external linkage in any of the following sections [what follows is a number of sections defining the standard library functions]…are always reserved for use as identifiers with external linkage. If an identifier is reserved, it means that the user is not allowed to redefine it. However, this is not a constraint, so it does not require an error message when it sees it happen. It just causes unportable, undefined behavior. In other words, if one of your function names is the same as a C library function name (deliberately or inadvertently), you have created a nonconforming program, but the translator is not obliged to warn you about it. We would much rather the standard required the compiler to issue a warning diagnostic, and let it make up its own mind about stuff like the maximum number of case labels it can handle in a switch statement.

Generating Linker Report Files Use the "-m" option to ld for a linker report that includes a note of symbols which have been interposed. In general, the "-m" option to ld will produce a memory map or listing showing what has

been put where in the executable. It also shows multiple instances of the same symbol, and by looking at what files these occur in, the user can determine if any interpositioning took place. The -D option to ld was introduced with SunOS 5.3 to provide better link-editor debugging. The option (fully documented in the Linker and Libraries Manual) allows the user to display the linkediting process and input file inclusion. It's especially useful for monitoring the extraction of objects from archives. It can also be used to display runtime bindings. Ld is a complicated program with many more options and conventions than those explained here. Our description is more than enough for most purposes, and there are four further sources of help, in increasing order of sophistication: • • • •

Use the ldd command to list the dynamic dependencies of an executable. This command will tell you the libraries that a dynamically linked program needs. The -Dhelp option to ld provides information on troubleshooting the linking process. Try the on-line manpages for ld. Read the SunOS Linker and Libraries Manual (part number 801-2869-10).

Some combination of these should provide information on any subtle linker special effects you need.

Handy Heuristic

When "botch" Appears Under SunOS 4.x, an occurrence of the word "botch" in an error message means that the loader has discovered an internal inconsistency. This is usually due to faulty input files. Under SunOS 5.x, the loader is much more rigorous about checking its input for correctness and consistency. It doesn't need to complain about internal errors, and the "botch" messages have been dropped.

Some Light Relief—Look Who's Talking: Challenging the Turing Test At the dawn of the electronic age, as the potential of computers first started to unfold, a debate arose over whether systems would one day have artificial intelligence. That quickly led to the question, "How can we tell if a machine thinks?" In a 1950 paper in the journal Mind, British mathematician Alan Turing cut through the philosophical tangle by suggesting a practical test. Turing proposed that a human interrogator converse (via teletype, to avoid sight and sound clues) with another person and with a computer. If the human interrogator was unable to correctly identify which was which after a period of five minutes, then the computer would be said to have exhibited artificial intelligence. This scenario has come to be called the Turing Test.

Over the decades since Turing proposed this trial, the Turing test has taken place several times, sometimes with astonishing results. We describe some of those tests and reproduce the dialogue that took place so you can judge for yourself.

Eliza One of the first computer programs to process natural language was "Eliza," named after the gabby heroine in Shaw's play Pygmalion. The Eliza software was written in 1965 by Joseph Weizenbaum, a professor at MIT, and it simulated the responses of a Rogerian psychiatrist talking to a patient. The program made a superficial analysis of the text typed to it, and spat back one of a large number of canned responses built into it. The illusion that the computer understood any of the conversation fooled quite a number of com-puter-naive people. Weizenbaum got the first indication of this phenomenon when he invited his secretary to try the system. After a couple of minutes typing with Eliza, the secretary (who had watched Weizenbaum work on the software for many previous months, and could be expected to understand better than most that it was just a computer program) asked Weizenbaum to leave the room so that she could continue her dialogue in private! The Turing test failed on its first outing, as the secretary plainly regarded this elementary software (with no pretensions to artificial intelligence) as human. Rather than the software showing intelligence, the humans showed gullibility. Eliza proved to be a popular program, and was adapted by Bolt Berenek and Newman, a Boston computer consul-tancy. This led to a more severe test when a BBN vice president found himself connected to Eliza instead of the programmer he was expecting.

Eliza Meets the VP Computer scientist Daniel Bobrow gave this account in the December 1968 issue of the SIGART Newsletter. One Saturday morning about 9am, a BBN vice-president in charge of selling our Telcomp commercial service arrived at our computer room hoping to be able to use the computer for a demonstration. He found a note on the teletype that read: "type all messages into this teletype, and end them with a period." The note was left there by one of the programming bums who had just succeeded in making run a Lisp version of the Doctor program (based on ELIZA as developed by Weizenbaum). However, the VP thought that this note indicated that this teletype was linked with the one in my home, since I often used the computer all night. Therefore, he started typing to "me", but in fact was typing to the Doctor program. The following conversation ensued: vp: teletype: vp: teletype: vp: teletype: vp: teletype: vp: teletype:

Do you think that I can use the computer this morning? Why do you ask? I might be able to make some additional telcomp sales. Why aren't you sure? My prospective customers have never seen a demonstration of the system. Does this mean very much to you? Of course it does. You seem very positive of that. This is ridiculous. What do you mean this is ridiculous?

vp: teletype: vp:

Is there any reason why I can't use the computer now? Tell me more about your prospective customers. Please call me

Note that after that remark the VP did not obey instructions and left out the period. Therefore the computer didn't answer him. This so infuriated the VP, who thought I was playing games with him, that he called me up, woke me from a deep sleep, and said: vp: Bobrow:

Why are you being so snotty with me? What do you mean why am I being snotty to you?

The VP angrily read the dialog that "we" had been having, and couldn't get any response but laughter from me. It took me a while to convince him it really was the computer. Notice that the Eliza/Doctor software usually responds with an open-ended question designed to get the user to type in more sentences. It makes no attempt to answer questions, because it has no ability to do so. Nonetheless the program fooled the computer-literate vice president. The reason it fools people is not because it is intelligent, and thus passing the Turing test. Though novel at the time, Eliza is a quite elementary program by today's standards. It fools people because people are too easy to fool, and thus the test is inadequate. So the Turing test was a failure on its second outing.

Doctor, Meet Doctor The Free Software Foundation, for reasons of whimsy, hack-value, and fun, built this psychiatrist dialogue software into their popular GNU editor product. The GNU-EMACS editor is given away free, and is available on most UNIX systems. Once you start up GNU-EMACS, issue the command "escape-x doctor". I thought it would be entertaining to hook up two instances of the Doctor program back-to-back, so that each was talking at the other. Here's the result of the experiment: 1: I am the psychotherapist. Please describe your problems. 2: What brings you to see me? 1: Can you elaborate on that? 2: I would appreciate it if you would continue. 1: What makes you believe you would appreciate it if I would continue? 2: Why do you say that? [doctor 1 now starts to make grammatical errors…] 1: When did you first know that do I say that? [doctor 2 also starts to make grammatical errors…] 2: What happened when did me first know that do you say that? 1: What would happen if did you first know that do I say that? 2: Is the fact that do you say that the real reason? 1: Is the fact that do I say that the real reason the real reason? 2: does the fact that do you say that the real reason the real reason explain anything else? 1: does the fact that do I say that the real reason the real reason explain anything else explain anything else? 2: Are you sure do you say that the real reason the real reason explain anything else explain anything

else? 1: Are you sure the real reason the real reason explain anything else explain anything else? 2: Earlier you said do I do what brings me to see you? 1: Earlier you said do you do what brings you to see me that I came to you? 2: Is it because you came to me that you came to me? 1: Is it because I came to you that you came to me? 2: Are you annoyed that you came to me that I came to you? [doctor 1 tries a new line, and recovers its grammar, only to get stuck in repetition] 1: Have you been annoyed often? 2: Have you been annoyed often? 1: What causes you to be annoyed? 2: Are you sure that you are annoyed? 1: What makes you believe that you are annoyed? 2: Have you been annoyed often? 1: Have you been annoyed often? Actually, as you can see, it wasn't very entertaining at all. This dialogue exposes the "echoing" that is Doctor 's main conversational device. Its shallow text analysis is easily thrown off by a complicated sentence. When Doctor tries to parrot back part of a previous remark as a question, it often creates a sentence too complicated for itself to handle. Although Doctor has deceived people into thinking it is human, no one looking at this transcript would be fooled.

The Prize in Boston Fast forward now to 1991. Computer systems are vastly faster and more powerful. People have worked on all aspects of artificial intelligence for more than a quarter century. We come to the most recent and most spectacular of the Turing tests. In November 1991 the Boston Computer Museum, along with the National Science Foundation and the Alfred P. Sloan Foundation, sponsored a public demonstration of the Turing test. The museum chose the human interrogators by placing ads in Boston news-papers. The museum wanted the interrogators to represent the average person (rather than scientists with special philosophical or linguistics knowledge). A panel of ten interrogators had three hours to converse, through terminals, with more than a dozen programs and human conversationalists. One typical dialogue went: terminal: A whim is a sudden capricious and often peculiar behavior. A bit of sunlight in a grim grey world. interrogator: Something like a hunch? terminal: Everybody sets out to do something and everybody does something, but no one does what he sets out to do. interrogator: Well, anyway I would imagine whimsical conversation is like a series of knight's moves in chess—sort of zigzag rather than following any particular line. terminal: A whimsical person is subject to erratic behavior and unpredictable thoughts…how nice to be unpredictable!

It comes as no surprise that the terminal above is actually a computer program. It's operating just as Eliza was; it analyzes the syntax and keywords in the text from the interrogator, and selects something with a matching topic from its huge database of canned phrases. It avoids the "doctor's dilemma" by not parroting back part of the interrogator's remark, instead keeping the talk flowing by continually raising new (though related) topics. It's also no surprise that the program represented above deluded five of the ten interrogators, who marked it down as human after this and more lengthy interchanges with it. Third time unlucky for the Turing test, and it's out for the count.

Conclusions The above program's inability to directly answer a straightforward question ("[do you mean] something like a hunch?") is a dead giveaway to a computer scientist, and highlights the central weakness in the Turing test: simply exchanging semi-appropriate phrases doesn't indicate thought— we have to look at the content of what is communicated. The Turing test has repeatedly been shown to be inadequate. It relies on surface appear-ances, and people are too easily deceived by surface appearance. Quite apart from the significant philosophical question of whether mimicking the outward signs of an activity is evidence of the inner human processes which accompany that activity, human interro-gators have usually proven incapable of accurately making the necessary distinctions. Since the only entities in everyday experience that converse are people, it's natural to assume that any conversation (no matter how stilted) is with another person. Despite the empirical failures, the artificial intelligence community is very unwilling to let the test go. There are many defenses of it in the literature. Its theoretical simplicity has a compelling charm; but if something does not work in practice, it must be revised or scrapped. The original Turing test was phrased in terms of the interrogator being able to distinguish a woman, from a man masquerading as a woman, over a teletype. Turing did not directly address in his paper that the test would probably be inadequate for this, too. One might think that all that is necessary is to reemphasize this aspect of the conversation; that is, require the interrogator to debate the teletype on whether it is human or not. I doubt that is likely to be any more fruitful. For simplicity, the 1991 Computer Museum tests restricted the conversation to a single domain for each teletype. Different programs had different knowledge bases, covering topics as diverse as shopping, the weather, whimsy, and so on. All that would be needed is to give the program a set of likely remarks and clever responses on the human condition. Turing wrote that five minutes would be adequate time for the trial; that doesn't seem nearly adequate these days. One way to fix the Turing test is to repair the weak link: the element of human gullibility. Just as we require doctors to pass several years of study before they can conduct medical examinations, so we must add the condition that the Turing interrogators should not be representatives of the average person in the street. The interrogators should instead be well versed in computer science, perhaps graduate students familiar with the capabilities and weaknesses of computer systems. Then they won't be thrown off by witty remarks extracted from a large database in lieu of real answers. Another interesting idea is to explore the sense of humor displayed by the terminal. Ask it to distinguish whether a particular story qualifies as a joke or not, and explain why it is funny. I think such a test is too severe—too many people would fail it.

Although a brilliant theoretician, Turing was often hopeless when it came to practical matters. His impracticality showed itself in unusual ways: at his office, he chained his mug to the radiator to prevent his colleagues from using it. They naturally regarded this as a challenge, picked the lock, and drank from it wilfully. He routinely ran a dozen or more miles to distant appointments, arriving sticky and exhausted, rather than use public transport. When war broke out in Europe in 1939, Turing converted his savings into two silver ingots which he buried in the countryside for safety; by the end of the war he was unable to remember where he cached them. Turing eventually committed suicide in a characteristically impractical fashion: he ate an apple that he had injected with cyanide. And the test which bears his name is a triumph of theory over practical experience. The difference between theory and practice is a lot bigger in practice than in theory.

Postscript Turing also wrote that he believed that "at the end of the century the use of words and general educated opinion would have altered so much that one will be able to speak of machines thinking without expecting to be contradicted." That actually happened much sooner than Turing reckoned. Programmers habitually explain a computer 's quirks in terms of thought processes: "You haven't pressed carriage return so the machine thinks that there's more input coming, and it's waiting for it." However, this is because the term "think" has become debased, rather than because machines have acquired consciousness, as Turing predicted. Alan Turing was rightly recognized as one of the great theoretical pioneers in computing. The Association for Computing Machinery established its highest annual prize, the Turing Award, in his memory. In 1983, the Turing Award was given to Dennis Ritchie and Ken Thompson in recognition of their work on UNIX and C.

Further Reading If you are interested in learning more about the advances and limitations of artificial intelligence, a good book to read is What Computers Still Can't Do: A Critique of Artificial Reason by Hubert L. Dreyfus, published by the MIT Press, Boston, 1992.

Chapter 6. Poetry in Motion: Runtime Data Structures #41: The Enterprise meets God, and it's a child, a computer, or a C program. #42: While boldly on the way to where only a few people have been recently, the Enterprise computer is subverted by a powerful alien life-form, shaped amazingly like a human. #43: Trekkers encounter hostile computer intelligence, and abuse philosophy or logic to cause it to self-destruct. #44: Trekkers encounter a civilization that bears an uncanny resemblance to a previous Earth society. #45: Disease causes one or more crew members to age rapidly. Also seen in reverse: key crew members regress to childhood, physically, mentally, or both. #46: An alien being becomes embedded in body of a Trekker and takes over. Still waiting to see this one in reverse…

#47: The captain violates the Prime Directive, then either endangers the Enterprise or has an affair with an attractive alien, or both, while trying to rectify matters. #48: The captain eventually brings peace to two primitive warring societies ("we come in peace, shoot to kill") on a world that is strangely reminiscent of Earth. —Professor Snopes' Book of Canonical Star Trek Plots, and Delicious Yam Recipes a.out and a.out folklore…segments…what the OS does with your a.out…what the C runtime does with your a.out…what happens when a function gets called: the procedure activation record…helpful C tools…some light relief—Princeton programming puzzle One of the classic dichotomies in programming language theory is the distinction between code and data. Some languages, like LISP, unite these elements; others, like C, usually maintain the division. The Internet worm, described in Chapter 2, was hard to understand because its method of attack was based on transforming data into code. The distinction between code and data can also be analyzed as a division between compiletime and runtime. Most of the work of a compiler is concerned with translating code; most of the necessary data storage management happens at runtime. This chapter describes the hidden data structures of the runtime system. There are three reasons to learn about the runtime system: • • •

a.out

It will help you tune code for the highest performance. It will help you understand more advanced material. It will help you diagnose problems more easily, when you run into trouble.

and a.out Folklore

Did you ever wonder how the name "a.out" was chosen? Having all output files default to the same name, a.out, can be inconvenient, as you might forget which source file it came from, and you will overwrite it on the next compilation of any file. Most people have a vague impression that the name originated with traditional UNIX brevity, and "a" is the first letter of the alphabet, so it's the first letter you hit for a new filename. Actually, it's nothing to do with any of this. It's an abbreviation for "assembler output"! The old BSD manpage even hints at this:

NAME a.out - assembler and link editor output format One problem: it's not assembler output—it's linker output! The "assembler output" name is purely historical. On the PDP-7 (even before the B language), there was no linker. Programs were created by assembling the catenation of all the source files, and the resulting assembler output went in a.out. The convention stuck around for the final output even after a linker was finally written for the PDP-11; the name had come to mean "a new program ready to try to execute." So the a.out default name is an example in UNIX of "no good reason, but we always did it that way"!

Executable files on UNIX are also labelled in a special way so that systems can recognize their special properties. It's a common programming technique to label or tag important data with a unique number identifying what it is. The labelling number is often termed a "magic" number; it confers the mysterious power of being able to identify a collection of random bits. For example, the superblock (the fundamental data structure in a UNIX filesystem) is tagged with the following magic number:

#define FS_MAGIC 0x011954 That strange-looking number isn't wholly random. It's Kirk McKusick's birthday. Kirk, the implementor of the Berkeley fast file system, wrote this code in the late 1970's, but magic numbers are so useful that this one is still in the source base today (in file sys/fs/ufs_fs.h). Not only does it promote file system reliability, but also, every file systems hacker now knows to send Kirk a birthday card for January 19. There's a similar magic number for a.out files. Prior to AT&T's System V release of UNIX, an a.out was identified by the magic number 0407 at offset zero. And how was 0407 selected as the "magic number" identifying a UNIX object file? It's the opcode for an unconditional branch instruction (relative to the program counter) on a PDP-11! If you're running on a PDP-11 or VAX in compatibility mode, you can just start executing at the first word of the file, and the magic number (located there) will branch you past the a.out header and into the first real executable instruction of the program. The PDP-11 was the canonical UNIX machine at the time when a.out needed a magic number. Under SVr4, executables are marked by the first byte of a file containing hex 7F followed by the letters "ELF" at bytes 2, 3, and 4 of the file.

Segments Object files and executables come in one of several different formats. On most SVr4 implementations the format is called ELF (originally "Extensible Linker Format", now "Executable and Linking Format"). On other systems, the executable format is COFF (Common Object-File Format). And on BSD UNIX (rather like the Buddha having Buddha-nature), a.out files have a.out format. You can find out more about the format used on a UNIX system by typing man a.out and reading the manpage. All these different formats have the concept of segments in common. There will be lots more about segments later, but as far as object files are concerned, they are simply areas within a binary file where all the information of a particular type (e.g., symbol table entries) is kept. The term section is also widely used; sections are the smallest unit of organization in an ELF file. A segment typically contains several sections. Don't confuse the concept of segment on UNIX with the concept of segment on the Intel x86 architecture. • •

A segment on UNIX is a section of related stuff in a binary. A segment in the Intel x86 memory model is the result of a design in which (for compatibility reasons) the address space is not uniform, but is divided into 64-Kbyte ranges known as segments.

The topic of segments on the Intel x86 architecture really deserves a chapter of its own. [1] For the remainder of this book, the term segment has the UNIX meaning unless otherwise stated. [1]

And it pretty near has one, too! See next chapter.

When you run size on an executable, it tells you the size of three segments known as text, data, and bss in the file:

% echo; echo "text data bss total" ; size a.out text 1548

+

data 4236

+

bss 4004 =

total 9788

Size doesn't print the headings, so use echo to generate them. Another way to examine the contents of an executable file is to use the nm or dump utilities. Compile the source below, and run nm on the resulting a.out.

char pear[40]; static double peach; int mango = 13; static long melon = 2001; main () { int i=3, j, *ip; ip=malloc(sizeof(i)); pear[5] = i; peach = 2.0*mango; } Excerpts from running nm are shown below (minor editing changes have been made to the output to make it more accessible):

% nm -sx a.out Symbols from a.out: [Index] Value Size Type Bind ... [29] |0x00020790|0x00000008|OBJT |LOCL [42] |0x0002079c|0x00000028|OBJT |GLOB [43] |0x000206f4|0x00000004|OBJT |GLOB [30] |0x000206f8|0x00000004|OBJT |LOCL [36] |0x00010628|0x00000058|FUNC |GLOB [50] |0x000206e4|0x00000038|FUNC |GLOB ...

Segment Name |.bss peach |.bss pear |.data mango |.data melon |.text main |UNDEF malloc

Figure 6-1 shows what the compiler/linker puts in each segment: Figure 6-1. What Kinds of C Statements End Up in Which Segments?

The BSS segment gets its name from abbreviating "Block Started by Symbol"—a pseudo-op from the old IBM 704 assembler, carried over into UNIX, and there ever since. Some people like to remember it as "Better Save Space." Since the BSS segment only holds variables that don't have any value yet, it doesn't actually need to store the image of these variables. The size that BSS will require at runtime is recorded in the object file, but BSS (unlike the data segment) doesn't take up any actual space in the object file.

Programming Challenge

Look at the Segments in an Executable 1. Compile the "hello world" program, run ls -l on the executable to get its overall size, and run size to get the sizes of the segments within it. 2. Add the declaration of a global array of 1000 ints, recompile, and repeat the measurements. Notice the differences. 3. Now add an initial value in the declaration of the array (remember, C doesn't force you to provide a value for every element of an array in an initializer). This will move the array from the BSS segment to the data segment. Repeat the measurements. Notice the differences. 4. Now add the declaration of a big array local to a function. Declare a second big local array with an initializer. Repeat the measurements. Is data defined locally inside a function stored in the executable? Does it make any difference if it's initialized or not? 5. What changes occur to file and segment sizes if you compile for debugging? For maximum optimization?

Analyze the results of the above "Programming Challenge" to convince yourself that: • • • •

the data segment is kept in the object file the BSS segment isn't kept in the object file (except for a note of its runtime size requirements) the text segment is the one most affected by optimization the a.out file size is affected by compiling for debugging, but the segments are not.

What the OS Does with Your a.out Now we see why the a.out file is organized into segments. The segments conveniently map into objects that the runtime linker can load directly! The loader just takes each segment image in the file and puts it directly into memory. The segments essentially become memory areas of an executing program, each with a dedicated purpose. This is shown in Figure 6-2. Figure 6-2. How the Segments of an Executable are Laid Out in Memory

The text segment contains the program instructions. The loader copies that directly from the file into memory (typically with the mmap() system call), and need never worry about it again, as program text typically never changes in value nor size. Some operating systems and linkers can even assign appropriate permissions to the different sections in segments, for example, text can be made read-andexecute-only, some data can be made read-write-no-execute, other data made read-only, and so on. The data segment contains the initialized global and static variables, complete with their assigned values. The size of the BSS segment is then obtained from the executable, and the loader obtains a

block of this size, putting it right after the data segment. This block is zeroed out as it is put in the program's address space. The entire stretch of data and BSS is usually just referred to jointly as the data segment at this point. This is because a segment, in OS memory management terms, is simply a range of consecutive virtual addresses, so adjacent segments are coalesced. The data segment is typically the largest segment in any process. The diagram shows the memory layout of a program that is about to begin execution. We still need some memory space for local variables, temporaries, parameter passing in function calls, and the like. A stack segment is allocated for this. We also need heap space for dynamically allocated memory. This will be set up on demand, as soon as the first call to malloc() is made. Note that the lowest part of the virtual address space is unmapped; that is, it is within the address space of the process, but has not been assigned to a physical address, so any references to it will be illegal. This is typically a few Kbytes of memory from address zero up. It catches references through null pointers, and pointers that have small integer values. When you take shared libraries into account, a process address space appears, as shown in Figure 6-3. Figure 6-3. Virtual Address Space Layout, Showing Shared Libraries

What the C Runtime Does with Your a.out Now we come to the fine detail of how C organizes the data structures of a running program. There are a number of runtime data structures: the stack, activation records, data, heap, and so on. We will look at each of these in turn, and analyze the C features that they support.

The Stack Segment The stack segment contains a single data structure, the stack. A classic computer science object, the stack is a dynamic area of memory that implements a last-in-first-out queue, somewhat similar to a stack of plates in a cafeteria. The classic definition of a stack says that it can have any number of plates on it, but the only valid operations are to add or remove a plate from the top of the stack. That is, values can be pushed onto the stack, and retrieved by popping them off. A push operation makes the stack grow larger, and a pop removes a value from it. Compiler-writers take a slightly more flexible approach. We add or delete plates only from the top, but we can also change values that are on a plate in the middle of the stack. A function can access

variables local to its calling function via parameters or global pointers. The runtime maintains a pointer, often in a register and usually called sp, that indicates the current top of the stack. The stack segment has three major uses, two concerned with functions and one with expression evaluation: • •

•

The stack provides the storage area for local variables declared inside functions. These are known as "automatic variables" in C terminology. The stack stores the "housekeeping" information involved when a function call is made. This housekeeping information is known as a stack frame or, more generally, a procedure activation record. We'll describe it in detail a little later, but for now be aware that it includes the address from which the function was called (i.e., where to jump back to when the called function is finished), any parameters that won't fit into registers, and saved values of registers. The stack also works as a scratch-pad area—every time the program needs some temporary storage, perhaps to evaluate a lengthy arithmetic expression, it can push partial results onto the stack, popping them when needed. Storage obtained by the alloca() call is also on the stack. Don't use alloca() to get memory that you want to outlive the routine that allocates it. (It will be overwritten by the next function call.)

A stack would not be needed except for recursive calls. If not for these, a fixed amount of space for local variables, parameters, and return addresses would be known at compiletime and could be allocated in the BSS. Early implementations of BASIC, COBOL, and FORTRAN did not permit recursive calls of functions, so they did not need a dynamic stack at runtime. Allowing recursive calls means that we must find a way to permit mul-tiple instances of local variables to be in existence at one time, though only the most recently created will be accessed — the classic specification of a stack.

Programming Challenge

Stack Hack Compile and run this small test program to discover the approximate location of the stack on your system:

#include <stdio.h> main() { int i; printf("The stack top is near %p\n", &i); return 0; } Discover the data and text segment locations, and the heap within the data segment, by declaring variables that will be placed in those segments and printing their addresses. Make

the stack grow by calling a function, and declaring some large local arrays. What's the address of the top of the stack now? The stack may be located at a different address on different architectures and for different OS revisions. Although we talk about the top of the stack, the stack grows downwards on most processors, towards memory addresses with lower values.

What Happens When a Function Gets Called: The Procedure Activation Record This section describes how the C runtime manages the program within its own address space. Actually, the runtime routines for C are remarkably few and lightweight. In contrast to, say, C++ or Ada, if a C program wants some service such as dynamic storage allocation, it usually has to ask for it explicitly. This makes C a very efficient language, but it does place an extra burden on the programmer. One of the services that is provided automatically is keeping track of the call chain—which routines have called which others, and where control will pass back to, on the next "return" statement. The classic mechanism that takes care of this is the procedure activation record on the stack. There will be a procedure activation record (or its equivalent) for each call statement executed. The procedure activation record is a data structure that sup-ports an invocation of a procedure, and also records everything needed to get back to where you came from before the call. (See Figure 6-4.) Figure 6-4. A Canonical Procedure Activation Record

The description of the contents of activation records is illustrative. The exact structure will vary from implementation to implementation. The order of the fields may be quite different. There may be an area for saving register values before making the call. The include file, /usr/include/sys/frame.h, shows how a stack frame looks on your UNIX system. On SPARC, a stack frame is large—several dozen words in size—because it provides room to save register windows. On the x86 architecture, the frame is somewhat smaller. The runtime maintains a

pointer, often in a register and usually called fp, which indicates the active stack frame. This will be the stack frame nearest to or at the top of the stack.

Software Dogma

Astonishing C Fact! Most modern algorithmic languages allow functions (as well as data) to be defined inside functions. C does not allow functions to be nested this way. All functions in C are at the top lexical level. This restriction slightly simplifies C implementations. In languages like Pascal, Ada, Modula-2, PL/I, or Algol-60, which do allow lexically nested procedures, an activation record will typically contain a pointer to the activation record of its enclosing function. This pointer is termed a static link, [1] and it allows the nested procedure to access the stack frame of the enclosing procedure, and hence the data local to the enclosing procedure. Remember that there may be several invocations of an enclosing procedure active at once. The static link in the nested procedure's activation record will point to the appropriate stack frame, allowing the correct instances of local data to be addressed. This type of access (a reference to a data item defined in a lexically enclosing scope) is known as an uplevel reference. The static link (pointing to the stack frame of the lexically enclosing procedure determined at compiletime) is so named because it contrasts with the dynamic link of the frame pointer chain (pointing to the stack frame of the immediately previous procedure invocation at runtime). In Sun's Pascal compiler, the static link is treated as an additional hidden parameter, passed at the end of the argument list when needed. This allows Pascal routines to have the same activation record, and thus to use the same code generator, and to work with C routines. C itself does not allow nested functions; therefore, it does not have uplevel references to data in them, and thus does not need a static link in its activation records. Some people are urging that C++ should be given the ability to have nested functions. [1]

Don't, for God's sake, confuse the static link in a procedure activation record, which permits uplevel references to local data in a lexically enclosing procedure, with a static link of the previous chapter that describes the obsolete method of binding a copy of all the libraries into an executable. In the previous chapter, static meant "done at compiletime". Here, it means "referring to the lexical layout of the program".

The code example below will be used to show the activation records on the stack at various points in execution. This is a hard concept to represent in a book, because we have to deal with the dynamic flow of control rather than the static code that a listing shows. Admittedly, this is difficult to follow, but as Wendy Kaminer remarked in her classic psy-chological text, I'm Dysfunctional; You're Dysfunctional, only people who die very young learn all they really need to know in kindergarten.

1 a (int i) {

2 if (i>0) 3 a(--i); 4 else 5 printf("i has reached zero "); 6 return; 7 } 8 9 main(){ 10 a(1); 11 } If we compile and run the above program, the flow of control is shown in Figure 6-5. Each dashed box shows a fragment of source that makes a function call. The statements executed are shown in bold print. As control passes from one function to another, the new state of the stack is shown underneath. Execution starts at main, and the stack grows downwards. Figure Figure6-5. An Activation Record is Created at Runtime for Each Function Call

Compiler-writers will try to speed up programs by not storing information that will not be used. Other optimizations include keeping information in registers instead of on the stack, not pushing a complete stack frame for a leaf function (a function that doesn't make any calls itself), and making the callee responsible for saving registers, rather than the caller. The "pointer to previous frame" within each frame simplifies the task of popping the stack back to the previous record when the current function returns.

Programming Challenge

The Stack Frame 1. Manually trace the flow of control in the above program, and fill in the stack frames at each call statement. For each return address, use the line number to which it will go back. 2. Compile the program for real, and run it under the debugger. Look at the additions to the stack when a function has been called. Compare with your desk checked results to work out exactly what a stack frame looks like on your system. Remember that compiler-writers will try to place as much of an activation record in registers as possible (because it's faster), so some of this may not be visible on the stack. Refer to the frame.h file to see the layout of a stack frame.

The auto and static keywords The description of how the stack implements function calling also explains why it doesn't work to return a pointer to a local automatic variable from a function, like this:

char * favorite_fruit () { char deciduous [] = "apple"; return deciduous; } The automatic variable deciduous is allocated on the stack when the function is entered; after the function is exited, the variable no longer exists, and the stack space can be overwritten at any time. Pointers that have lost their validity in this way (by referencing something that is no longer live) are known as "dangling pointers"—they don't reference anything useful, just kind of dangle in space. If you need to return a pointer to something defined in a function, then define the thing as static. This will ensure that space for the variable is allocated in the data segment instead of on the stack. The lifetime of the variable is thus the lifetime of the program, and as a side effect it retains its value even after the function that defines it exits. That value will still be available when the function is next entered. The storage class specifier auto is never needed. It is mostly meaningful to a compiler-writer making an entry in a symbol table—it says "this storage is automatically allocated on entering the block" (as opposed to statically allocated at compiletime, or dynamically allocated on the heap). Auto is pretty much meaningless to all other programmers, since it can only be used inside a function, but data declarations in a function have this attribute by default. The only use we have ever found for the auto keyword is to make your declarations line up neatly, like this:

register int filbert;

auto int almond; static int hazel; instead of:

register int filbert; int almond; static int hazel;

A Stack Frame Might Not Be on the Stack Although we talk about a "stack frame" being "pushed on the stack," an activation record need not be on the stack. It's actually faster and better to keep as much as possible of the activation record in registers. The SPARC architecture takes this to the limit with a concept called "register windows" in which the chip has a set of registers solely dedicated to holding parameters in procedure activation records. Empty stack frames are still pushed for each call; if the call chain goes so deep that you run out of register windows, the registers are reclaimed by spilling them to the frame space already reserved on the stack. Some languages, such as Xerox PARC's Mesa and Cedar, have activation records allocated as linked lists in the heap. Activation records for recursive procedures were also heap-allocated for the first PL/I implementations (leading to criticism of slow performance, because the stack is usually a much faster place from which to acquire memory).

Threads of Control It should now be clear how different threads of control (i.e., what used to be called "lightweight processes") can be supported within a process. Simply have a different stack dedicated to each thread of control. If a thread calls foo(), which calls bar(), which calls baz(), while the main program is executing other routines, each needs a stack of its own to keep track of where each is. Each thread gets a stack of 1Mb (grown as needed) and a page of red zone betweeen each thread's stack. Threads are a very powerful programming paradigm, which provide a performance advantage even on a uniprocessor. However, this is a book on C, not on threads; you can and should seek out more details on threads.

setjmp

and longjmp

We can also mention what setjmp() and longjmp() do, since they are implemented by manipulating activation records. Many novice programmers do not know about this powerful mechanism, since it's a feature unique to C. They partially compensate for C's limited branching ability, and they work in pairs like this: •

setjmp(jmp_buf j) must be called first. It says use the variable j to remember where

•

longjmp(jmp_buf j,int i) can then be called. It says go back to the place that the j is remembering. Make it look like you're returning from the original setjmp(), but return the value of i so the code can tell when you actually got back here via longjmp().

you are now. Return 0 from the call.

Phew!

•

The contents of the j are destroyed when it is used in a longjmp().

Setjmp saves a copy of the program counter and the current pointer to the top of the stack. This saves some initial values, if you like. Then longjmp restores these values, effectively transferring control and resetting the state back to where you were when you did the save. It's termed "unwinding the stack," because you unroll activation records from the stack until you get to the saved one. Although it causes a branch, longjmp differs from a goto in that: • •

A goto can't jump out of the current function in C (that's why this is a "longjmp"— you can jump a long way away, even to a function in a different file). You can only longjmp back to somewhere you have already been, where you did a setjmp, and that still has a live activation record. In this respect, setjmp is more like a "come from" statement than a "go to". Longjmp takes an additional integer argument that is passed back, and lets you figure out whether you got here from longjmp or from carrying on from the previous statement.

The following code shows an example of setjmp() and longjmp().

#include <setjmp.h> jmp_buf buf; #include <setjmp.h> banana() { printf("in banana()\n"); longjmp(buf, 1); /*NOTREACHED*/ printf("you'll never see this, because I longjmp'd"); } main() { if (setjmp(buf)) printf("back in main\n"); else { printf("first time through\n"); banana(); } } The resulting output is:

% a.out first time through in banana() back in main

Point to watch: the only reliable way to ensure that a local variable retains the value that it had at the time of the longjmp is to declare it volatile. (This is for variables whose value changes between the execution of setjmp and the return of longjmp.) A setjmp/longjmp is most useful for error recovery. As long as you haven't returned from the function yet, if you discover a unrecoverable error, you can transfer control back to the main input loop and start again from there. Some people use setjmp/longjmp to return from a chain of umpteen functions at once. Others use them to shield potentially dangerous code, for example, when dereferencing a suspicious pointer as shown in the following example.

switch(setjmp(jbuf)) { case 0: apple = *suspicious; break; case 1: printf("suspicious is indeed a bad pointer\n"); break; default: die("unexpected value returned by setjmp"); } This needs a handler for the segmentation violation signal, which will do the corresponding longjmp(jbuf,1) as explained in the next chapter. Setjmp and longjmp have mutated into the more general exception routines "catch" and "throw" in C++.

Programming Challenge

Jump to It! Take the source of a program you have already written and add setjmp/longjmp to it, so that on receiving some particular input it will start over again. The header file <setjmp.h> needs to be included in any source file that uses setjmp or longjmp. Like goto's, setjmp/longjmp can make it hard to understand and debug a program. They are best avoided except in the specific situations described.

The Stack Segment Under UNIX On UNIX, the stack grows automatically as a process needs more space. The programmer can just assume that the stack is indefinitely large. This is one of the many advantages that UNIX has over operating environments such as MS-DOS. UNIX implementations generally use some form of virtual memory. When you try to access beyond the space currently allocated for the stack, it generates a hardware interrupt known as a page fault. A page fault is processed in one of several ways, depending on whether the reference was valid or invalid. The kernel normally handles a reference to an invalid address by sending the appropriate signal (segmentation fault probably) to the offending process. There's a small "red zone" region just below the top of the stack. A reference to there doesn't pass on a fault; instead, the operating system increases the stack segment size by a good chunk. Details vary among UNIX implementations, but in effect, additional virtual memory is mapped into the address space following the end of the current stack. The memory mapping hardware ensures you cannot access memory outside that which the operating system has allocated to your process.

The Stack Segment Under MS-DOS In DOS, the stack size has to be specified as part of building the executable, and it cannot be grown at runtime. If you guess wrongly, and your stack gets bigger than the space you've allocated, you and your program both lose, and if you've turned checking on, you'll get the STACK OVERFLOW! message. This can also appear at compiletime, when you've exceeded the limits of a segment. Turbo C will tell you Segment overflowed maximum size if too much data or code had to be combined into a single segment. The limit is 64Kbytes, due to the 80x86 architecture. The method of specifying stack size varies with the compiler you're using. With Microsoft compilers, the programmer can specify the stack size as a linker parameter. The

STACK:nnnn parameter tells the Microsoft linker to allow nnnn bytes for the stack. The Borland compilers use a variable with a special name:

unsigned int _stklen = 0x4000; /* 16K stack */ Other compiler vendors have different methods for doing this. Check the programmer reference guides under "stack size" for the details.

Helpful C Tools This section contains some lists (grouped by function) of helpful C tools you should know about, and what they do, in tables 6-1 through 6-4. We've already hinted at a few of these that help you peek inside a process or an a.out file. Some are specific to Sun OS. This section provides an easy-to-read summary of what each does, and where to find them. After studying the summary here, read the

manpage for each, and try running each on a couple of different a.out's—the "hello world" program, and a big program. Go through these carefully—investing 15 minutes now in trying each one will save you hours later in solving a hard bug.

Table 6-1. Tools to Examine Source Tool

cb

Where to Find It Comes with the compiler

indent cdecl This book cflow Comes with

the compiler cscope Comes with the compiler

ctags /usr/bin lint

Comes with the compiler /usr/ccs/bin

sccs vgrind /usr/bin

What It Does C program beautifier. Run your source through this filter to put it in a standard layout and indentation. Comes from Berkeley. Does the same things cb does. Comes from AT & T. Unscrambles C declarations. Prints the caller/callee relationships of a program. An interactive ASCII-based C program browser. We use it in the OS group to check the impact of changes to header files. It provides quick answers to questions like: "How many commands use libthread?" or "Who are all the kmem readers?" Creates a tags file for use in vi editor. A tags file speeds up examining program source by maintaining a table of where most objects are located. A C program checker. A source code version control system. A formatter for printing nice C listings.

Doctors can use x-rays, sonograms, arthroscopes, and exploratory operations to look inside their patients. These tools are the x-rays of the software world.

Table 6-2. Tools to Examine Executables Tool

dis dump Lv ldd nm strings sum

Where to Find It /usr/ccs/bin

Object code disassembler

/usr/ccs/bin

Prints dynamic linking information

/usr/bin

Prints the dynamic libraries this file needs

/usr/ccs/bin

Prints the symbol table of an object file

/usr/bin

Looks at the strings embedded in a binary. Useful for looking at the error messages a binary can generate, built-in file names, and (sometimes) symbol names or version and copyright information. Prints checksum and block count for a file. An-swers questions like: "Are two executables the same version?" "Did the transmission go OK?"

/usr/bin

What It Does

Table 6-3. Tools to Help with Debugging

Tool

truss ps ctrace

Where to What It Does Find It /usr/bin The SVr4 version of trace. This tool prints out the system calls that an executable makes. Use it to see what a binary is doing, and why it's stuck or failing. This is a great help! /usr/bin Displays process characteristics.

Comes with compiler debugger Comes with compiler /usr/bin file

Modifies your source to print lines as they are executed. A great tool for small programs! Interactive debugger. Tells you what a file contains (e.g., executable, data, ASCII, shell script, archive, etc.).

Table 6-4. Tools to Help with Performance Tuning Tool

Where to Find It collector Comes with debugger analyzer Comes with debugger /usr/ccs/bin gprof

prof tcov time

What It Does (SunOS only) Collects runtime performance data under the control of the debugger. (SunOS only) Analyzes collected performance data.

/usr/ccs/bin

Displays the call-graph profile data (identifies the computeintensive functions). Displays the percentage of time spent in each routi ne.

Comes with compiler /usr/bin/time

Displays a count of how often each statement is executed (identifies the compute-intensive loops within a function). Displays the total real and CPU time used by a program.

If you're working on the OS kernel, most of the runtime tools are not available to you, because the kernel does not run as a user process. The compiletime tools, like lint, work, but otherwise we have to use the stone knives and flint axes: putting nonrandom patterns in memory to see when they are overwritten (two favorites are the hex constants dead-beef and abadcafe), using printf 's or their equivalent, and logging trace information.

Software Dogma

Debugging the Ker nel with grep A kernel "panics", or comes to an abrupt halt, when it detects a situation that "cannot" arise. For example, it finds a null pointer when looking for some essential data. Since there is no way it can recover from this, the safest course is to halt the processor before more data disappears. To solve a panic, you must first consider what happened that could possibly

frighten an operating system. The kernel development group at Sun had one obscure bug that was very difficult to track down. The symptom was that kernel memory was getting overwritten at random, occasionally panicking the system. Two of our top engineers worked on this, and they noticed that it was always the 19th byte from the beginning of a memory block that was being creamed. This is an uncommon offset, unlike the more usual 2, 4, or 8 that occur everywhere. One of the engineers had a brainwave and used the offset to home in on the bug. He suggested they use the kernel debugger kadb to disassemble the kernel binary image (it took an hour!) into an ASCII file. They then grepped the file, looking for "store" instructions where the operand indicated offset 19! One of these was almost guaranteed to be the instruction causing the corruption. There were only eight of these, and they were all in the subsystem that dealt with process control. Now they were pretty sure where the problem was, and just needed to find out what it was. Further effort eventually yielded the cause: a race condition on a process control structure. This resulted in one thread marking the memory for return to the system before another thread had truly finished with it. Result: the kernel memory allocator then gave the memory away to someone else, but the process control code thought it still held it, and wrote into it, causing the otherwise impossible-to-find corruption! Debugging an OS kernel with grep—what a concept. Sometimes even source tools can help solve runtime problems! While on the subject of useful tools, Table 0-5 lists some ways to see exactly what the configuration of a Sun system is. However, none of these tools can help you unless you practice using them.

Table 6-5. Tools to Help Identify Your Hardware What It Identifies Kernel architecture sun4c

Typical Output

How to Invoke It

no patches are installed

/usr/kvm/arch -k /usr/bin/showrev -p

lots

/usr/sbin/prtconf

40MHz processor

/usr/sbin/psrinfo -v /usr/ucb/hostid

Any OS patches applied Various hard ware things CPU clock rate hostid

554176fe

Memory Serial number ROM revision Mounted disk

32Mb 4290302 2.4.1 198Mb disk

Swap space

40Mb

Ethernet address

8:0:20:f:8c:60

Displayed on power up Displayed on power up Displayed on power up

/usr/bin/df -F ufs -k /etc/swap -s /usr/sbin/ifconfig -a The ethernet address is built into the machine

IP address

le0=129.144.248.36

Floating-point hardware

The IP address is built into the network FPU's frequency appears to be 38.2 fpversion comes with the compiler MHz

/usr/sbin/ifconfig -a

Some Light Relief—Programming Puzzles at CMU A few years ago, the Computer Science Department at Carnegie-Mellon University regularly ran a small programming contest for its incoming graduate students. The contest was intended to give the new researchers some hands-on experience with the department systems and let them demonstrate their star potential. CMU has a long and distinguished involvement with computers, stretching back to the pioneering days, so you'd typically expect some outstanding entries in a programming competition there. The form of the contest varied from year to year, and in one particular year it was very simple. Contestants had to read a file of numbers and print the average. There were only two rules: 1. The program had to run as fast as possible. 2. The program had to be written in Pascal or C. The rival programs were grouped together and submitted in batches by a faculty member. Students could submit as many entries as they wished; this encouraged the use of non-deterministic probabilistic algorithms (algorithms that would guess at certain characteristics of the data set and use the guess to obtain faster performance). The golden rule was that the program which ran in the shortest amount of time won the contest. The graduate students duly retired to dark corners and started whacking their various programs. Most of them had three or four versions to enter and run. At this point, the reader is invited to guess the techniques that helped the programs run fast.

Programming Challenge

How to Exceed the Speed Limit? Imagine that you have been given the task of writing a program to read a file of 10,000 numbers and calculate the average. Your program must run in the shortest amount of time. What programming and compiler techniques will you use to achieve this? Most people guess the biggest wins will come through code optimization, either explicitly in the code, or implicitly by using the correct compiler options. The standard code optimizations are techniques

like loop unrolling, in-line function expansion, common subexpression elimination, improved register allocation, omitting runtime checks on array bounds, loop-invariant code motion, operator strength reduction (turning exponen-tiation into multiplication, turning multiplication into bit-shifting and/or addition), and so on. The data file contained about 10,000 numbers, so assuming it took a millisecond just to read and process each number (about par for the systems at the time), the fastest possible program would take about ten seconds. The actual results were very surprising. The fastest program, as reported by the operating system, took minus three seconds. That's right—the winner ran in a negative amount of time! The next fastest apparently took just a few milliseconds, while the entry in third place came in just under the expected 10 seconds. Obviously, some devious programming had taken place, but what and how? A detailed scrutiny of the winning programs eventually supplied the answer. The program that apparently ran backwards in time had taken advantage of the operating system. The programmer knew where the process control block was stored relative to the base of the stack. He crafted a pointer to access the process control block and overwrote the "CPU-time-used" field with a very high value. [2] The operating system wasn't expecting CPU times that large, and it misinterpreted the very large positive number as a negative number under the two's complement scheme. [2]

A flagrant abuse of the rules, ranking equal to the time Argentinian soccer ace Diego Maradona used his forearm to nudge the ball into the England net for the winning goal in the 1986 soccer World Cup.

The runner-up, whose program took just a few milliseconds, had been equally cunning in a different way. He had used the rules of the competition rather than exotic coding. He submitted two different entries. One entry read the numbers, calculated the answer in the normal way, and wrote the result to a file; the second entry spent most of its time asleep, but woke up briefly every few seconds and checked if the answer file existed. When it did, it printed it out. The second process only took a few milliseconds of total CPU time. Since contestants were allowed to submit multiple entries, the smaller time counted and put him in second place. The programmer whose entry came in third, taking slightly less than the calculated minimum time, had the most elaborate scheme. He had worked out the optimized machine code instructions to solve the problem, and stored the instructions as an array of integers in his program. It was easy to overwrite a return address on the stack (as Bob Morris, Jr. later did with the Internet worm of 1988) and cause the program to jump into and start executing this array. These instructions solved the problem honestly in record time. There was an uproar among the faculty when these stratagems were uncovered. Some of the staff were in favor of severely reprimanding the winners. A younger group of professors proposed instead awarding them an extra prize in recognition of their ingenuity. In the end, a compromise was reached. No prizes were awarded, no punishments were handed down, and the results stuck. Sadly, the contest was a casualty of the strong feelings, and this incident marked the final year it was held.

For Advanced Students Only A word to the wise: it's possible to embed assembler code in C source. This is usually only done for the most machine-specific things in the depths of the OS kernel, like setting specific registers on changing from supervisor mode to user mode. Here's how we plant a no-op (or other instruction) in a C function using a SunPro SPARCompiler:

banana() { asm("nop"); } Here's how you embed assembly language instructions using Microsoft C on a PC:

__asm mov ah, 2 __asm mov dl, 43h You can also prefix the assembler code with the keyword "_ _asm". You can also use the keyword once, and put all the assembler code in a pair of braces, like this:

__asm { mov ah, 2 mov dl, 43h int 21h } Little checking is done for you, and it's easy to create programs that bomb out. But it's a good way to experiment with and learn the instruction set of a machine. Take a look at the SPARC architecture manual, the assembler manual (which mostly talks about syntax and directives), and a databook from one of the SPARC vendors, such as Cypress Semiconductor's "SPARC RISC User's Guide."

Chapter 7. Thanks for the Memory A master was explaining the nature of the Tao to one of his novices. "The Tao is embodied in all software—regardless of how insignificant," said the master. "Is the Tao in a hand-held calculator?" asked the novice. "It is," came the reply. "Is the Tao in a video game?" continued the novice. "It is even in a video game," said the master. "And is the Tao in the DOS for a personal computer?" The master coughed and shifted his position slightly. "That would be in the stack frame Bob, and the lesson is over for today," he said. —Geoffrey James, The Tao of Programming the Intel 80x86 family…the Intel 80x86 memory model and how it got that way…virtual memory…cache memory…the data segment and heap… memory leaks…bus error—take the train…some light relief—the thing king and the paging game This chapter starts with a discussion of memory architecture for the Intel 80x86 processor family (the processor at the heart of the IBM PC). It contrasts PC memory with the virtual memory feature found on other systems. A knowledge of memory architecture helps a programmer to understand some of the C conventions and restrictions.

The Intel 80x86 Family Modern Intel processors can trace their heritage all the way back to the earliest Intel chips. As customers became more sophisticated and demanding in their use of chip sets, Intel was always ready with compatible follow-on processors. Compatibility made it easy for customers to move to newer chips, but it severely restricted the amount of innovation that was possible. The modern Pentium is a direct descendant of Intel's 8086 from 15 years before, and it contains architectural irregularities to provide backwards compatibility with it (meaning that programs compiled for an 8086 will run on a Pentium). Referring to the innovation/compatibility trade-off, some people have unkindly commented that "Intel puts the 'backward' in 'backward compatible'…" (see Figure 7-1). Figure 7-1. The Intel 80x86 Family: Putting the "Backward" in "Backward Compatible"

The Intel 4004 was a 4-bit microcontroller built in 1970 to satisfy the specific needs of a single customer, Busicom—a Japanese calculator company. The Intel design engineer conceived the idea of producing a general-purpose programmable chip, instead of the custom logic for each different customer that was the rule at the time. Intel thought they'd sell a few hundred, but a general-purpose design turned out to have vastly wider applicability. Four bits was too limiting, so in April 1972 an 8bit version, the 8008, was launched. Two years later, that in turn spawned the 8080, which was the first chip powerful enough to be called a microcomputer. It included the entire 8008 instruction set and added 30 more instructions of its own, initiating a trend that continues to this day. If the 4004 was

the chip that got Intel started, the 8080 was the chip that made its fortune, boosting the company's turnover to more than $1 billion annually and placing it high on the Fortune 500 list. The 8085 processor took advantage of advances in integration technology to squeeze a three chip combination onto one. In essence, it was an 8080 combined with the 8224 clock driver and the 8228 controller, all on a single chip. Although its internal data bus was still 8-bit, it used a 16-bit address bus, so it could address 216 or 64 Kbytes of memory. Introduced in 1978, the 8086 improved on the 8085 by allowing 16 bits on the data bus and 20 bits on the address bus permitting a massive (at the time) 1 Mbyte of possible memory. In an unusual design decision, the addresses are formed by overlapping two 16-bit words to form a 20-bit address, rather than concatenating them to form a 32-bit address (see Figure 7-2). The 8086 was not 8085-compatible at the instruction-set level, but assembler macros helped convert programs to the newer architecture easily. Figure 7-2. How the Intel 8086 Addresses Memory

Its irregular addressing scheme allowed the 8086 to run 8085-ported code more simply; once the segment registers were loaded with a fixed value they could be ignored, allowing the 16-bit addresses of the 8085 to be used directly. The design team rejected the idea of forming addresses by concatenating the segment word; that would provide 32 bits of addressing or 4 Gbytes, which was an impossibly large amount of memory at the time. Now that this basic addressing model was laid down, all subsequent 80x86 processors had to follow it or give up compatibility. If the 8080 was the chip that brought Intel to prominence, the 8086 was the chip that kept it there. We'll probably never know exactly why IBM selected the Intel 8088 (an 8-bit sibling of the 8086) as the CPU for its new PC back in 1979, in the face of so many technically superior alternatives from companies like Motorola or National Semiconductor. By selecting an Intel chip, IBM made Intel's fortune for the next two decades, just as IBM also made Microsoft's fortune by selecting MS-DOS as the operating executive. Ironically, in August 1993 Intel's stock valuation of $26.6 billion rose above IBM's stock valuation of $24.5 billion, and Intel eclipsed IBM as the most valuable electronics company in America.

Intel and Microsoft have effectively become the new IBM, reaping undeserved windfall profits from their closed proprietary systems. IBM is trying desperately to regain its former position by using the PowerPC to challenge Intel's hardware monopoly, and using OS/2 to challenge Microsoft's software monopoly. The OS/2 challenge has probably failed, but it's too early to pass judgment on the PowerPC. The 8088 processor, as used in the first IBM PC, was just a cheap 8-bit version of the 8086, allowing the wealth of existing 8-bit support chips to be used. All further 80x86 refinements were of the "smaller, faster, costlier, and more instructions" variety. The 80186 took this path, introducing 10 notvery-important new instructions. The 80286 was just the 80186 (minus some built-in peripheral ports) but with the first attempt to extend the address space. It moved the memory controller off-chip, and provided an ambitiously named virtual mode in which the segment register isn't added to the offset, but is used to index a table holding the actual segment address. This kind of addressing was also known as protected mode and it was still 16-bit-based. MS-Windows uses 286 protected mode as its standard addressing mode. The 80386 is the 80286 with two new addressing modes: 32-bit protected mode and virtual 8086 mode. Microsoft's new flagship operating system, NT, and MS-Windows in enhanced mode both use 32-bit protected mode. This is why NT requires at least a 386 to run. The other kind of addressing, virtual 8086 mode, creates a virtual 8086 machine with 1 Mbyte of address space. Several of them can run at once, supporting multiple virtual MS-DOS sessions, each of which thinks it's running on its own 8086. At about this time, you should be thinking that the gyrations necessary to cope with the limitations of the original addressing scheme are pretty incredible, and you'd be right. The 80x86 is a difficult and frustrating architecture for which to write compilers and application programs. All these processors can have coprocessors, usually to implement floating point in hardware. The 8087 and 80287 coprocessors are identical, except that the 287 can address the same extended memory as the 286. The 387 can address using the same modes as the 386, but also adds some built-in transcendental functions.

Software Dogma

Choosing Components for the IBM PC Some, perhaps most, of the IBM decisions about the PC were definitely made on nontechnical grounds. Before deciding on MS-DOS, IBM arranged a meeting with Gary Kildall of Digital Research to consider CP/M. On the day of the meeting, so the story runs, the weather was so good that Gary decided to fly his private plane instead. The IBM managers, perhaps annoyed at being stood up, soon cut a deal with Microsoft instead. Bill Gates had bought the rights to Seattle Computer Product's QDOS, [1] cleaned it up a little, and renamed it "MS-DOS". The rest, as they say, is history. IBM was happy, Intel was happy, and Microsoft was very, very happy. Digital Research was not happy, and Seattle Computer Products became successively unhappier over the years as they realized they had pretty much given away the rights to the best-selling computer program ever. They did retain the right to sell MS-DOS if they sold the hardware at the same time, and this was why you used to see copies of MS-DOS available from Seattle Computer Products, improbably

bundled with alarmingly useless Intel boards and chips, to fulfill the letter of their contract with Microsoft. Don't feel too sorry for Seattle Computer Products—their QDOS was itself extensively based on Gary Kildall's CP/M, and he'd rather be flying. Bill Gates later bought a super-fast Porsche 959 with his cut of the profits. This car cost three-quarters of a million dollars, but problems arose with U.S. Customs on import. The Porsche 959 cannot be driven in the U.S.A. because it has not passed the government-mandated crash-worthiness tests. The car lies unused in a warehouse in Oakland to this day—one Gates product that will definitely never crash. [1]

This literally stood for "Quick and Dirty Operating System."

The 80486 is a repackaged 80386 that is a little faster because the bus lacks states that allow coprocessors. The 486 coprocessor is either built in or disallowed, called DX and SX, respectively. The 486 adds a few modest instructions and has an on-board cache (fast processor memory), which accounts for most of the rest of the performance improvement. That brings us to the present day, where, in a tremendous burst of innovation and trademark squabbling, Intel named its latest chip the Pentium, not the 80586. It's faster, more expensive, supports all previous instructions, and introduces some new ones. It's safe to anticipate that the 80686 is planned to be faster and more expensive, and will provide some additional instructions. Intel's internal motto for their continual introduction of new chips is "be fast or be dead," and they certainly live by it. As my old grandmother used to say as she worked away at her spinning wheel, "Those who do not remember history are doomed to have serious backward compatibility problems, especially if they change the addressing modes or wordsize of their architecture." The Intel 80x86 Memory Model and How It Got That Way As we saw in the previous chapter, the term segment has at least two different meanings (there's also a third OS memory-management-related meaning): A segment on UNIX is a section of related stuff in a binary. A segment in the Intel x86 memory model is the result of a design in which (for compatibility reasons) the address space is not uniform, but is divided into 64-Kbyte ranges known as segments. In its most basic form, a segment started out on the 8086 as a 64-Kbyte region of memory that was pointed to by a segment register. An address is formed by taking the value in a segment register and shifting it left four places (or equivalently, multiplying by 16). Yet a third way of looking at this is to consider that the segment register value has been made a 20-bit quantity by appending four zeros. Then the 16-bit offset says where the address is in that segment. If you add the contents of the segment register to the offset, you will obtain the final address. One quirk: just as there are many different pairs of numbers that total, for example, 24, there are many different segment + offset pairs that point to the same address. Handy Heuristic

Different-Looking Pointers, Same Address An address on the Intel 8086 is formed by combining a 16-bit segment with a 16-bit offset. The segment is shifted left four places before the offset is added. This means that many different segment/offset pairs can point to the same one address. offset

segment

resulting address

(shifted left 4 bits) A0000 :

+ FFFF

= AFFFF

AFFF0

+ 000F

= AFFFF

In general, there will be 0x1000 (4096) different segment/offset combinations that point to the same one address. A C compiler-writer needs to make sure that pointers are compared in canonical form on a PC, otherwise two pointers that have different bit patterns but designate the same address may wrongly compare unequal. This will be done for you if you use the "huge" keyword, but does not occur for the "large" model. The far keyword in Microsoft C indicates that the pointer stores the contents of the segment register and the offset. The near keyword means the pointer just holds a 16-bit offset, and it will use the value already in the data or stack segment register. Handy Heuristic

A Guide to Memory Prefix Use Prefix Kilo Mega Giga Tera Bubba

Power of Two 210 220 230 240 264

Meaning One thousand bytes One million bytes One billion bytes One trillion bytes Eighteen billion billion bytes

Number of Bytes 1,024 1,048,576 1,073,741,824 1,099,511,627,776 18,446,744,073,709,551,616

While on the subject of these numbers, note that all disk manufacturers use decimal rather than binary notation for disk capacity. Thus a 2-Gbyte disk will hold 2,000,000,000 bytes and not 2,147,483,648 bytes. A 64-bit address space is really large. It can fit an entire movie for high-definition TV in memory at once. They haven't yet settled the specification for high-definition TV, but it will probably be close to SVGA which is 1024 x 768 pixels, each of which needs, say, three bytes of color information. At 30 frames per second (as in the current NTSC standard) a two-hour movie would take: 120 minutes= x 60 seconds x 30 frames x 786,432 pixels x 3 color bytes = 509,607,936,000 bytes = 500Gbytes of memory So you could fit not just one, but 36 million high-definition TV movies (orders of magnitude more than every movie ever made, and then some) into a 64-bit virtual address space. You still have to leave room for the operating system, but that's OK. The UNIX kernel is constrained by the current SVID [1] to 512 Mbytes. Of course, there's still the small matter of physical disk to back up this virtual memory. [1] The SVID – System V Interface Definition – is a weighty document that describes the System V API.

The real challenge in computer architecture today is not memory capacity, but memory speed. Your brand new shiny red Pentium chip isn't going to win you anything if your software is actually constrained by disk and memory latency (access time). To be precise, there is a wide and increasing gap between memory and CPU performance. Over the past decade CPU's have doubled in speed every one-and-a-half to two years. Memory gets twice as dense (64-Kb chips increase to 128 Kb) in the same period, but its access time only improves by 10%. Main memory access time will be even more important on huge address space machines. When you have access to huge amounts of data, the latency for moving it around will start to dominate software performance. Expect to see a lot more use of cache and related technologies in the future. Handy Heuristic

Where the MS-DOS 640Kb Limit Comes From There's a hard memory limit of 640Kbytes available to applications that run under MSDOS. The limit arises from the maximum address range of the Intel 8086, the original DOS machine. The 8086 supported 20-bit addresses, restricting it to 1 Mbyte memory in total. That address space was further limited by reserving certain segments (64Kbyte chunks) for

system use:

F0000 to FFFFF

segment

reserved for 64 Kb for permanent ROM area BIOS, diagnostics, etc.

D0000 to EFFFF

128 Kb for cartridge ROM area

C0000 to CFFFF

64 Kb for BIOS extensions (XT hard disk)

B0000 to BFFFF

64 Kb for conventional memory display

A0000 to AFFFF

64 Kb for display memory extension

leaving 00000 to 9FFFF

640 Kb for application program use.

A billion and a trillion have different meanings in the U.S. and England. In the U.S. they are a thousand million (109) and a million million (1012), respectively. In England they are bigger, a million million (1012) and a million million million (1018)), respectively. We prefer the American usage because the magnitude increments are consistent from thousand (103) to million (106) to billion (109) to trillion (1012). A billionaire in England is much richer than a billionaire in the U.S.—until the exchange rate sinks to £1,000 per $1, that is. There's a 640 Kb limit in MS-DOS that arises from the 1 Mbyte total address space of the 8086 chip. MS-DOS reserves six entire segments for its own use. This only leaves the 10 64-Kbyte segments starting at address 0 available to applications (and the lowest addresses in 0 block are also reserved for system use as buffers and MS-DOS working store). As Bill Gates said in 1981, "640 K ought to be enough for anybody." When the PC first came out, 640Kb seemed like a tremendous amount of memory. In fact, the first PC came configured with only 16K of RAM as standard. Handy Heuristic

PC Memory Models Microsoft C recognizes these memory models: small

All pointers are 16 bits, limiting code and data to a single segment each, and the overall program size to 128 K. large All pointers are 32 bits. The program can contain many 64-K segments. medium Function pointers are 32 bits, so there can be many code segments. Data pointers are 16 bits, so there can only be one 64-K data segment. compact The other way around from medium: function pointers are 16 bits, so the code must be less than 64K. Data pointers are 32 bits so the data can occupy many segments. Stack data is still limited to a single 64-K segment, though.

Microsoft C recognizes these non-standard keywords; when applied to an object pointer or a function pointer, they override the memory model for that particular pointer only. __near A 16-bit pointer __far A 32-bit pointer, but the object pointed to must be all in one segment (no object may be larger than 64 K), i.e., once you load the segment register you can address all of the object from it. __huge A 32-bit pointer, and the restriction about all being in one segment is lifted.

Example: char __huge * banana; Note that these keywords modify the item immediately to their right, in contrast to the const and volatile type qualifiers which modify the pointer immediately to their left. In addition to the defaults, you can always explicitly declare near, far, and huge pointers in any model. Huge pointers always do comparisons and pointer arithmetic based on canonical [1] values. In canonical form, a pointer offset is always in the range 0 to 15. If two pointers are in canonical form, then an unsigned long comparison will produce accurate results. It is difficult and error-prone to compile the interaction between array and struct sizes, pointer sizes, memory models, and 80x86 hardware operating modes. [1]

We know, we know. We're using "canonical" in the canonical way.

As the spreadsheets and word processors gradually proved themselves, they placed ever-increasing demands on memory. People have devoted a tremendous amount of energy to coping with the limited address space of the IBM PC. A variety of memory expanders and extenders have been produced, but there is no satisfactory portable solution. MS-DOS 1.0 was essentially a port of CP/M to 8086. All later versions retained compatibility with the earliest one. This is why DOS 6.0 is still single-tasking and still uses the "real-address" (8086-compatible) mode of an 80x86, thus maintaining the limits on user program address space. The 8086 memory model has other undesirable effects. Every program that runs on MS-DOS runs with unlimited privilege, permitting easy attacks by virus software. PC viruses would be almost unknown if MS-DOS used the memory and task protection hardware built into every Intel x86 processor from the 80286 onwards. Virtual Memory If it's there and you can see it—it's real If it's not there and you can see it—it's virtual If it's there and you can't see it—it's transparent If it's not there and you can't see it—you erased it! —IBM poster explaining virtual memory, circa 1978

It is very inconvenient for a program to be restricted by the amount of main memory installed on a machine, as happens on MS-DOS. So early on in computing, the concept of virtual memory was developed to remove this restriction. The basic idea is to use cheap but slow disk space to extend your fast but expensive main memory. The regions of memory that are actually in use by a program at any given instant are brought into physical memory. When regions of memory lie untouched for a while, they are likely to be saved off to disk, making room to bring in other memory areas that are being used. All modern computer system from the largest supercomputers to the smallest workstations, with the sole exception of PC's, use virtual memory. Moving unused parts out to disk used to be done manually by the programmer, back in the early days of computing when snakes could walk. Programmers had to expend vast amounts of effort keeping track of what was in memory at a given time, and rolling segments in and out as needed. Older languages like COBOL still contain a large vocabulary of features for expressing this memory overlaying—totally obsolete and inexplicable to the current generation of programmers. Multilevel store is a familiar concept. We see it elsewhere on a computer (e.g., in registers vs. main memory). In theory, every memory location could be a register. In practice, this would be prohibitively expensive, so we trade off access speed for a cheaper implementation. Virtual memory just extends this one stage further, using disk instead of main memory to hold the image of a running process. So we have a continuum. Handy Heuristic

Processes on SunOS execute in a 32-bit address space. The OS arranges matters so that each process thinks it has exclusive access to the entire address space. The illusion is sustained by "virtual memory," which shares access to the machine's physical memory and uses disk to hold data when memory fills up. Data is continually moved from memory to disk and back again as a process runs. Memory management hardware translates virtual addresses to physical addresses, and lets a process run anywhere in the system's real memory. Application programmers only ever see the virtual addresses, and don't have any way to tell when their process has migrated out to disk and back into memory again, except by observing elapsed time or looking at system commands like "ps". Figure 7-3 illustrates the virtual memory basics. Figure 7-3. The Basics of Virtual Memory

Virtual memory is organized into "pages." A page is the unit that the OS moves around and protects, typically a few Kbytes in size. You can look at the pagesize on your system by typing /usr/ucb/pagesize. When a memory image travels between disk and physical memory we say it is being paged in (if going to memory) or paged out (going to disk). Potentially, all the memory associated with a process will need to be used by the system. If the process is unlikely to run soon (perhaps it is low-priority or is sleeping), all of the physical memory resources allocated to it can be taken away and backed up on disk. The process is then said to be "swapped out." There is a special "swap area" on disk that holds memory that has been paged or swapped. The swap area will usually be several times bigger than the physical memory on the machine. Only user processes ever page and swap. The SunOS kernel is always memory-resident. A process can only operate on pages that are in memory. When a process makes a reference to a page that isn't in memory, the MMU generates a page fault. The kernel responds to the event and decides whether the reference was valid or invalid. If invalid, the kernel signals "segmentation violation" to the process. If valid, the kernel retrieves the page from the disk. Once the page gets back into memory, the process becomes unblocked and can start running again—without ever knowing it had been held up for a page-in event. SunOS has a unified view of the disk filesystem and main memory. The OS uses an identical underlying data structure (the vnode, or "virtual node") to manipulate each. All virtual memory operations are organized around the single philosophy of mapping a file region to a memory region. This has improved performance and allowed considerable code reuse. You may also hear people talk about the "hat layer"—this is the "hardware address translation" software that drives the MMU. It is very hardware-dependent and has to be rewritten for each new computer architecture. Virtual memory is an indispensable technique in operating system technology now, and it allows a quart of processes to run in a pint pot of memory. The light relief section at the end of this chapter has an additional description of virtual memory, written as a fable. It's a classic. Programming Challenge

How Much Memory Can You Allocate? Run the following program to see how much memory you can allocate in your process.

#include <stdio.h> #include <stdlib.h> main() { int Mb = 0; while ( malloc(1 main() { int c; /* The terminal driver is in its ordinary line-at-a-time mode */ system("stty raw"); /* Now the terminal driver is in character-at-a-time mode */ c = getchar(); system("stty cooked"); /* The terminal driver is back in line-at-a-time mode */

} That last line—system("stty cooked");—is necessary because the terminal characteristics persist after the program finishes. If a program sets the terminal into a funny mode, it will stay in a funny mode. This is quite unlike, say, setting an environment variable, which disappears when the process does. Raw I/O achieves a blocking read—if no character is available, the process waits there until one comes in. If you need a nonblocking read, you can use the ioctl() (I/O control) system call. It provides a fine level of control over terminal characteristics, and can tell you if a key has been pressed under SVr4. This code uses an ioctl to only do a read if there is a character waiting to be read. This type of I/O is known as polling, as you continually ask the device for its opinion on whether it has a character to give you yet.

#include <sys/filio.h> int kbhit() { int i; ioctl(0, FIONREAD, &i); return i; /* return a count of chars available to read */ } main() { int i = 0; intc=''; system("stty raw -echo"); printf("enter 'q' to quit \n"); for (;c!='q';i++) { if (kbhit()) { c=getchar(); printf("\n got %c, on iteration %d",c, i); } } system("stty cooked echo"); } Handy Heuristic

Check errno After Library Calls Whenever you're using system calls (like ioctl()), it's a good idea to check the global

variable errno that is part of ANSI standard C. If a library or system call encounters problems, it will set errno to indicate the cause of the problem. However, the value of errno is only valid if there was a problem—the call will have some way of indicating this (usually by its return code). A typical use might be:

errno=0; if (ioctl(0, FIONREAD, &i) /* uses curses library, and the kbhit() function defined above */ main() { int c=' ', i=0; initscr(); /* initialize curses functions */ cbreak(); noecho(); /* do not echo pressed character */ mvprintw(0, 0, "Press 'q' to quit\n"); refresh(); while (c!='q') if (kbhit()) { c = getch(); /* won't block, as we know a character is waiting */ mvprintw(1, 0, "got char '%c' on iteration %d \n",c, ++i); refresh(); } nocbreak();

echo(); endwin(); /* finish curses */ } Compile this with cc foo.c -lcurses. Notice how much neater the output is when run under curses. There's a Nutshell book titled UNIX Curses Explained (which is not at all the book of programmer swear words most people think it is when they pick it up) that describes curses well. The curses library only offers character-based screen control func-tions. It's a lower common denominator than software written using specific bit-mapped graphics windowing libraries, but the curses software is far more portable. Finally, there is a non-polling read in which the operating system will send your process a signal each time it has some input ready. If a program uses interrupt-driven I/O, when it is not handling input it can be doing other processing in the main function. This is a very efficient use of resources if input is sporadic and there is much processing to be done. Interrupt-driven programs are much more complex and difficult to get working, but the paradigm enables a process to make productive use of time otherwise spent waiting for input. The use of threads diminishes the need to use interrupt-driven I/O techniques. Programming Challenge

Write an Interrupt-Driven Input Routine on Your System Interrupt-driven input is a breeze on MS-DOS. The system provides such spartan services that it is easy to brush them aside and pluck characters direct from the I/O port. Under SVr4, you will need to do the following: 1.

Create a signal handler routine that will be invoked to read a character when the OS sends a signal that one is ready. The signal to catch is SIGPOLL.

2.

The signal handler should read a character, and also reset itself as the handler for this signal each time it is invoked. Have it echo the character it just read, and quit if it was a 'q'. Note: this is just for teaching purposes. In practice the results are usually undefined if you call any standard library function from within a signal handler.

3.

Make an ioctl() call to inform the OS that you require a signal to be sent every time input comes in on the standard input. Look at the manpages for streamio. You will need a command of I_SETSIG and an argument of

S_RDNORM. 4.

Once the signal handler has been set up, the program can do something else until input comes in. Have it increment a counter. Print the value of the

counter in the handler routine.

Every time a character is sent from the keyboard, the SIGPOLL signal will be sent to the process. The signal handler will read the character, and reset itself to be the handler. Implementing a Finite State Machine in C A finite state machine is a mathematical concept that can be very useful when embodied in a program. It's a protocol for progressing through a limited ("finite") number of subroutines ("states"), each of which does some processing and then chooses the next state, usually based on the next piece of input. A finite state machine (FSM) can be used as the control structure of a program. FSMs are well-suited to programs that loop over several different alternative actions based on input. A coin-operated vending machine is a good candidate for an FSM. It will have states like "accept coin", "select item", "deliver item", and "make change". The inputs will be coins, and the outputs will be the items for sale. The basic idea is to have a table that holds all the possible states, and lists the actions to do when you enter each state. The last action is to calculate (often by a further table lookup based on the state you are in and the next input token) what state to enter next. You start in a state known as the "initial state." Along the way, your transition table might tell you to enter an error state, signifying an unexpected or erroneous input. You continue to make state transitions until you arrive at the end state. There are several ways to express an FSM in C, but most of them are based on an array of pointers to functions. An array of pointers to functions can be declared like this:

void ( *state[MAX_STATES] )(); If you know the function names, you can initialize the array like so:

extern int a(), b(), c(), d(); int (*state[])() = { a, b, c, d }; A function can be called through a pointer in the array like this:

(*state[i])(); The functions must all take the same arguments and have the same return value (unless you make the array element a union…). Pointers to functions are funny. Notice, too, how the pointer can be dropped, so our call can equally be made as

state[i](); or even

(******state[i])();

This is an unfortunate quirk popularized with ANSI C: calls to a function and calls to a function through a pointer (or any level of pointer indirection) can use the same syntax. There's a corresponding quirk applying to arrays. It further undermines the flawed "declaration looks like use" philosophy. Programming Challenge

Write an FSM Program Implement the C declaration analyzer from Chapter 3 as a finite state machine. 1.

Review the "decoder ring" diagram,

Figure 3-3

on page 76. This is a simple

state machine diagram! Program it this way, perhaps by modifying the cdecl program you wrote back in 2.

Chapter 3.

(You did write it, didn't you?)

First, write the code to control progress from state to state. Make each action routine simply print out the fact that it has been invoked. Debug this fully.

3.

Add the code to process and decode the input declaration.

The decoder ring is a simple state machine; most of the state transitions are in serial order regardless of the input. This means that you don't have to create a table of transitions matching state/input to get the next state. You can have a simple variable (of type pointerto-function). In each state, one of the things you will do is assign the next state. In the main loop, the program will call the function pointed at, and so on until the end function or an error state is reached. How does the FSM-based program compare with the non-FSM version in terms of ease of coding and debugging? In terms of ease of adding a different action, or modifying the order in which actions occur? If you want to get fancier, you can have the state function return a pointer to a generic successor function, which you cast to the appropriate type. Then you don't need a global variable. If you want to get less fancy, you can use a switch statement as a poor man's state machine, by assigning to the control variable and putting the switch inside a loop. One final point on FSMs. If your state functions seem to need a variety of arguments, consider using an argument count and an array of pointers to strings, just as the main routine does. The familiar int argc, char *argv[] mechanism is very general and can be borrowed with equal success for functions that you define. Software Is Harder than Hardware! Did you ever notice that software and hardware are named the wrong way round—software is easier to change, but harder in all other respects? Because software is so difficult to develop and get right, as programmers we need to find ways to make it as easy as possible. One way to do that (and it applies to

all languages, not just C) is to code for debuggability. When you write the program, provide the debugging hooks. Handy Heuristic

Debugging Hooks Did you know that most debuggers allow you to make function calls from the debugger command line? This can be very useful if you have complicated data structures. Write and compile a function to traverse the data structure and print it out. The function won't be called anywhere in the code, but it will be part of the executable. It is a "debugger hook." When you debug the code and you're stopped at a breakpoint you can easily check the integrity of your data structures by manually issuing a call to your print routine. Obvious once it's pointed out to you; not obvious if you've never seen it before. We already hinted at coding for debuggability in the previous section, where we suggested coding an FSM in two distinct phases: first do the state transitions, and only when they are working provide the actions. Don't confuse incremental development with "debugging code into existence"—a technique common among junior programmers, and those writing under too-strict time deadlines. Debugging code into existence means writing a fast slapdash first attempt, and then getting it working by successive refinements over a period of weeks by changing parts that don't work. Meanwhile, anyone who relies on that system component can pull their hair out. "Sendmail" and "make" are two well known programs that are pretty widely regarded as originally being debugged into existence. That's why their command languages are so poorly thought out and difficult to learn. It's not just you— everyone finds them troublesome. Coding for debuggability means breaking the system down into parts, and getting the program structure working first. Only when you have got the basic program working should you code the complicated refinements, the performance tweaks, and the algorithm optimizations. Handy Heuristic

Hash with Panache Hashing is a way to speed up access to an element in a table of data. Instead of searching

the table serially, you get a jumpstart to the likeliest element to contain your value. This is achieved by loading the table carefully, and not in serial order. What you do instead is apply some kind of transformation (known as a hashing function) on a data value from an element to be stored. The hashing function will yield a value in the range 0…tablesize-1, and that becomes the index where you try to store that record. If the slot is already taken, search forward from that point in the table for the next empty slot. Alternatively, you can set up a linked list hanging off that element, and simply add it to the end (either end, by the way). Or you can even hang a second hash table off the element. When you look up a data item, you don't need to start searching entries from element zero. Instead, again hash the value you want to locate, and start looking from that point in the table. Hashing is a tried and tested table lookup optimization, and it's used everywhere in systems software: in databases, operating systems, and compilers. If I were stranded on a desert island and could only take one data structure with me, it would be the hash table. A colleague had to write a program that at one point stored filenames and information about each file. The data was stored in a table of structs, and he decided to use hash lookup. Here's where the coding for debuggability came in. He didn't try to get every part of the program working in one swell foop. He got it working for the simplest case first, by making the hash function always return the constant zero. The function looked like this:

/* hash_file: Placeholder for more sophisticated future routine */ int hash_filename (char *s) { return 0; } The code that called it looked like this:

/* * find_file: Locate a previously created file descriptor or * make a new one if necessary. */ file find_filename (char *s) { int hash_value = hash_filename(s); file f; for (f = file_hash_table[hash_value]; f != NIL;f=f->flink) {

}

if (strcmp(f->fname, s) == SAME) { return f; }

/* file not found, so make a new one: */ f = allocate_file(s); f->flink = file_hash_table[hash_value]; file_hash_table[hash_value] = f; return f; } The effect was as though a hash table was not used; all the elements would be stored in a linked list off element zero. This made it simple to debug, because you didn't have to calculate where anything really should be. The ace programmer was able to quickly get the rest of the code working because he did not have to worry about the interaction with hashing. When he was satisfied that the main routines worked perfectly, he took some performance measurements, and decided to activate the hash function. This was a two-line change in a single function. Here's the current version involving, as he put it, "brain, pain, and gain".

int hash_filename (char *s) { int length = strlen(s); return (length+4*(s[0]+4*s[length/2])) % FILE_HASH; } Sometimes taking the time to break a programming problem into smaller parts is the fastest way of solving it. Programming Challenge

Write a Hash Program Type in the fragment of code above, and supply enough of the missing types, data, and code to get it running as a program. Then (horrors!) debug it into existence. How and Why to Cast The term "cast" has been applied since the dawn of C to mean both "type conversion" and "type disambiguation." If you say something like

(float) 3 it's a type conversion and the actual bits change. If you say

(float) 3.0 it's a type disambiguation, and the compiler can plant the correct bits in the first place. Some people say that casts are so-named because they help something broken to limp along.

It is easy to cast something to an elementary type: write the name of the new type (for example, int) in brackets before the expression you wish to cast. It is not quite so obvious how to cast to a more complicated type. Say you have a pointer to a void that you know actually contains a function pointer. How do you do the typecast and call the function all in one statement? Even complicated casts can be written following this three-step process. 1.

Look at the declaration of the object to which you wish to assign the casted result.

2.

Remove the identifier (and any storage class specifiers like extern), and enclose what remains in parentheses.

3.

Write the resulting text immediately to the left of the object you wish to cast.

As a practical example, programmers frequently discover that they need to cast to use the qsort() library routine. The routine takes four parameters, one of which is a pointer to a comparison routine. Qsort is declared as

void qsort(void *base, size_t nel, size_t width, int (*compar) (const void *, const void *)); When you call qsort() you will provide a pointer to your favorite comparison routine as argument compar. Your comparison routine will take an actual type rather than void * arguments, so will likely look somewhat like this:

int intcompare(const int *i, const int *j) { return(*i - *j); } This does not exactly match what qsort expects for argument compar() so a cast is required. [2] Let's assume we have an array a of ten integers to sort. Following the three step cast process outlined above, we can see that the call will look like [2] If you have a perverse and unpopular computer that makes the size of a pointer vary according to the type it points to, then you will have to do the cast in your comparison routine, rather than the call. Try to move to a better designed architecture as soon as possible.

qsort( a,

);

10, sizeof(int), (int (*)(const void *, const void *)) intcompare

As an impractical example, you can create a pointer to, for example printf(), with

extern int printf(const char*,...); void *f = (void*)printf; You can then call printf through a properly-cast pointer, in this manner:

(*(int(*)(const char*,...))f)("Bite my shorts. Also my chars and ints\n"); Some Light Relief—The Inter national Obfuscated C Code Competition The C language combines all the power of assembly language with all the ease-of-use of assembly language. —Ancient Peasant Proverb It's possible to abuse any programming language. Most good programmers can write programs that are so intense, it hurts your eyes just to look at them. Code that you can proudly show to programmers in the next office, and challenge them to figure out what it does. Code that, six months after writing it, you can't figure out what it does. You can write these kinds of programs in any language; it just seems to be easier with C. The International Obfuscated C Code Competition (IOCCC) is an annual contest run since 1984 over USENET by Landon Curt Noll and Larry Bassel. It started when Landon looked at the source for the Bourne shell and decided, "Nah! It's just too outré." He began to wonder how far you could go if you actively tried to make C code look confusing, rather than just achieving this as an accidental side effect. The competition has become an annual tradition. Entries are accepted in Winter, judging takes place over the Spring, and the winners are announced at the Summer Usenix conference. There are usually about ten categories of winner: "strangest abuse of the rules," "most creative source layout," "best oneliner," and so on. The overall "best of show" winner is whoever produces the most unreadable, and bizarre (but working) C program. The IOCCC is a lot of fun, and can extend your knowledge in surprising ways, whether you enter or merely analyze the prize-winning code afterward. For example, in 1987, David Korn of Bell Labs submitted this winning entry:

main() { printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"0x60);}

What does that print? (Hint: it's not "have fun"!) David wrote the eponymous Korn shell, which is widely regarded as much cleaner than the version 7 /bin/sh, so presumably the IOCCC also acts as a safety valve for the happy hacker. A 1988 winner was an obfuscated version of cdecl, submitted by programmer Gopi Reddy. Recall that we needed about 150 lines to program this unobfuscated. The obfuscated code is less than a dozen lines

#include<stdio.h> #include #define w printf #define p while #define t(s) (W=T(s)) char*X,*B,*L,I[99];M,W,V;D(){W==9?(w("`%.*s' is ",V,X),t(0)):W==40? (t(0),D(),t(41)):W==42?(t(0),D(),w("ptr to ")):0;p(W==40?(t(0), w("func returning "),t(41)):W==91?(t(0)==32?(w("array[0..%d] of ", atoi(X)-1),t(0)):w("array of "),t(93)):0);}main(){p(w("input: "), B=gets(I))if(t(0)==9)L=X,M=V,t(0),D(),w("%.*s.\n\n",M,L);}T(s) {if(!s||s==W) {p(*B==9||*B==32)B++;X=B;V=0;if(W=isalpha(*B)?9:isdigit(*B)?32 :*B++) if(W< = * and + are also used for boolean AND and boolean OR boolean expressions evaluate to 0 for false and 1 for true Editing: Line editor using line re-entry. A line number with nothing following it deletes the line. Input format: Free format positioning of tokens on the line. No space is allowed before the line number.

Exactly one space is needed between the OLD or SAVE command and the filename. ALL INPUT MUST BE UPPERCASE. Limits: Line numbers: Line length: FOR nesting: GOSUB: Program: Expressions:

1–10000 999 characters 26 999 levels Dynamically allocated -32768–32767 for 16-bit machines, -2147483648–2147483647 for 32-bit machines

Error checking / error reports: No error checking is performed. The message "core dumped" signifies a syntax or semantic error. Hosting environment: ANSI C, traditional K&R C ASCII or EBCDIC character set 48 Kbytes memory The sample BASIC program provided was the old lunar lander game:

10 REM Lunar Lander 20 REM By Diomidis Spinellis 30 PRINT "You are on the Lunar Lander about to leave the spacecraft." 60 GOSUB 4000 70 GOSUB 1000 80 GOSUB 2000 90 GOSUB 3000 100 H=H-V 110 V=((V+G)*10-U*2)/10 120 F=F-U 130 IFH>0THEN 80 135 H = 0 140 GOSUB 2000 150 IFV>5THEN 200 160 PRINT "Congratulations! This was a very good landing."

170 GOSUB 5000 180 GOTO 10 200 PRINT "You have crashed." 210 GOTO 170 1000 REM Initialise 1010 V = 70 1020 F = 500 1030 H = 1000 1040 G = 2 1050 RETURN 2000 REM Print values 2010 PRINT " Meter readings" 2015 PRINT " --------------" 2020 PRINT "Fuel (gal):" 2030 PRINT F 2040 GOSUB 2100 + 100 * (H 0) 2050 PRINT V 2060 PRINT "Height (m):" 2070 PRINT H 2080 RETURN 2100 PRINT "Landing velocity (m/sec):" 2110 RETURN 2200 PRINT "Velocity (m/sec):" 2210 RETURN 3000 REM User input 3005 IFF=0THEN 3070 3010 PRINT "How much fuel will you use?" 3020 INPUT U 3025 IFU (to get, or "extract") are used instead of functions like putchar() and getchar().

The > operators are still used for shift left and right as in C, but they are overloaded for C++ I/O. The compiler looks at the types of their operands to decide whether to generate code for a shift, or for I/O. If the leftmost operand is a stream, I/O was intended. Using operators, not functions, has four big advantages: •

The operators can be defined for every type. Thus, we do not need an individual function or (equivalently) a string format specifier like "%d" for each different type.

•

There is some notational convenience in using an operator rather than a function when you wish to output multiple messages. Just as you can write an expression i+j+k+l, the leftassociativity of the operator ensures that you can sensibly chain multiple I/O operands together:

•

cout class Fruit { public: void peel(){printf("peeling a base class fruit\n");} slice(); juice(); private: int weight, calories_per_oz; } ; When we declare a fruit object, and invoke the peel() method like this,

Fruit banana; banana.peel(); we will get the message

peeling a base class fruit So far, so good. Now consider deriving our apple class, and giving this its own method for peeling! After all, apples are peeled somewhat differently than bananas: you can peel a banana with your thumbs, but you need a knife to peel an apple. We know we can have methods with the same name, as C++ can cope with overloading.

class Apple : public Fruit { public: void peel() {printf("peeling an apple\n");} void make_candy_apple(float weight); }; Let's declare a pointer to a fruit, then make it point to an apple object (which inherits from the fruit class), and see what happens when we try peeling.

Fruit * p; p = new Apple; p->peel();

Whoa! If you try it, you'll get output like this:

% CC fruits.cpp % a.out peeling a base class fruit In other words, the Apple-specific method peel() wasn't called, the base class's peel() was! Explanation The reason is that C++ demands that you warn it when you start supplanting base class methods with derived class ones. You warn it by adding the keyword virtual to the base class method that you might be replacing. You can see now why we were reticent about explaining virtual back on page 301 where we first discovered it. You needed a lot of background information, which we have now covered. Handy Heuristic

Virtually Impractical Why isn't virtual the default? After all, you can always get the method from the base class by saying:

p->Fruit::peel(); It's pretty much for the same reason that C originally used the register keyword—it's a dumb optimization. So that every method call doesn't involve an extra indirection at runtime, you have to tell the compiler which ones do. The word virtual is a bit of a misnomer in this context. Elsewhere throughout computer science, "virtual" means letting the user see something that is not really there, and supporting the illusion by some means. Here, it means not letting the user see something that is really there (the base class method). A more meaningful (though impractically long) keyword would be

choose_the_appropriate_method_at_runtime_for_whatever_object_t his_is or more simply, placeholder.

How C++ Does Polymorphism Adding the keyword virtual to our base class method, and making no other changes results in

#include <stdio.h> class Fruit { public: virtual void peel(){printf("peeling a base class fruit\n");} slice(); juice(); private: int weight, calories_per_oz; } ; And the compilation and execution is

% CC fruits.cpp % a.out peeling an apple Exactly as desired. So far, this could all have been achieved at compile time, but polymorphism is a runtime effect. It refers to the process of C++ objects deciding at runtime which function should be called to carry out a particular operation. The runtime system looks at the object that has called the overloaded method, and chooses the method that goes with that class of object. If this is a derived object, we don't want it to call the base class version. We want it to call the derived class version, but this may not have been seen by the compiler when the base class was compiled. Therefore, this must be done dynamically at runtime, or, in C++ terminology, "virtually." Single inheritance is usually implemented by having each object contain a pointer vptr to a vector vtbl of function pointers. There is one of these vectors for each class, and there is one entry in the vector for each method in the class. In this way, the implementation code is shared by all objects of a given class. The vector is laid out so that a given function pointer lies at the same offset in the virtual tables for all subclasses of a class. Each method call can be mapped to a vtbl offset at compiletime. At runtime, the call is made indirectly through the pointer at the appropriate offset. Multiple inheritance requires a slightly more complicated scheme with another layer of indirection. If that didn't make sense, draw yourself a picture of it; it's the end of the line for this particular bus. Fancy Pants Polymorphism There are a lot more fancy tricks that you can pull with polymorphism, and sometimes it's downright essential. It makes a derived class's method preferred over those of the base class, but still allows the base class ones to be used if no derived ones have been defined. Sometimes a method does not know at compiletime whether it is operating on an object of its own class or one derived from it. Polymorphism says this has to work correctly.

main() { Apple apple;

Fruit orange; Fruit * p; p=&apple; p->peel(); p=&orange; p->peel();

}

At runtime, the results will be:

% a.out peeling an apple peeling a base class fruit Software Dogma

Deep Thought—Polymorphism Has Something in Common with Interposing Polymorphism and interposing both allow multiple functions to have the same one identifier. Interposing is a bit of a blunt instrument, as it binds every occurrence of the name to the same one definition at compile time. Polymorphism is a little more discerning, as it makes the binding decision on an object-by-object basis at runtime. Other Corners of C++ There are plenty of smaller C++ concepts not covered in this brief review of the high points. And there are plenty more detailed rules that apply to the concepts mentioned here. However, if you master the material in this chapter you will have a basic understanding of the OOP concepts and how they are expressed in C++. You will have enough of a head start to begin writing experimental C++ programs. And that is the real way, the only way, to learn any programming language. Among the concepts not covered here, C++ also has: •

Exceptions: borrowed from Ada and also from Clu (an experimental language developed at MIT, in which the key idea is a "cluster"). These are for changing the flow of control for error-handling. They simplify some kinds of error-handling by automatically diverting processing to a part of the program that can process the error.

•

Templates: support parameterized types. Like the class/object relationship, a template/function can be thought of as providing a "cookie-cutter" approach to algorithms. Once you get the basic algorithm down, you can plug different types into it. They are similar to the generic facility in Ada and parameterized modules in Clu. They have quite complicated semantics. This code:

•

template T min (T a, T b) { return (a < b) ? a : b; } allows one to assign any arbitrary type T (for which the < operator has been defined) to the variables a and b and the function min. Some people refer to templates as providing compile-time polymorphism. It's a bit of a stretch, but they mean that a stated operation can be done on a variety of different types, and it's all figured out at compiletime.

•

In-lining of functions: a programmer can stipulate that a particular function should be expanded in-line (as though it were a macro) in the instruction stream rather than generating a function call.

•

The operators new and delete, to replace malloc() and free() function calls. The operators are slightly more convenient (the sizeof calculation is done implicitly for example, and the proper constructor/destructor is called). new truly creates an object, whereas malloc just allocates memory.

•

Call-by-reference: C uses only call-by-value (except for arrays). C++ brings call-by-reference into the language.

Software Dogma

C++ Design Goals: That Was Then. This Is Now. From SIGPLAN Notices, vol. 21, no. 10, October 1986 "An Overview of C++," by Bjarne Stroustrup Section 6. What is Missing? C++ was designed under severe constraints of compatibility, internal consistency, and efficiency: no feature was included that 1.

[1] would cause a serious incompatibility with C at the source or linker levels.

2.

[2] would cause run-time or space overheads for a program that did not use it.

3.

[3] would increase run-time or space requirements for a C program.

4.

[4] would significantly increase the compile time compared with C.

5.

[5] could only be implemented by making requirements of the programming environment (linker, loader, etc.) that could not be simply and efficiently implemented in a traditional C programming environment.

Features that might have been provided but weren't because of these criteria include garbage collection, parameterized classes, exceptions, multiple inheritance, support for concurrency, and integration of the language with a programming environment. Not all of these possible extensions would actually be appropriate for C++, and unless great constraint is exercised when selecting and designing features for a language, a large, unwieldy, and inefficient mess will result. The severe constraints on the design of C++ have probably been beneficial and will continue to guide the evolution of C++. Ah, What years those were! The Reagan years, when tomato ketchup was a vegetable, trees were the major source of pollution, and C++ was assured of remaining unencumbered by parameterized classes, exceptions, and multiple inheritance. If I Was Going There, I Wouldn't Start from Here There is a property of programming languages known as orthogonality. This refers to the degree to which different features follow the same underlying principles. For example, in Ada, a programmer who learns how packages work will be able to apply this knowledge to generic packages, too. Unhappily, much of C++ is quite unorthogonal. Mastering one feature in C++ provides no clue or mental model that can be applied to other features. Most programmers will take the approach of only using a simpler subset of C++. Software Dogma

A Simple Subset of C++ C++ features to use: •

classes

•

constructors and destructors, but only with very simple bodies

•

overloading, including operator overloading and C++ I/O

•

single inheritance and polymorphic functions

C++ features to avoid using: •

templates

•

exceptions

•

virtual base classes

•

multiple inheritance

The major purpose of a programming language is to provide a framework for expressing problem solutions in terms which a computer can process. The better a language is at representing this discourse, the more successful it will be. The Fortran language, one of the first high-level languages, provided a powerful means of expressing mathematical formulae (the name "Fortran" means "Formula translation"). The COBOL language addressed itself to file processing, decimal arithmetic and output editing. And it is highly successful in that domain. C gave systems programmers access to many hardware-supported operators. The language didn't "get in the way" with many layers of abstraction. A language will be successful if its constructs are useful "building blocks" for solving problems in a given domain. Deciding on the "building blocks" of the language is the most important part of language design; the details, like choosing whether a semicolon is a terminator or a separator, cannot be ignored, but the building blocks are critical. How good a language C++ is will be decided on the basis of whether its features are good "building blocks" for solving interesting problems and whether the language can be reliably used by normal programmers. Some people claim that C++ classes will revolutionize software reuse. Reuse is a nebulous goal in software. Inheritance is not necessarily the panacea that it seems. Those with long memories are reminded of inflated claims made for Ada a decade ago. Let's make an analogy by saying that a computer program is like a book. Then you have libraries of both. And you want to reuse some routines in one of your programs. This corresponds to some paragraphs in the book. Software Dogma

Design Challenge: The C++ Machine In the past, some people have built special-purpose computer hardware that would be very efficient at executing a particular language: Algol-60: Lisp: Ada:

early Burroughs processors Symbolics Inc. Rational Computers

What would a C++ machine look like? Why have all special language processors come to a sticky end? This is a trick question—there's no common theme. The market for a single language is always less than for a general machine. Workstations ate the Lisp machine for lunch. The end of the Cold War killed the Ada machine. Burroughs ploughs on as part of Unisys.

The problem is that you can't create any kind of new worthwhile text by cutting and pasting entire paragraphs from other books. The level of abstraction is wrong. You can share text on the level of individual words or letters (this corresponds to individual lines of code or characters), but the effort involved in laboriously cutting them out is higher than the effort involved in deriving them afresh for the new work. And in just the same way, software reuse at the library level has empirically turned out to be less than originally envisioned. There is a small number of special-purpose routines that can be and are shared: mathematical libraries, a few data structure routines, and sorting and searching libraries. That's about it. These correspond to diagrams or reference tables in a book, which can be lifted wholesale and understood somewhere else. C++ may be more successful at software reuse than previous languages because its style of inheritance, based on objects, allows data to be inherited as well as code. Ada generics allow this too, but the Ada feature is cumbersome and too abstract for most programmers. To continue the analogy above, C++ makes it easier to check books out from the library, but you still have the problem of copying the relevant parts sensibly.

It May Be Crufty [4], but It's the Only Game in Town [4]

"Crufty /kruhf'tee/ [origin unknown] adj. possibly over-complex"—The New Hacker's Dictionary, ed. Eric Raymond, Cambridge, MA, MIT Press, 1991.

Having seen some of the serious failings in C in the first few chapters of this book, it would be really nice to say that C++ addresses these while retaining the flavor of C. It would be really nice, but it isn't going to happen, because it isn't true. C++ has some point improvements, but it retains many of the flaws of C, and piles up another big layer of complexity on top. The original C philosophy of "no features that need invisible runtime support" has been compromised. Software Dogma

Improvements in C++ Over C •

The error-prone construct of initializing a char array without enough room for the trailing nul is regarded as an error. char b[3]="Bob"; will cause an error in C++, but not in C.

•

A typecast can be written in the more normal-looking format of float(i) as well as the strange-looking C style of (float)i.

• • •

C++ allows a constant integer to define the size of an array.

const int size=128; char a[size]; is allowed in C++, but will generate an error message from C.

•

Declarations can be intermingled with statements, dropping the C requirement that all declarations precede all statements in a block. It's great that this arbitrary rule was dropped. Since this fix causes an incompatibility with C, why not go the whole way and provide a simpler alternative that fixes the horrible C declaration syntax, too?

Although C++ may be crufty, it's the only game in town. All the major players are behind it. All new development at AT&T is said to be in C++ now. The graphics part of Windows NT (which was later, slower and bigger than expected) was written in C++. Most new software development tools, applications libraries, and advanced technologies are now written in C++, or at least the ANSI C subset of it. But how long will it be before we start to see spectacular bugs like the AT&T network shutdown, caused or aggravated by features in C++ rather than C?

It doesn't matter. C++ will become widely used in spite of its flaws, and we hope it will eventually lead the way to something better. Handy Heuristic

Transitioning from C to C++ The best way to get started with C++ is to start programming in its ANSI C subset. Avoid the early translators based on cfront, which generated C code rather than machine code. Using C as a portable machine language really complicated linking and debugging, as cfront mangled all the function names to encode argument information. Name-mangling is a horrible kludge which will likely live on in C++ for a long time. Contrast this with Ada, which does it properly and does not define the semantics by a hacked implementation. Name-mangling is a hack for doing type checking between different files. But it implies that all your C++ rules must be complied with the same compiler as the name-mangling scheme may differ among compilers. This is a big defect in the C++ reuse model as it effectively prevents reuse at the binary level. Here is a representative sample of the ways that C is not a C++ subset, to give an idea of potential trouble spots. Restrictions in C++ that are not in C: •

The main() routine may not be called by user code in C++. This is permitted (but most unusual) in C.

•

Function prototypes are mandatory in C++, but optional in C.

•

Typedef names cannot clash with struct tags in C++, but they can in C (they belong to separate namespaces).

•

A cast is required to assign a void * pointer to another pointer type in C++; no cast is require d in C.

•

Features in C++ that mean something different in C:

•

There are more than a dozen new keywords in C++. These may be used as identifiers in a C program, but such use will usually generate an error message from a C++ compiler.

•

A declaration can appear anywhere a statement can in C++; in C, declarations must appear before statements in a block.

•

A struct name in an inner scope will hide an object name in an outer scope in C++, but not in C.

•

Character literals have type char in C++, but type int in C. That is, sizeof('a') yields 1 in C++ and a larger value in C.

•

Pathological cases involving the // comment convention of C++ (as shown in ).

Chapter 2

There are many more differences, but you now know enough to be dangerous. So go out there and be dangerous. When you're comfortable with the compiler and all the tools working in the ANSI C subset, then spread your wings and start defining your own classes. Choose a good C++ book—look at several, and choose one in a style you like. Make sure it is current with the language, which is still evolving. Make sure it covers exceptions and templates, which were the latest two things added. Just as with C, C++ standardization is now a joint effort of ISO (Working Group 21), and ANSI X3J16. The most optimistic estimates predict that it will take around six years to standardize the language, finishing in 1996, but make sure your C++ book mentions the ANSI C++ direction. So What Is a Protected Abstract Virtual Base Pure Virtual Private Destructor? Let's break this down and take it a little bit at a time. The phrase actually decomposes into two parts: a pure virtual private destructor that is inherited from a protected abstract virtual base. •

A private destructor is the function called when an object goes out of scope. "Private" means that it can only be called by a member or friend of the class.

•

A pure virtual function contains no code itself, but is used to act as a guide for other derived functions through inheritance.

•

A pure virtual destructor only makes sense if defined by a derived class. Since a destructor automatically does default clean-up actions on a class, like call member or base destructors, there is often no need to explicitly write any code in the destructor definition.

Simple enough. Tackling the second phrase •

An abstract virtual base means that the base class is shared by the multiplyinherited classes (it's a "virtual base"), and that it contains at least one pure virtual function, from which other classes derive through inheritance (an "abstract base"). Virtual base classes also have special initialization semantics.

•

A protected abstract virtual base class is one we inherited "protectedly," so our children know our parentage, but outsiders don't.

So, putting it all together, a protected abstract virtual base pure virtual private destructor is a destructor function, that •

can only be called by members or friends of the class, and

•

has no definition in the base class that declares it, but will be defined later in a derived class,

•

that (refering to the derived class) shares the multiply inherited base

•

which (refering to the base class) is inherited in a protected way.

And the last time we needed one was…well, we haven't yet! Does this start to remind anyone of the program proof of the Fast Fourier Transform? It certainly ranks alongside it in complexity. In C++ code, this might look like:

class vbc { protected: virtual void v()=0; private: virtual ~vbc()=0; // private destructor }; // vbc is an abstract class because it contains pure virtual functions classX:virtual protected vbc { // X inherits vbc virtually, and does it in a way such that // vbc's protected members are protected members of X. // So vbc is a "protected abstract virtual base" class of X. protected: void v() {} ~X() { /* do some X destruction */ } }; // When an X object is destroyed, X::~X is called, and then... // X's "protected abstract virtual base pure virtual private destructor" // is called too. So even though it's declared pure, it must be defined. These are the kind of semantics that gives C++ a reputation for being overly complicated. The problem is not any one feature, but rather the complexity of how all the different features interact. We'll stop at this point, and allow the reader to form his or her own conclusions.

Some Light Relief—The Dead Computers Society There are many and varied computer-related organizations, but the prize for most unusual surely goes to The Dead Computers Society!

Modeled after the "Dead Poets Society," which was actually an appreciation group for classical rhymesters, the Dead Computers Society is an appreciation group for computer architectures that no longer exist. It started as an informal discussion panel at the 1991 ASPLOS ("Architecture Support for Programming Languages and OS's") conference in Santa Clara, California. A group of friends and colleagues attending the conference noticed that many of them had worked on systems that were now discontinued. They decided to make light of this by forming the Dead Computers Society and holding an open forum round table on the issues involved. The hope was that an intelligent retrospective would allow future designers to learn from the lessons of the past. Membership of the Dead Computers Society is open to anyone who has helped design, build, or program a computer system that no longer exists, ideally for a company that no longer exists. There are a lot of these; a partial list is shown in Table 11-3.

Table 11-3. Dead Computers

The Dead Computers Honor Roll American Supercomputer Inc. Intel iPSC/1 Ametek/Symult Intel iPSC/2 Astronautics Intel/Siemens BiiN Burroughs BSP Masscomp/Concurrent CDC 7600, Cyberplus Multiflow CHoPP Myrias Culler Scientific Niche Cydrome Prisma Denelcor SCS Elxsi SSI Evans & Sutherland CD Star Technologies ETA/CDC Supertek FLEX (Flexible Computer) Suprenum/Siemens Goodyear Aerospace/Loral DataFlow Systems Texas Instruments ASC Guiltech/SAXPY Topologix Floating Point Systems AP-line and T-series Unisys ISP Intel 432 On the other hand, membership is also open to anyone who just thinks it's a kind of neat idea. At the inaugural meeting, there were over 350 attendees. The panel moderator tried to draw the members out on the "one single thing that, more than anything else, was responsible for your dead computer." The Elxsi designer said that they had tried to push the technology too much and used ECL (emitter-coupled logic) before it was ready for prime time. However, the chief architect from Multiflow, which went down the tubes around the same time as Elxsi, felt that their decision not to use ECL was one of several factors that ultimately caused Multiflow's demise! About the only consensus was that management and market conditions were responsible for many, many more bankruptcies than were technical failures. This is understandable; companies that don't

listen to their customers always go out of business. Companies that try to push the state of the art often succeed. There were some minor technical themes, like making your product hard to program (e.g. the CDC7600 with two-level memory, or one's complement arithmetic, or the cruel and unusual punishment of 60-bit words) doesn't help. It's not too surprising that no major common technical theme emerged. Maybe there isn't one. One thing is certain, though: we all learn far more from our mistakes than from our successes. Some Final Light Relief—Your Certificate of Merit! [Instructions: cut out from book, write your name in, and hand to boss]

Further Reading One C book I have found very helpful is C, A Reference Manual, written by Samuel P. Harbison and Guy L. Steele, ( Englewood Cliffs, Prentice Hall, 1991). Harbison and Steele wrote this book based on their experience developing a family of C compilers for a wide range of different architectures, and their practical insights shine off every page. Secrets of Programmer Job Interviews A little hardware knowledge is a dangerous thing. One programmer dismantled one of those novelty Christmas cards that plays a carol, and retrieved the piezoelectric melody chip. He secretly installed it into his boss's keyboard, and connected it to one of the LEDs. It turns out that the voltage over a lighted LED is sufficient to drive one of these chips.

Then I, oops, I mean "he", amended the system editor, so it turned on the LED when it started and turned it off when it exited. Result: the boss's terminal played continuous Christmas carols whenever he used the editor! The people in neighboring offices formed a lynch mob after half-an-hour and made him stop all work until the cause was uncovered. —The Second Official Handbook of Practical Jokes [1] [1]

By Peter van der Linden, © 1991 by Peter van der Linden. Used by permission of Dutton Signet, a Division of Penguin Books USA Inc.

…how can you detect a cycle in a linked list?… what are the different C increment statements for?… how is a library call …how is a file descriptor different from a file pointer?… write some code to determine if a variable is signed or not… what is the time complexity of printing the values in a binary tree?… give me a string at random from this file… some light relief—how to measure a building with a Silicon Valley programmer interviews different from a system call? barometer

Silicon Valley Programmer Interviews This provides some hints about the interview process for C programmers looking for a position with the top companies. One of the best things about the sharp end of the computer business is the unusual way we choose new employees to join the team. In a lot of industries, the supervisor or manager makes all the hiring decisions; indeed, is often the only one who even meets the job applicants. But on the leading edge of software development, and especially in high-tech start-ups, programmers are sometimes more qualified than managers to judge which of the candidates has the best technical skills as an "individual contributor." The talent needed to do some systems programming is so rare and specialized that sometimes technical ability is the single most important characteristic that you look for in an interview. So a distinctive style of interview has evolved. The manager follows company policies to select the pool of candidates to interview. These prospects are then given a technical grilling by every person on a development team, not just the manager. A typical job interview will last all day, involving successive hour-long sessions with six or seven different engineers—and all of them need to be convinced of how well the applicant can program before a job offer is made. Engineers often develop some favorite questions, and this chapter contains a collection of favorites. There's no harm in revealing these "secrets"—the kind of person who reads this book probably already knows enough to be hired by a good software company. Many of these problems had their origins in real algorithms we were trying to program, and have since been replaced by other, newer problems. Of course, part of what you look for when you interview people is not just what they respond with, but how they respond. Do they think carefully about a question and come up with several possibilities, or do they just blurt out the first response that comes into their head? How persuasively do they argue the case for their ideas? Are they insistent on an obviously wrong strategy, or do they have the flexibility to adapt their answer? Some of the following interview questions have yielded the strangest answers. Try them for yourself, and see how you fare! How Can You Detect a Cycle in a Linked List? This starts off as the simple question, "How can you detect a cycle in a linked list?" But the questioner keeps adding extra constraints, and it soon gets quite fiendish. Usual first answer:

Mark each element as you visit it; keep traversing; if you reach an already marked element, the list has a cycle in it. Second constraint: The list is in read-only memory, and you cannot mark elements. Usual second answer: As you visit each element, store its address in an array; check each successive element to see if it is already in the array. Sometimes poor candidates get hung up at this point on the details of hash tables to optimize array lookup. Third constraint: Uh-oh! There is a limited amount of memory, and it is insufficient to hold an array of the necessary size. However, you can assume that if there is a cycle, it is within the first N elements. Usual third answer (if the candidate gets this far): Keep a pointer at the beginning of the list. Check this pointer for a match against the next N - 1 elements in the list. Then move the pointer on one, and check against the next N - 2 elements. Keep doing this until a match is found, or all N elements have been compared to each other. Fourth constraint: Oh no! The list is arbitrarily long, and the cycle might be anywhere in it. (Even good candidates can get stuck at this point). Final answer: First, eliminate the pathological case of a 3-element list with a cycle from element 2 to 1. Keep two pointers. One at element 1 and the next at element 3. See if they are equal; if not, move P1 by one and P2 by two. Check and continue. If P1 or P2 is null, there is no loop. If there is one, it will definitely be detected. One pointer will eventually catch up with the other (i.e., have the same value), though it might take several traversals of the cycle to do it. There are other solutions, but the ones above are the most common. Programming Challenge

Knocked for a Loop

Convince yourself that the algorithm in the final answer above will detect a loop, if there is one. Make a loop; dry run your code; make the loop longer; dry run again; repeat until the induction condition jumps out at you. Also, convince yourself that the algorithm will terminate if there is no loop. Hint: write the program, and extrapolate from there. What Are the Different C Increment Statements For? Consider the four C statements below:

x = x+1; ++x; x++; x += 1;

/* /* /* /*

regular */ pre-increment */ post-increment */ assignment operator */

Clearly, these four statements all do the same thing, namely, increase the value of x by one. As shown here, with no surrounding context, there is no difference between any of them. Candidates need to (implicitly or explicitly) supply the missing context in order to answer the question and distinguish between the statements. Note that the first statement is the conventional way of expressing "add one to x" in an algorithmic language. Hence, this is the reference statement, and we need to find unique characteristics of the other three. Most C programmers can immediately point out that ++x is a pre-increment that adds one to x before using its value in some surrounding expression, whereas x++ is a post-increment that only bumps up x after using its original value in the surrounding expression. Some people say that the "++" and "--" operators were only put in C because *p++ is a single machine instruction on a PDP-11 (the machine for which the first C compiler was written). Not so. The feature dates from the B language on the PDP-7, but it turns out that the increment and decrement operators are incredibly useful on all hardware. Some programmers give up at this point, overlooking that x+=1 is useful when x is not a simple variable, but an expression involving an array. If you have a complicated array reference, and you want to demonstrate that the same index is used for both references, then

node[i >> 3] += ~(0x01 = 0 && ~a >= 0) Alternatively, assuming the argument is to be a type, one answer would use type casts:

#define ISUNSIGNED(type) ((type)0 - 1 > 0) The correct interpretation is key here! Listen carefully, and ask for a better explanation of any terms that you don't understand or weren't well defined. The first code example only works with K&R C. The new promotion rules mean it doesn't work under ANSI C. Exercise: explain why, and provide a solution in ANSI C. Most Microsoft questions have some element of determining how well you can think under pressure, but they are not all overtly technical. A typical nontechnical question might be, "How many gas stations are there in the U.S.?" or "How many barber shops are there in the U.S.?" They want to see if you can invent and justify good estimates, or suggest a good way of finding more reliable answers. One suggestion: call the state licensing authorities. This will give you exact answers with only 50 phone calls. Or call half-a-dozen representative states, and interpolate from there. You could even respond as one environmentally-conscious candidate did, when asked "How many gas stations?" "Too many!" was her annoyed response. What Is the Time Complexity of Printing the Values in a Binary Tree? This question was asked during an interview for a position on a compiler team with Intel. Now, the first thing you learn in complexity theory is that the notation O(N) means that as N (usually the number of things being processed) increases, the time taken increases at most linearly. Similarly, O(N2) means that as N increases, the processing time increases much, much faster, by up to N-squared in fact. The second thing that you learn in complexity theory is that everything in a binary tree has O(log(n)), so most candidates naturally give this answer without even thinking about it. Wrong! This turned out to be a question similar to Dan Rather's famous "What is the frequency, Kenneth?" question—an enquiry designed to discomfit, confuse, and annoy the listener rather than solicit information. To print all the nodes in a binary tree you have to visit them all! Therefore, the complexity is O(n). Colleagues have reported similar chicanery in an interview question for electronic engineers at Hewlett-Packard. The question is posed in the form of a charged and uncharged capacitor suddenly connected together, in an ideal circuit with no resistance. Mechanical engineers were asked the same question about two massless springs displaced from equi-librium and then released. The interviewer

then derives two inconsistent end-states using two different physical laws (i.e., conservation of charge and conservation of energy in the case of the capacitors), and queries the hapless prospect on the cause of the inconsistency. The trick here is that at least one expression of the end-state uses a formula that involves integration over the event separating the start and end conditions. In the real world this is fine, but in the theoretical experiment posed it causes integration over a discontinuity (since the moderating effects have been idealized away); hence, the formula is useless. Engineers are most unlikely to have encountered this before. Yep, those massless springs and circuits without resistance will get you every time! Yet another curve ball was pitched in an interview for a position as a software consultant with a large management consulting company. "What does the execve system call return, if successful?" was the question. Recall that execve() replaces the calling process image with the named executable, then starts executing it. There is no return from a successful execve hence there is no return value. Trick questions are fun to pose to your friends, but they don't really belong in an interview. Give Me a String at Random from This File Another favorite Microsoft question. The interviewer asks the applicant to write some code that selects one string at random from a file of strings. The classic way of doing this is to read the file, counting the strings and keeping a record of the offset at which each begins. Then pick a random number between 1 and the total number, go to the corresponding offset in the file, and that's your string. What makes it very hard to solve is that the interviewer sets the condition that you can only make one sequential pass through the file, and you may not store any additional information like a table of offsets. This is another of those questions where the interviewer is mostly interested in seeing how you solve problems. He or she will feed you hints if you ask for them, so most people eventually get it. How well you do depends on how quickly you get it. The basic technique is to pick survivors, and recalculate as you go along. This is so computationally inefficient that it's easy to overlook. You open the file and save the first string. At this point you have one candidate string with a 100% probability of picking it. While remembering the first string, you read the next string. You now have two candidates, each with a 50% probability. Pick one of them, save it, and discard the other. Read the next string, and choose between that and the saved string, weighting your choice 33% toward the new string and 67% toward the survivor (which represents the winner of the previous two-way selection). Save the new survivor. Keep on doing this throughout the entire file, at each step reading string N, and choosing between that (with probability 1/N) and the previous survivor, with probability (N - 1)/N. When you reach the end of the file, the current survivor is the string picked at random! This is a tough problem, and you either have to figure out the answer with the minimum of hints, or have cleverly prepared yourself by reading this book. Some Light Relief—How to Measure a Building with a Barometer We find these kinds of questions so much fun that we even pose them to ourselves in a non-computing context. Sun has a "junk mail" e-mail alias for employees to share thoughts of random interest; occasionally people will pose problems on this alias, and challenge other engineers to compete in submitting the best answer. Here is one such puzzle, recently set.

There is an old story about a physics student finding novel ways to measure the height of a building using a barometer. The story is retold by Alexander Calandra in The Teaching of Elementary Science and Mathematics. [3] [3]

St. Louis, Washington University, 1961.

A student failed an exam after he refused to parrot back what he had been taught in class. When the student protested, I was asked to act as arbiter. I went to the professor's office and read the examination question: "Show how it is possible to determine the height of a tall building with the aid of a barometer." The student had answered: "Take the barometer to the top of the building, attach a long rope to it, lower the barometer to the street and then bring it up, measuring the length of the rope. The length of the rope is the height of the building." A high grade is supposed to certify competence in physics, but the answer did not confirm this. I suggested that the student have another try at answering the question. I gave the student six minutes, with the warning that his answer should show some knowledge of physics. In the next minute he dashed off his answer, which read: "Take the barometer to the top of the building and lean over the edge of the roof. Drop the barometer, timing its fall with a stopwatch. Then, using the formula for the distance travelled by a falling object, S = 1/2 a t2 calculate the height of the building." At this point, I gave the student full credit. The student went on to propose three other methods of measuring a building's height with a barometer: Take the barometer out on a sunny day and measure the height of the barometer, the length of its shadow, and the shadow of the building. By the use of a simple proportion, determine the height of the building. Take the barometer and begin to walk up the stairs. As you climb the stairs, mark off the length of the barometer along the wall. Then count the number of marks, and this will give you the height of the building in barometer units. Last (and probably least) offer the barometer as a gift to the building superintendent if he will just tell you how high the building is. When this old story was rehashed at Sun as a "science puzzler," 16 more ingenious new methods for barometric building measurement were suggested! The new responses were of the following types: The Pressure Method: Measure the air pressure at the bottom and top of the building, then compute the height of the building from the pressure difference. This is the only method that actually uses the barometer for its designed purpose of measuring air pressure. Although aircraft altimeters often work by this principle, it is one of the least accurate methods for measuring a building's height. The Pendulum Method: Go onto the building's roof, and lower the barometer on a string until it almost reaches the ground. Swing the barometer and measure the pendulum's period of oscillation. From this the length of the pendulum, and hence the building, can be calculated. The Avarice Method: Pawn the barometer to raise seed money for a chain letter campaign. Then stack the currency thus obtained against the building, measuring it in units of currency thickness. No comment on how to stay ahead of the police long enough to measure the building.

The Mafia Method: Extort the building's height from the superintendent, using the barometer as a weapon. The Ballistic Method: Fire the barometer from ground level with a mortar, just high enough to reach the top of the building. You may need to take some ranging shots to get the explosive charge and firing elevation just right. A table of standard ballistic calculations will then yield the height reached by the projectile. The Paperweight Method: Use the barometer as a paperweight while looking over the building plans. The Sonic Method: Drop the barometer from the top of the building, and time the interval between seeing the barometer hit the ground and hearing it. For practical distances, the sight will be instantaneous, and the sound will travel at the speed of sound (1150 feet/second at standard temperature-pressure and mean sea level), giving the height. The Reflective Method: Use the glass face of the barometer as a mirror and time the interval it takes light to traverse the round trip between the top of the building and the ground. Since the speed of light is a known quantity, the distance can be calculated. The Mercantile Method: Sell the barometer, and buy some proper equipment. The Analog Method: Attach the barometer to a string, wind the string around the shaft of a small generator, and measure the amount of electrical energy produced as the barometer falls from the top of the building to the ground. The generated energy is proportionate to the number of revolutions of the shaft, and hence the distance to the ground. The Trigonometric Method: Pick a spot on the ground a known distance from the building, go to the top of the building with the barometer and a protractor level, and wait for the sun to reach the horizon. Then, using the barometer as a mirror, aim a spot of sunlight at the place you previously picked and measure the angle of the mirror with the level. Use trigonometry to calculate the building's height. The Proportion Method: Measure the height of the barometer. Bring a friend and a tape measure. Stand at a known distance from the building and sight past the barometer to the building. Move the barometer toward or away from you until the top and bottom of the barometer line up with the top and bottom of the building. Then have your friend measure the distance from the barometer to your eye. Compute the building height by proportion. The Photographic Method: Set up a tripod with a camera at a known distance from the building. Hold the barometer a known distance from the camera and take a picture. From the relative heights of the barometer and the building in the picture, you can compute the height of the building. The Gravitational Method I: Suspend the barometer from a yard of string. Measure the oscillation period of the pendulum thus formed at the top and bottom of the building. Compute the building's height from the difference in gravitational acceleration. The Gravitational Method II: Weigh the barometer at the top and bottom of the building with a spring balance (a beam balance won't work for this!). Differences in the readings will be due to the different gravitational acceleration, due to the differing distances from Earth. (A reader tells me that the Lacoste Romberg gravimeter provides a practical way of getting the required accuracy.) Compute the building's height from the two measurements.

The Calorific Method: Drop the barometer from the roof into a container of water covered with a narrow slit to minimize splashing. The rise in temperature of the water is an analog of the energy that the barometer had on impact, and hence the height it fell from. And you thought questions like this only came up in algebra problems! Further Reading If you have enjoyed this book you might also enjoy the text Bartholomew and the Oobleck, by Dr. Seuss, ( New York, Random House, 1973). As Dr. Seuss explains, Oobleck "squiggles around in your fist like a slippery potato dumpling made of rubber." He doesn't tell you how to make it, so I will here. How to make Oobleck 1.

Take one cup of cornstarch.

2.

Add some drops of green food coloring; Oobleck should always be green in color.

3.

Slowly knead in up to half a cup of water.

Oobleck has some extremely strange, non-Newtonian properties. It flows like water through your fingers, except when you work it like dough—then it immediately assumes a solid consistency. Except when you stop kneading it, it turns back to liquid. If you strike it quickly with something hard, you can shatter it! Like all Dr. Seuss books, Bartholomew and the Oobleck can be read and enjoyed on several levels. For example, One Fish Two Fish, Red Fish Blue Fish can be deconstructed as a searing indictment of the narrow-minded binary counting system. Software engineers can benefit from a sophisticated reading of Bartholomew and the Oobleck. The world would be a better place if every programmer would just play with Oobleck occasionally. The good programmers would be more rested and refreshed and the bad programmers would probably get their heads stuck to the table. But always remember, no matter how lowly or exalted a hacker you are, you are a child process of the universe, no less than the disk controller, or the stack frame mechanism (explained at such length in Chapter 6). They say that when you stare deeply into an abyss, the abyss also stares into you. But if you stare deeply into this book, well, it's rude to stare, plus you might get a headache or something. I can't think of a job I'd rather do than computer programming. All day, you create patterns and structure out of the formless void, and you solve dozens of smaller puzzles along the way. The wit and ingenuity of the human brain is pitted against the remorseless speed and accuracy of the electronic one. Thus has it ever been. Humanity's higher purpose is to strive, to seek, to create. Each and every computer programmer should seek out and grasp opportunities to—whoa! That's your 75,000 words already—