,
Dr. Dobbs J O U R N A L
#369 FEBRUARY 2005
SOFTWARE TOOLS FOR THE PROFESSIONAL PROGRAMMER
http://www.ddj.com
WEB SERVICES The Java Web Services Developer Pack Amazon.com Web Services & Ruby GIS Web Services & MS Word SOAs & ESBs Building the DDJ Eclipse Web Search Plug-In Enhancing .NET Web Services Report Services & ASP.NET Whidbey C++ Syntax
Automating Localization Java Cryptography &
X.509 Authentication Algorithms For Dynamic Shadows Extending UML Inside the uIP Stack
C O N T E N T S FEATURES Java Web Services & Application Architectures 16
FEBRUARY 2005 VOLUME 30, ISSUE 2
by Eric J. Bruno
Eric use the Java Web Service Developer Pack to build a financial portal.
SOAs & ESBs 24 by James Pasley
James examines Service-Oriented Architectures and the role of the Enterprise Service Bus when deploying SOAs.
Ruby/Amazon & Amazon Web Services 30 by Ian MacDonald
The Ruby/Amazon toolkit is a high-level abstraction of Amazon Web Services for the Ruby programming language.
GIS Web Services and Microsoft Word 36 by Keith Bugg
Here’s a Geographical Information Systems application that lets you embed a map into a Microsoft Word document.
Java Cryptography & X.509 Authentication 40 by Snezana Sucurovic and Zoran Jovanovic
Our authors use the Java Crypto API to implement X.509 authentication in a distributed system.
Building an Eclipse Web-Search Plug-In 43 by Michael Pilone
This Eclipse plug-in lets you search DDJ’s online archives.
Automating Localization 47 by Hew Wolff
Converting existing systems to local languages can be a laborious process — making it ideal for automation.
Algorithms for Dynamic Shadows 50 by Sergei Savchenko
Geometrically correct dynamic graphic shadows can add realism to your graphical applications.
Extending UML 56 by Timothy E. Meehan and Norman Carr
Extended Activity Semantics (XAS) notation lets you model user interactions by extending UML.
Integrating Reporting Services into ASP.NET S1 by David Lloyd
David designs and deploys a report using Reporting Services, a free add-on to SQL Server 2000.
New Syntax C++ in .NET Version 2 S6 by Richard Grimes
Richard examines how generics are declared in Whidbey C++ and how they relate to C++ templates.
Enhancing .NET Web Services S12 by Eric Bergman-Terrell
SoapEx demonstrates how you can modify SOAP requests/responses and debug web-service calls with a network packet analyzer.
EMBEDDED SYSTEMS Inside the uIP Stack 61 by Drew Barnett and Anthony J. Massa
uIP lets you to add network support to small form factor, DSP-based embedded systems.
COLUMNS Programming Paradigms 66 by Michael Swaine
Embedded Space 69
FORUM EDITORIAL 8 by Jonathan Erickson LETTERS 10 by you Dr. Ecco's Omniheurist Corner 12 by Dennis E. Shasha NEWS & VIEWS 14 by Shannon Cochran OF INTEREST 78 by Shannon Cochran SWAINE’S FLAMES 80 by Michael Swaine
RESOURCE CENTER As a service to our readers, source code, related files, and author guidelines are available at http:// www.ddj.com/. Letters to the editor, article proposals and submissions, and inquiries can be sent to
[email protected], faxed to 650-513-4618, or mailed to Dr. Dobb’s Journal, 2800 Campus Drive, San Mateo CA 94403. For subscription questions, call 800-456-1215 (U.S. or Canada). For all other countries, call 902-563-4753 or fax 902-563-4807. E-mail subscription questions to ddj@neodata .com or write to Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 803226188. If you want to change the information you receive from CMP and others about products and services, go to http://www.cmp .com/feedback/permission.html or contact Customer Service at the address/number noted on this page. Back issues may be purchased for $9.00 per copy (which includes shipping and handling). For issue availability, send e-mail to
[email protected], fax to 785838-7566, or call 800-444-4881 (U.S. and Canada) or 785-8387500 (all other countries). Back issue orders must be prepaid. Please send payment to Dr. Dobb’s Journal, 4601 West 6th Street, Suite B, Lawrence, KS 66049-4189. Individual back articles may be purchased electronically at http://www.ddj.com/.
by Ed Nisley
Chaos Manor 72 by Jerry Pournelle
NEXT MONTH: The future is here, as we March into the world of 64-bit computing.
Programmer’s Bookshelf 77 by Martin Heller
DR. DOBB’S JOURNAL (ISSN 1044-789X) is published monthly by CMP Media LLC., 600 Harrison Street, San Francisco, CA 94017; 415-947-6000. Periodicals Postage Paid at San Francisco and at additional mailing offices. SUBSCRIPTION: $34.95 for 1 year; $69.90 for 2 years. International orders must be prepaid. Payment may be made via Mastercard, Visa, or American Express; or via U.S. funds drawn on a U.S. bank. Canada and Mexico: $45.00 per year. All other foreign: $70.00 per year. U.K. subscribers contact Jill Sutcliffe at Parkway Gordon 01-49-1875-386. POSTMASTER: Send address changes to Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 80328-6188. Registered for GST as CMP Media LLC, GST #13288078, Customer #2116057, Agreement #40011901. INTERNATIONAL NEWSSTAND DISTRIBUTOR: Worldwide Media Service Inc., 30 Montgomery St., Jersey City, NJ 07302; 212-332-7100. Entire contents © 2005 CMP Media LLC. Dr. Dobb’s Journal is a registered trademark of CMP Media LLC. All rights reserved.
http://www.ddj.com
Dr. Dobb’s Journal, February 2005
5
,
Dr.Dobbs J O U R N A L
PUBLISHER Michael Goodman
SOFTWARE TOOLS FOR THE PROFESSIONAL PROGRAMMER
EDITOR-IN-CHIEF Jonathan Erickson
EDITORIAL MANAGING EDITOR Deirdre Blake MANAGING EDITOR, DIGITAL MEDIA Kevin Carlson SENIOR PRODUCTION EDITOR Monica E. Berg NEWS EDITOR Shannon Cochran ASSOCIATE EDITOR Della Wyser ART DIRECTOR Margaret A. Anderson SENIOR CONTRIBUTING EDITOR Al Stevens CONTRIBUTING EDITORS Bruce Schneier, Ray Duncan, Jack Woehr, Jon Bentley, Tim Kientzle, Gregory V. Wilson, Mark Nelson, Ed Nisley, Jerry Pournelle, Dennis E. Shasha EDITOR-AT-LARGE Michael Swaine PRODUCTION MANAGER Douglas Ausejo INTERNET OPERATIONS DIRECTOR Michael Calderon SENIOR WEB DEVELOPER Steve Goyette WEBMASTERS Sean Coady, Joe Lucca AUDIENCE DEVELOPMENT AUDIENCE DEVELOPMENT DIRECTOR Kevin Regan AUDIENCE DEVELOPMENT MANAGER Karina Medina AUDIENCE DEVELOPMENT ASSISTANT MANAGER Shomari Hines AUDIENCE DEVELOPMENT ASSISTANT Melani Benedetto-Valente MARKETING/ADVERTISING MARKETING DIRECTOR Jessica Hamilton ACCOUNT MANAGERS see page 79 Michael Beasley, Cassandra Clark, Ron Cordek, Mike Kelleher, Andrew Mintz, Erin Rhea, SENIOR ART DIRECTOR OF MARKETING Carey Perez DR. DOBB’S JOURNAL 2800 Campus Drive, San Mateo, CA 94403 650-513-4300. http://www.ddj.com/ CMP MEDIA LLC Gary Marshall President and CEO John Day Executive Vice President and CFO Steve Weitzner Executive Vice President and COO Jeff Patterson Executive Vice President, Corporate Sales & Marketing Mike Mikos Chief Information Officer William Amstutz Senior Vice President, Operations Leah Landro Senior Vice President, Human Resources Mike Azzara Vice President/Group Director Internet Business Sandra Grayson Vice President & General Counsel Alexandra Raine Vice President Communications Robert Faletra President, Channel Group Vicki Masseria President CMP Healthcare Media Philip Chapnick Vice President, Group Publisher Applied Technologies Michael Friedenberg Vice President, Group Publisher InformationWeek Media Network Paul Miller Vice President, Group Publisher Electronics Fritz Nelson Vice President, Group Publisher Network Computing Enterprise Architecture Group Peter Westerman Vice President, Group Publisher Software Development Media Joeseph Braue Vice President, Director of Custom Integrated Media Solutions Shannon Aronson Corporate Director, Audience Development Michael Zane Corporate Director, Audience Development Marie Myers Corporate Director, Publishing Services
American Buisness Press
6
Dr. Dobb’s Journal, February 2005
Printed in the USA
http://www.ddj.com
EDITORIAL
Looking Back, Catching Up
I
n the spirit of “if it was important enough to mention once, it’s worth mentioning twice,” it’s time to look back and catch up on some of the items I’ve previously wasted this space on. In the November 2004 issue, for instance, I talked about how Teresis (a Los Angeles-based provider of digital workflow solutions for television) and Rapidtext (a national transcription services company) teamed up to bring jobs to Marshall, Missouri. As it turns out, the Marshall Plan isn’t the only kid on the back forty, at least in terms of outsourcing jobs to rural America. Rural Sourcing Inc. (http://www.ruralsource.com/) is a Jonesboro, Arkansas, startup founded in 2003 with the goal of supporting economic expansion by creating high-technology employment opportunities in rural America. Rural Sourcing, founded by Kathy White (former CIO of Cardinal Health, one of Information Week’s top 10 CIOs in 1997, and former associate professor of information technology at the University of North Carolina, Greensboro), has set up two facilities in Arkansas, one in New Mexico, and a soon-to-be-opened operation in North Carolina. In March 2002, I looked at the prospects of wind-generated electricity. At the time, wind turbines in the U.S. were generating about 4500 megawatts of electricity. The good news is that, by the end of 2003, U.S. capacity reached 6374 megawatts, and utility wind-power projects underway will create another 3000 megawatts of wind capacity in the U.S. over the next five years. According to some estimates, producing more than 1 million megawatt-hours of electricity by the 170 wind turbines at three-year-old Gray County Wind Farm in Montezuma, Kansas, alone would have required 606,000 tons of coal, or more than 12 billion cubic feet of natural gas. Moreover, fossil-fuel plants generating the same amount of electricity would have released into the air 1 million tons of carbon dioxide, 2650 tons of sulfur dioxide, 2170 tons of nitrogen oxides, and 52 pounds of mercury. In October 2004, I pointed out that iTunes downloads were pushing iPod sales, while iPods sales were driving Macintosh sales. More interestingly, iPods seemed to be propelling the migration of Windows users to Macs. According to a survey by financial firm Piper Jaffray, this trend is continuing, with 6 percent of iPod users switched from Windows-based PCs to Macs, and another 7 percent saying they are planning to. Factors influencing the decision to switch include ease of use, the entertainment value, and the perception of better security. Last month, I prattled on about how new programming languages seemed to pop up all the time. While Caml, Delphi, and Python aren’t new, there is news about them. For starters, Borland has recently released Delphi 2005 (http://www.borland.com/delphi/), which lets you target both Win32 and .NET. Second, Python 2.4 (http://www.python.org/2.4/) has finally been released with a boatload of new features, ranging from decimal data types to multiline imports. Third, it’s okay to admit that you don’t know much about Caml, a functional language developed in 1984 by INRIA (a French computer science research institute). However, you can learn more about it in Developing Applications with Objective Caml, a book by Emmanuel Chailloux, Pascal Manoury, and Bruno Pagano that was recently translated into English from French and made freely available in PDF or HTML format (http://caml.inria.fr/oreilly-book/). Also in the January 2005 issue, I made reference to a recent DDJ survey. In truth, we routinely participate in a number of studies. The NOP/CMP Professional Developer Panel, for instance, surveys several hundred software engineers, many of whom are DDJ readers. To me, one of the more interesting factoids that turned up was that 84 percent of the respondents agreed with the statement that “open-source tools are an important addition to my toolkit,” and 95 percent concurred that “open source is here to stay.” What was reassuring, however, was that this exuberance for open source was tempered by realism, in that only 40 percent agreed that “in an ideal world, all software would be open source,” while 98 percent “prefer using both open source and proprietary software.” As for programming languages, C/C++ was the language used professionally by 61 percent of those surveyed, SQL by 55 percent, Java by 44 percent, Perl by 28 percent, and C# by 23 percent. This study was generally in sync with the TIOBE Programming Community Index (http://www.tiobe.com/tiobe_index/tekst.htm), which attempts to gauge language popularity via Google and Yahoo! searches. According to a recent TPC Index, C held the top spot for languages with a 17.9 percent rating, followed by Java with 14.8 percent, C++ with 13.8 percent, Perl with 9.7 percent, SQL with 3.0 percent, and C# with 1.5 percent. (Note that I’ve abbreviated the TPC list for purposes of comparison.) Finally, in July 2004, I mentioned that “SOCOM II: U.S. Navy SEALs” was my current favorite online PlayStation2 game. Still is. Okay, I’ve tried “Red Dead Revolver,” “Star Wars: Battlefront,” and “MVP Baseball 2004” (and I still can’t hit a curve ball, let alone throw one). That said, “Tom Clancy’s Ghost Recon 2,” with its over-the-shoulder camera view and great graphics, looks promising. I’ll let you know the next time I catch up and look back.
Jonathan Erickson editor-in-chief
[email protected] 8
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
LETTERS
, PO S S O BB
T
D
2
C
T EN
S
2
Ed’s Shibboleth Dear DDJ, I’d like to offer a different perspective on Ed Nisley’s “Shibboleth” column in the April 2004 issue. I’ve been working in the tech industry, first as a UNIX (mostly SunOS/Solaris) system administrator and later as a web application programmer, for 11 years, in the Silicon Valley. It seems to me that there are some shibboleths that distinguish between east coast and west coast techies, because my experience differs somewhat from Ed’s in a few respects: • I’ve never heard anyone say “bine” for /bin, and “libe” for /lib is very rare in my experience. It’s always “bin” and almost always “lib.” • I don’t see any distinction in the way “G” is pronounced in “GNU” versus “GNOME” versus “GNUPLOT.” • SQL is nearly always “see’ kwel” out here. I work at Oracle, but MS Sequel Server and My Sequel are pronounced that way, too. (Side note: San Carlos Airport, a general aviation airport just one freeway exit south of Oracle’s HQ campus, has the airport code SQL. But it had that code (due to the city name) before Oracle existed, and Larry uses San Jose anyway…). • The GIF debate was settled by CompuServe years ago, “jiff.” But the hard “g” still lingers on in some circles. • Would anyone actually pronounce “coax” as “cokes”? (shudder) Yes, that is a true shibboleth. • 68030 is simple: “six’-tee-ate-oh-threeoh.” But I see Ed’s point. And a few things Ed’s column made me think of: • Another shibboleth: / is often pronounced “back’-slash” erroneously by nontechies. It drives me nuts because that’s what \ is called. I’m surprised he didn’t mention that one. 10
• I’d love to see an article about the history of the names for some of the funny punctuation symbols on our keyboards. Octothorpe is my favorite too, by the way. • Out west, “geek” is a badge of honor and “nerd” is an insult. I hear that it’s the opposite on the east coast, at least the MIT area. • Regarding last names, I believe Teddy was “roo’-zee-velt” while FDR was “row’-zee-velt.” • My wife went to Lehigh too (class of ’78). Back then the team was the “Engineers.” We went back a few years ago and found they had changed it to “Mountain Hawks.” I guess that means her BSEE degree is now an “Electrical Mountain Hawking” degree? William R. Ward
[email protected] Database Systems Dear DDJ, The “Database Systems” article by Shasha and Bonnet (DDJ, December 2004) was an excellent description of some of the pitfalls of databases and their use. Although it has probably already been pointed out, Listing One, as is, would mean that Bob and Alice are surely headed to divorce court. However, if we change the second executeUpdate statement to read nbrows = stmt.executeUpdate(tosavings), we may well save their marriage. George B. Durham
[email protected] Dough, Ray, Me Dear DDJ, In a letter to the editor December 2004, Remi Ricard complains that si rather than ti is the seventh solfege note of the scale. Here is an excellent page on the subject — http://www.brainyencyclopedia.com/ encyclopedia/s/so/solfege.html. I think ti has precedence and comes from the original Italian. Galt Barber
[email protected] Do Re Mi II Dear DDJ, Who cares what the French say (Do Re Mi was invented by an Italian, to start with), when the correct American way of saying is: fa-so-la-ti-do. And Rodgers and Hammerstein are not to blame for this: That honor goes to one John Curwen of Heckmondwike, West Yorkshire (1816 –1880), who modified the solfege so that every note would start with a different letter. The lesson to be learned from this: Why listen to your teacher, when you can look it up in Dr. Dobb’s Journal, February 2005
the Wikipedia (http://en.wikipedia.org/wiki/ Solfege/). Jost Riedel
[email protected] The Numbers Game Dear DDJ, Jonathan Erickson’s characterization of the Bureau of Labor Statistics employees who produce employment statistics as “crooked” is outrageous and totally unfounded (“Editorial,” DDJ, November 2004). I work as a consultant with employees of federal statistical agencies including the BLS. I know of no instance of fabrication or illegality in creation of employment statistics that would support his slur. Integrity requires you either substantiate your allegation or retract it and apologize in print. Seth Grimes
[email protected] Jonathan responds: Thanks for your note Seth, and I appreciate your taking the time to write. It sounds like that you’re clearly more an expert in statistical analysis than I, but I’ll do the best I can here. In particular, I’d point to two recent reports out of the BLS: In the July “birth/death” adjustment, the BLS added 182,000 jobs to avoid reporting a net job loss of more than 100,000 for June. Moreover, in its “payroll” surveys, the BLS excludes proprietors, self-employed, farm workers, and domestic workers. Considering the ever increasing number of freelancers and consultants due to fulltime employment loss, this seems disingenuous and designed to spread erroneous information. And as I mentioned in my editorial, the difference between either 23,200 or 51,800 lost jobs in Missouri is significant, particularly since the “adjustment” was made so quickly. It is interesting how the changes are always down in numbers when it comes to reporting job losses in an election year. They’re never adjusted upwards. I’m sure there are similar scenarios. So yes, the pattern does suggest that something is going on — that the numbers are being tweaked for whatever reasons. In no way did I imply — nor do I believe — that all of the hard-working employees of the BLS are crooked. I respect their work and the effort they put forward. That said, someone, somewhere, somehow does seem to be adjusting the numbers for whatever purposes. Thanks again and I hope I have addressed some of your concerns.
DDJ http://www.ddj.com
DR. ECCO’S OMNIHEURIST CORNER
Dig That!
Dennis E. Shasha
S
aturday afternoon in Ecco’s MacDougal Street apartment: Tyler and Liane were comparing strategies for the Voronoi game. Ecco and I discussed connections between monster groups and string theory. “The math is pretty,” Ecco said after a while. “My current avocation is to work out an experimental design to see whether a rapidly pulsed energy source could give evidence of those high-dimensional vibrational strings…” We all have our hobbies. The buzzer rang. Ecco looked out from his second-floor window. “Some official visitors,” he said not altogether approvingly. He collected his papers and put them into a large envelope — his filing system. “Children,” he said to Liane and Tyler, “our spook friends are back.” Liane found a pad. She gave a sheet to Tyler. He started to doodle. Our four visitors all wore suits. Three took out a kind of wand and scanned the room. The other one, evidently in authority, sat and watched. The three nodded and stood behind the seated figure.
Dennis is a professor of computer science at the Courant Institute, New York University. His latest books include Dr. Ecco’s Cyberpuzzles: 36 Puzzles for Hackers and Other Mathematical Detectives (W.W. Norton, 2002) and Database Tuning: Principles, Experiments, and Troubleshooting Techniques (Morgan Kaufman, 2002). He can be contacted at
[email protected]. 12
“My code name is Purple,” the seated man said. “I will tell you only the details you have to know. We need to resolve this problem in the next 20 minutes if possible. “Imagine a road grid with seven rows as in Figure 1. Some bad guys have placed a tunnel from the bottom, beginning at the road marked Start, to the top at End. The tunnel follows the roads somehow but may wind around. It is also a simple path (no dead ends and no loops along the way). You want to probe a minimum number of times and yet be able to find the exact route of the tunnel.
End
1
2
3
4
5
6
7
Start
Figure 1. Dr. Dobb’s Journal, February 2005
“A probe device takes an hour to set up. If you set up a probe on a street, you can tell whether it follows the street. If you set it up on an intersection, you can tell whether the tunnel passes through the intersection, and if so, which adjacent streets it goes to. “If the tunnel is at most eight blocks long and starts at Start and ends at End, then what is the minimum number of probe devices you would need to guarantee determining the precise route of the tunnel in one hour? “Also, if you had only one probe, what is the minimum time it would take to guarantee determining the outcome?” Ecco, Liane, and Tyler conferred for a while. Finally, they sketched an answer. Mr. Purple studied it. “I can’t thank you enough,” he said. “I think this is exactly what I need, though I’m not completely sure. The tunnel might be 10 or even 12 blocks long. We don’t think so, but we can’t rule it out. “Can you answer the above two questions if the tunnel could be 10 blocks or 12 blocks long?” I left before Ecco and his talented family could find an answer. For the solution to last month’s puzzle, see page 68.
DDJ http://www.ddj.com
SECTION
A
MAIN NEWS
Dr. Dobb’s
News & Views
OASIS Ratifies UBL After six years of development, the OASIS standards group has ratified the Universal Business Language (UBL) 1.0 specification (http://docs.oasis-open.org/ubl/cd-UBL-1.0/). UBL “is designed to provide a universally understood and recognized commercial syntax for legally binding business documents and to operate within a standard business framework such as ISO 15000 (ebXML).” It provides a library of XML schemas for common, reusable data components (“Address,” “Item,” and “Payment,” for example); a set of XML schemas for business documents (“Order,” “Dispatch Advice,” “Invoice,” and the like); and support for customization.
Google Scholar Launched Google has launched the Google Scholar service, a new search facility that indexes academic research, including technical reports, preprints, abstracts, and peerreviewed papers (http://scholar.google .com/). Participating groups include the ACM, IEEE, and the Online Computer Library Center. Google Scholar orders search results by relevance to your query. This ranking takes into account the full text of each article, along with the author, publication, and how often it has been cited in scholarly literature. The search also analyzes and extracts citations and presents them as separate results.
Remembering Kenneth Iverson APL inventor Kenneth E. Iverson died in Toronto, Ontario at the age of 84. Among his distinctions were the U.S. National Medal of Technology and the Turing Award; he was also an IBM Fellow. It was as an assistant professor at Harvard in the 1950s that Iverson first began working on APL. The Iverson Notation, as it was originally known, was designed to express mathematical algorithms clearly and concisely. At IBM in the 1960s, he and Adin D. Falkoff developed APL, short for “a programming language.” In the early 1990s, Iverson and Roger Hui created the J language, which is based on APL as well as the FP and FL functional programming languages. Unlike APL, J requires no special character set; it uses ASCII characters for all primitives. Iverson worked with Jsoftware Inc. to document and promote this language. Along with the National Medal of Technology and the Turing Award, Iverson was 14
granted the Harry M. Goode Memorial Award, and was named a Computer Pioneer Charter Recipient by IEEE CS.
IBM Sweeps Supercomputer Rankings NEC’s Earth Simulator, which topped the biannual Top500 rankings (http://www .top500.org/) of the world’s most powerful supercomputers for two-and-a-half years, has finally lost its crown. The latest edition of the Top500 list puts IBM’s Blue Gene/L in first place. Blue Gene/L— now located in Rochester, New York but destined for the Lawrence Livermore National Laboratory in California — clocked at 70.72 teraflops on the Linpack benchmark. IBM says it hopes to deliver a theoretical maximum of 360 teraflops next May, when the system is complete. It will comprise 64 racks carrying 65,000 dual-processor chips, but will consume only a fraction of the power that Earth Simulator requires. The second-place supercomputer is also new: SGI’s Columbia system at the NASA Ames Research Center in Mountain View, California, which performed at 51.87 teraflops. Columbia consists of 20 machines connected by InfiniBand. Each one has 512 processors and runs its own instance of Linux, although SGI is working to connect the machines into groups of four sharing an operating system. Earth Simulator, a mammoth 32,500square-foot system located in Yokohama, Japan, now holds third place on the Top500 list. That spot was formerly occupied by Virginia Tech’s Mac-based System X, last year’s surprise entry into the list. System X has since been upgraded to Apple Computer’s Xserve G5 server, which improved its benchmark score by 19 percent; however, even its new 12.25 teraflop performance isn’t enough to compete against the mightiest of the new supercomputers. System X has slipped to seventh place in the rankings. It’s likely, however, that System X remains the winning system when performance is considered against price. It costs $5.2 million to deploy and $600,000 to upgrade to the Xserve G5— as compared to nearly $100 million for Blue Gene/L.
“Trusted Mobile Platform” Specification Drafted IBM, Intel, and NTT DoCoMo are promoting their “Trusted Mobile Platform,” defining security-oriented hardware, softDr. Dobb’s Journal, February 2005
DR. DOBB’S JOURNAL February 1, 2005
ware, and protocol requirements for mobile devices (http://www.trusted-mobile .org/). According to the companies, “The specification will help make advanced mobile devices/applications more secure and help protect against viruses and other threats…In addition, it defines a set of protocols that allow the security state of a device to be shared with other devices in the network, enabling device-level trust to be extended into the larger network.”
NASA Gives Researchers Millions NASA has awarded a $58 million grant to three professors at the University of Southern California’s Information Sciences Institute for the development and analysis of systems that are construction based, computer based, and logistical, respectively. Senior research scientist Wei Min Shen received $28 million for his “Modular, Multifunctional Reconfigurable Superbot” study, to build and analyze shape-shifting robot modules that are self reconfiguring and capable of autonomous connection to each other. John Damoulakis received $18.1 million for his “Fault-Aware, Modular, Reconfigurable Space Processor” project, to produce reliable space-computing systems capable of fault comprehension and self repair. Robert Neches received $15 million for “Coordinated Multisource Maintenance-on-Demand,” which will examine ways to help the space agency use improved cost-benefit analysis as it relates to the expected requirements for equipment against the probability of malfunction.
AI Isn’t a Crime The Classification System for Serial Criminal Patterns (CSSCP), a neural networkbased system developed by computer scientists Tom Muscarello and Kamal Dahbur at DePaul University, examines crime case records, assigns numerical values to aspects of each crime (type of offense, perpetrator’s gender, age, height, and type of weapon), then builds a descriptive profile of the crime. Unlike neural networks that require training, the CSSCP is based on Kohonen networks, which do not require training or human intervention. When CSSCP detects similarities between crimes, it compares the time and place to see if it is possible for the criminal to have committed both crimes. http://www.ddj.com
Java Web Services & Application Architectures Building better architectures with web services ERIC J. BRUNO
A
web site can see, potentially, millions of Internet users in a day. This makes performance and scalability a big issue in web-site development. Web-site performance is measured in response time, or how long it takes for a user’s browser to receive HTML after a request. Scalability is the capacity to maintain good performance even when large numbers of users simultaneously— and continuously— make requests. To achieve scalability, the hardware and software that make up a web site are typically segmented into tiers like those in Figure 1. The web tier is where user requests are first handled and queued. The application tier is where the business logic — and code — behind the web site resides. The data tier, sometimes called the “enterprise information system” (EIS) tier, contains databases or legacy systems as data sources. Depending on the web site, some tiers can be combined or segmented even more. Sun, for instance, defines further subtiers with its model-viewcontroller architecture (MVC). The Service Tier Web services are often used to expose application functionality to paying customers, or to integrate legacy systems with newer applications. However, web services can become part of an application’s design to help achieve component isolation, gain greater scalability, and ease development. In this article, I propose adding a new tier to the traditional multitiered architecture — the service tier. An architecture without a service tier has a huge flaw— the lack of data hiding or abstraction. Without a service tier, appliEric is a consultant in New York, and has worked extensively in Java and C++ developing real-time trading and financial applications. He can be contacted at
[email protected]. 16
cation code must have intimate knowledge of the data sources and associated details, such as database schemas or low-level software APIs. Adding a service tier to your architecture — one that contains web services that abstract the data sources from the core business logic code — solves this problem.
“An architecture without a service tier has a huge flaw” A service tier should be designed to achieve the following goals: • Abstraction. Hide the details of the data sources (schemas, APIs, and so on). • Simplicity. Generally simplify the process of accessing data. The purpose of adding a tier is to add simplicity to the design of the software as a whole. Introducing complexity here defeats the purpose of abstraction. • Loose coupling. As with all software layers, the service tier and the other tiers should be loosely coupled, with an interface that truly hides all implementation details. • Performance. Although nothing is free, the addition of the service tier should not introduce a significant amount of performance overhead.
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
(continued from page 16) In general, a service tier that achieves these goals can provide other benefits to application developers. First, there is greater potential for software reuse as the service tier is accessed through a generic interface. Second, a more efficient development process is achieved as the application tier and the service tier can be developed independently. For example, developers strong in JSP or ASP can concentrate on building applications, while database developers (or developers familiar with an external API) can build portions of the service tier. This helps to avoid maintenance problems that come from mixing database queries in application-tier code. Third, system interdependencies (and associated complexity) are greatly reduced. A well-designed service tier lets backend data systems change and evolve independently and transparently from the software components that use them. Finally, true implementation independence is achieved,
DMZ App Server
Legacy System
Web Server
App Server
Database
Web Tier
Application Tier
Data or Enterprise Information System Tier
Figure 1: Typical web application architecture.
HTTP HTML .NET
Service Tier
XML .NET
Data Tier
API, JDBC, ODBC, ADO… Raw Data
Database Legacy System
HTML
Quotes Web Service (JWSDP/SAAJ)
Fundamentals Web Service (JWSDP/SAAJ) XML HTTP
yahoo.com
SEC.gov Raw Data
Other
Figure 2: Web architecture with a service tier. 18
Financial Portal (Java Servlet on Tomcat 5.0)
SOAP/HTTP
HTTP
PHP
Web/Application Tier
Example 1: Financial portal SOAP request message.
Service Tier
Web/Application Tier
Client Browser or Application
Java (WSDP)
• SOAP with attachments. • Web Services Description Language (WSDL). MSFT Microsoft Corp 0000789019
External Services
Java (Servlet, JSP, EJB)
The Java Web Service Developer Pack Java has long been the leading choice for web application development. Sun positioned Java as a powerful platform with which to build web services when it introduced the Java Web Service Developer Pack (WSDP) about two years ago. Now at Version 1.4, the Java WSDP provides all of the APIs, tools, and infrastructure you need to build web services that support the important standards such as:
Data Tier
Client Browser
Web Server
as the service tier components can be written in a language or environment different than that of the application. The Simple Object Access Protocol (SOAP) is perhaps the best choice when implementing components of a service tier. SOAP not only achieves the architectural goals and benefits, but yields other practical benefits as well. For example, a SOAP web service can be deployed across multiple servers, and load balanced using existing HTTP load-balancing technology. The SOAP protocol itself defines a strict contract for communication between layers. In addition, SOAP— being based on XML— provides the right level of abstraction, allowing for implementation independence. In Figure 2, a web application architecture enhanced with a service tier, the web tier and application tier have been combined for simplicity. The elegance of the design comes from the degree of separation and abstraction achieved across the tiers. The application code becomes simpler as it no longer deals with the details of individual data sources. SOAP adds consistency to the way data is requested from and returned to the application code.
Figure 3: Sample financial portal architecture.
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
• Universal Description, Discovery, and Integration registry (UDDI). • Web Services Interoperability (WS-I). • Web Services Security (WSS and XWS-Security). • Security Assertion Markup Language (SAML). For more information on the Java WSDP (along with software downloads), see http://java.sun.com/webservices/index.jsp. There are download bundles available for Sun’s J2EE 1.4-compatible application server, Sun’s Java web server, and Apache’s Tomcat web server/Servlet container. I suggest starting with the Tomcat container, which is an open-source project from Apache-Jakarta (http://jakarta.apache.org/tomcat/). It’s free, works well, and is available on many platforms. The Java WSDP integrates XML parsing using JAXP; WSDL and low-level SOAP messaging using JAX-RPC; XML to Java binding using JAXB; UDDI using JAXR; and high-level SOAP messaging with attachments using SAAJ. For a detailed overview, see the Java Web Service’s Tutorial (http://java.sun.com/webservices/docs/1.4/tutorial/doc/index.html). A Sample Application: The Financial Portal The sample web application (available electronically; see “Resource Center,” page 5) I present here is a basic financial portal. The portal displays stock quotes, company fundamental data, and company financial data (balance sheet, income statement, and cash flow statement) when a company’s ticker symbol is entered. I’ve chosen to use the Java WSDP bundle that comes with the Tomcat 5.0 web container. The code uses the SOAP with Attachments API for Java (SAAJ) to build two SOAP servers and one client, as well as JAXP for parsing the XML payloads of the SOAP messages. Figure 3 illustrates the basic architecture for the application. The application contains two SOAP web services — a Quotes Service and a Fundamentals Service — which are written using SAAJ. To use SAAJ, your Java code must extend the class,
Example 2: Sample application directory structure. http://www.ddj.com
Dr. Dobb’s Journal, February 2005
19
com.sun.xml.messaging.soap.server.SAAJServlet, contained within the file saaj-coms.jar. Doing this, your class basically becomes a Java Servlet capable of receiving SOAP requests by overriding the method, onMessage. This method provides the SOAP request as a parameter, and lets you return a SOAP response. The Quotes and Fundamentals services both wait for a SOAP request that looks like the XML in Example 1. The contents of the SOAP body are in bold, and contain the company’s stock symbol, name, and the central index key (CIK) as assigned by the Securities Exchange Commission (SEC). When the Quotes Service receives a SOAP request, it is parsed and checked to ensure it contains all of the data needed. Listing One shows the code that receives and parses a simple SOAP request message. After the request is parsed, the Quotes Service makes an HTTP request to Yahoo using the company’s stock ticker. The result from Yahoo is comma-delimited quote data, which is parsed and formatted into an XML SOAP body to be sent back to the SOAP client. The code for generating the SOAP response is shown in Listing Two. It is important to note that using Yahoo for quote data is for illustrative purposes only, and should not be distributed in any way. The Fundamentals Service works in a similar fashion. SOAP requests are received and parsed to gather the needed data, which includes the company’s ticker symbol and CIK. Next, the service makes an FTP request to the SEC’s FTP servers to obtain the latest financial filings for the given company. The FTP response contains basic company information — such as address and business classification — as well as the financial filings in HTML form. Next, the service creates a SOAP response that con-
Figure 4: Company ticker request HTML form.
Figure 5: Financial portal company data screen. 20
tains the company’s fundamental data in the SOAP body, with the HTML filings added as a SOAP attachment. The code for this can be seen in Listing Three. SOAP requests are made to both web services (Listing Four) from a Java Servlet that contains the application code called the “Financial Servlet.” This Servlet generates an HTML form for users to enter company ticker symbols, as in Figure 4. When a symbol is submitted, a SOAP request is made to the Quotes service as well as the Fundamentals service. The SOAP responses are received and parsed, and an HTML web page (Figure 5) is generated containing all of the quote, fundamental, and financial data from the SOAP messages and the attachment. This application illustrates how segmenting retrieval into a service tier can be a major design improvement. For example, the web services in this application can be distributed across multiple servers, where the work of gathering quote and fundamental data can occur in parallel instead of serially. Future modifications, such as a change in the source for quote data, can be made without changing any deployed applications. Only the Quotes service would need to change. The SOAP interface completely hides the implementation details. Building and Deploying Web Services The sample financial portal application assumes that the application will be deployed and run within the Tomcat 5.0 web container, although it should run within any J2EE web container without major changes. The Java WSDP comes with the Apache Ant build tool (http://ant.apache.org/). To use Ant, you must write XML scripts that build and deploy your Java software. An Ant script for the financial portal application is included with the sample code. The script contains paths to the JAR files needed during compilation, such as those for SAAJ, JAXP, and J2EE. You may need to modify these paths for your system. It is assumed that the application code exists in the path c:\dev\FinancialWebApp\FinancialPortal, with the directory structure in Example 2, which is a standard J2EE WAR structure. The source code files exist within the directory, WEB-INF/src, where the complete path matches each component’s package name. The same subdirectory structure is used for the compiled class files within WEB-INF/classes. The Ant script contains the target, “compile,” which looks for the source files in the proper source paths, and places the compiled class files in the proper output paths. The WEB-INF directory contains the deployment descriptor, web.xml, for the entire web application, containing the Quotes web service, the Fundamental web service, and the financial portal Servlet (Listing Five). The Ant script contains the target, “war,” which uses the JAR utility to package up the entire directory structure into a file named “FinancialPortal.war.” This archive contains the class files, the application deployment descriptor, and other miscellaneous files that make up the financial portal application. The next step is to inform Tomcat of the existence of the sample web application. First, locate the path, tomcat-jwsdp1.4/conf/Catalina/localhost, in the directory where you installed the Java WSDP/Tomcat bundle. In this location, add an application context file named “financialportal.xml,” which tells Tomcat where to find the application’s WAR file. The contents of this file are simple and can be seen in Listing Six. Once these steps are complete, the web application is ready to run. When Tomcat is started, it will automatically read the file, financialportal.xml, and attempt to deploy the web application (the financial portal) referenced within. When loaded, the application reads an index file that tells it where to locate the
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
financial filings for all U.S. public companies. This step requires around 30 seconds, after which the application will be ready for requests. The index file that the application uses is located on the SEC’s FTP servers at ftp://ftp.sec.gov/edgar/full-index/master.idx/. Companies file with the SEC periodically, so you will need to download this file occasionally to keep up to date. The Fundamentals Service can be enhanced to download this file automatically based on a schedule. Finally, open a browser and type http://localhost:8080/ financialportal to display the company ticker request page. If you chose a port other than 8080 when you installed Tomcat, you must modify this URL accordingly. Enter a valid U.S. company ticker, such as EBAY, and around one minute later, you will see that company’s quote data, fundamentals, and financial filings. Conclusion Building a SOAP web service is ideal when you need to expose data and/or functionality to customers. Even if customers are never directly exposed to it, building a service tier (an entire layer of web services within your application) adds value to an application’s architecture by limiting the coupling between components and by creating the potential for future reuse. Conceivably, the service tier can be extended beyond the boundaries of one application. The service tier can become a company-wide deployment of web services available to all applications in your organization. Thus, the need for a metadata service — a service that describes all available web services — may arise. The Java Web Service Developer Pack contains all of the tools and support necessary to build web services in Java. Combined with the open-source web container, Tomcat, you can build web services to be deployed on a Windows or UNIX system with little or no modification. If you choose to deploy to Tomcat on Linux, for example, your web service infrastructure costs will be nothing more than the hardware to run on — with no OS or application server licenses to purchase. That makes Java a powerful choice when building web services. DDJ Listing One public SOAPMessage onMessage(SOAPMessage message) { try { // The envelope contains the message header, body, and attachments SOAPEnvelope env = message.getSOAPPart().getEnvelope(); SOAPBody bdy = env.getBody(); // Get the ticker symbol Iterator iter = bdy.getChildElements(); SOAPElement node = getNode("GetData", iter); if ( node != null ) { iter = node.getChildElements(); symbol = getNodeValue("symbol", iter); } // Get the quote data from Yahoo String data = getQuoteData( symbol ); // ... } catch(Exception e) { return null; } }
Listing Two // Create the SOAP reply message SOAPMessage replyMsg = msgFactory.createMessage(); // part of SAAJ SOAPEnvelope env = replyMsg.getSOAPPart().getEnvelope(); SOAPBody bdy = env.getBody(); // Add the quote data to the message bdy.addChildElement(env.createName("symbol")).addTextNode( symbol ); bdy.addChildElement(env.createName("last")).addTextNode( last );
(continued on page 22) http://www.ddj.com
Dr. Dobb’s Journal, February 2005
21
SOAPMessage quoteReply = con.call( soapMsg, urlEndpoint );
(continued from page 21) bdy.addChildElement(env.createName("date")).addTextNode( date ); bdy.addChildElement(env.createName("time")).addTextNode( time ); bdy.addChildElement(env.createName("change")).addTextNode( change ); bdy.addChildElement(env.createName("open")).addTextNode( open ); bdy.addChildElement(env.createName("low")).addTextNode( low ); bdy.addChildElement(env.createName("high")).addTextNode( high ); bdy.addChildElement(env.createName("volume")).addTextNode( volume ); return replyMsg;
Listing Three // Create the SOAP reply message SOAPMessage replyMsg = msgFactory.createMessage(); SOAPEnvelope env = replyMsg.getSOAPPart().getEnvelope(); SOAPBody bdy = env.getBody(); // Add the fundamental data to the SOAP body ... // Add the financial HTML as a SOAP Attachment AttachmentPart ap = replyMsg.createAttachmentPart(financialHTML,"text/html"); replyMsg.addAttachmentPart(ap); return replyMsg;
Listing Four // Create a SOAP request message MessageFactory msgFactory = MessageFactory.newInstance(); SOAPMessage soapMsg = msgFactory.createMessage(); SOAPEnvelope envelope = soapMsg.getSOAPPart().getEnvelope(); SOAPBody bdy = env.getBody(); // Add the request data to the SOAP body SOAPBodyElement bdyElem = bdy.addBodyElement( envelope.createName("GetData")); bdyElem.addChildElement(envelope.createName("symbol")).addTextNode( symbol ); bdyElem.addChildElement(envelope.createName("name")).addTextNode(companyName); bdyElem.addChildElement(envelope.createName("CIK")).addTextNode( companyCIK ); SOAPConnectionFactory scf = SOAPConnectionFactory.newInstance(); con = scf.createConnection(); // Send the request to the Quote Service URL urlEndpoint = new URL( "http://localhost:8080/quotes" );
// Send the request to the Fundamentals Service urlEndpoint = new URL( "http://localhost:8080/fundamentals" ); SOAPMessage fundReply = con.call( soapMsg, urlEndpoint );
Listing Five Financial Portal Displays company quote and financial data fundservlet com.fundamentals.FundServlet 1 quoteservlet com.quotes.QuoteServlet 2 FinancialServlet com.financialportal.FinancialServlet 3 fundservlet /fundamentals quoteservlet /quotes FinancialServlet /financialportal
Listing Six
DDJ 22
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
SOAs & ESBs Ensuring that your ESB delivers an SOA JAMES PASLEY
T
he design principals of ServiceOriented Architectures (SOAs) are widely considered to be best practice when solving integration problems or as part of business process modeling. An Enterprise Service Bus (ESB) gives you the tools and infrastructure necessary to build an SOA. But with all the components and tools provided, you can lose sight of the SOA while working on the ESB. Consequently, care must be taken to ensure that the end result is a new architecture, and not just a better way to perform point-topoint integration. In this article, I present an overview of SOA, describe the ESB’s role, and present tasks you can expect to tackle when deploying an SOA. Service-Oriented Architecture An SOA is a way of approaching the task of creating software. You can think of it as either a collection of architectural concepts or a programming model. These concepts should be understood and followed carefully to make effective use of an ESB. Within an SOA, all functionality is provided by services. This provides a level of flexibility in deploying and reusing services not previously attained — services that: • Are remotely accessible over asynchronous or synchronous transports. James is chief architect at Cape Clear Software. He can be contacted at james
[email protected]. 24
• Have a well-defined interface described in an implementation- independent manner. • Are self contained, and can be used independently of any other services. • Perform a specific task. • Are loosely coupled. In other words, loosely coupled APIs should be: coarse-grained, representing an overall business function or service; based on the exchange of documents rather than the datatypes used in RPC-style APIs (this typically means that operations will have a single input or output parameter that is defined using XML Schema); accept a range of different documents that contain the requisite information; not include explicit or implicit state assumptions (that is, be connectionless); and are dynamically discoverable at runtime rather than statically bound. However, an SOA is more than just a collection of services — it adds a number of principals, of which interoperability is the most important and the core to the idea of SOA. Interoperability is achieved through the use of Standards. The webservices Standards stand out above all others in satisfying this requirement. Another core feature of an SOA is the reuse of existing assets. Deployment of an SOA within an organization is an exercise in creating services from the existing IT systems and infrastructure. Once an SOA is in place, the existing IT systems within an organization can be viewed as services that provide business functions. These are easily integrated because they provide well-defined interfaces and can be accessed using standard protocols and transports. This provides the basis on which to orchestrate services into new services that reflect business processes. Dr. Dobb’s Journal, February 2005
The Role of the ESB The deployment of an SOA requires the conversion of existing systems into services. The tasks involved in achieving this may be repeated for each system and a
“Care must be taken to ensure that the end result is a new architecture, and not just a better way to perform point-to-point integration” common set of components may be needed to provide additional functionality. An ESB is a collection of servers and components that provide these functions and a set of tools to assist in the conversion of existing systems to services. The ESB: • Solves integration problems at the transport layer by supporting Standardsbased transports such as HTTP. It also supports a variety of commonly used transports such as MQSeries, or even shared folders for batch-processing applications, so that existing systems can be exposed as services supporting new transports. • Provides for the routing of messages across various transports. • Provides tools to help expose the existing systems as web services; for example, a wizard to convert CORBA IDL http://www.ddj.com
(continued from page 24) to WSDL. In addition, it provides the runtime piece to convert SOAP requests to the web service into IIOP requests to the CORBA Object. You can expect to find such tools for multiple classes of systems such as Java, J2EE, databases, and mainframes. • Provides adapters to a variety of packaged applications such as SAP or PeopleSoft. However, with the proliferation of web-services Standards, many of these applications are providing webservice interfaces directly. Therefore, the need for the ESB to provide such adapters reduces over time. • May also host services. While the emphasis remains on the reuse of existing systems, an ESB can be used as a powerful application-development environment. Features such as database integration and adapters to applications (SAP, for example) ensure rapid application development. • Addresses quality-of-service issues such as security and data validation, which arise as a result of exposing services to a variety of clients. • Provides for the transformation of both XML and other document formats. The Lifecycle of an Integration Project The lifecycle of a software project that uses an ESB can be thought of as constructing an SOA in layers, starting at the existing infrastructure. As such, there is a
natural progression to the development of services within an SOA; see Figure 1. The steps involved can be grouped together into three categories: • Exposing the existing systems as core services. • Building business services from these core services. • Using the business services to achieve business process modeling. An SOA emphasizes the reuse of existing assets, so exposing the existing systems is a natural starting point. This must be balanced with the emphasis on creating services that reflect business needs and not technology. This means that the top-down approach should also be considered. That said, in this article, I follow a bottom-up approach because it facilitates a better explanation of the tasks involved. Exposing Core Services Exposing existing IT systems as services is typically the first task you perform when using an ESB. ESBs provide a number of tools to help with this. For programming APIs (such as Java, EJB, or CORBA), wizards automatically generate WSDL from the APIs. For messaging systems, transports are provided to expose particular queues as web services. For batch-processing systems, the documents processed are often text-based and
• Can be invoked remotely, over a standard transport by a variety of web-service clients. • Has an XML-based payload format. • Uses a standardized protocol envelope (SOAP Envelope). • Has a formal description of the interface (WSDL and XML Schema). • The payloads in messages are validated against the WSDL/XML Schema to ensure that invalid data is not passed on to the application. Building Business Services The first stage in the lifecycle of a project solves many integration issues by creating web services from the existing systems. This fulfills a number of the characteristics of a service as defined by the SOA. The web services can be invoked remotely, have well- defined interfaces, and (depending on the systems they were created from) are self contained. However, other aspects may not be there. Web services that conform to the SOA definition can be thought of as “business services,” based on the requirement that they perform a specific task or business function. This is the point at which you need to take a step back and consider what you’ve done. To turn these web services into services as defined by SOA, you may have to reconsider the API to each service and evaluate whether they can be described as loosely coupled. The web services created from mainframe or batch-processing systems most likely already meet these remaining requirements because they process documents in an asynchronous manner. The
Figure 1: Phases of an ESB project. 26
typically consist of fixed-width fields. The payload of the exposed web service should be XML to achieve maximum interoperability. This requires transformation of the payload document. ESBs can provide tools (such as Cape Clear’s Data Interchange) to facilitate the creation of transforms between varieties of document formats. Because no formal description of the data formats may exist, this task cannot reach the same level of automation as is achieved for programming APIs. Depending on the nature of the existing systems, the creation of these transforms can be a significant part of the development effort. As a result, the tools to support this should be reviewed carefully when selecting an ESB, to ensure that this task does not become a programming one. With the tools provided by the ESB, you have solved a number of integration issues and have added web-service front-ends to your existing systems. The web services have the following features that, depending on the nature of the existing systems, may not have been present before:
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
(continued from page 26) web services created from the other systems are more likely to be fine-grained, RPC-based, synchronous systems. If the web services automatically generated from the existing systems cannot be described as loosely coupled, then new descriptions (WSDL) should be created. To illustrate loosely coupled services, consider how a customer might purchase car insurance. In the first scenario, the customer could use the telephone to carry out the transaction. This approach is synchronous — it requires an operator to be available to handle the customer’s request. The telephone conversation forms a connection in the context of which the interactions take place. The exchanges are finegrained — the operator asks a series of individual questions to obtain the necessary information. Now consider a second scenario, in which the customer fills out a form and mails it to the insurance company. This approach is course grained and handles documents. It is asynchronous — it does not require an operator to be present at the exact time the customer fills out the form. This second scenario is consistent with the principals of an SOA, while the first one is not. Look at this example again, but this time in code. Consider the Java API in Listing One that is to be exposed as a web service. It has a method to initiate the interaction — one that provides some additional data. Another method is used to return the quoted price, and finally, one that processes the application and bills the customer. Running a JAX-RPC-compliant, WSDLgeneration tool over this kind of a Java interface results in the WSDL in Listing
Two. This solves a number of integration issues; for example, a .NET client that invokes on this web service can now be written. However, to create a loosely coupled service from this, a new WSDL interface must be designed. For this, you need a WSDL editor. The granularity of the API should be increased to provide fewer operations, but ones that accept documents as parameters. At the same time, the API should be made connectionless by eliminating the use of the reference parameter. This leaves you with two operations —calculateQuote and processApplication. Both of these operations accept the application document as a parameter. This may seem inefficient, but it is done to ensure that the API is connectionless. Applying these principles to the WSDL results in Listing Three. In this new WSDL: • The data types from the original WSDL have been combined under a single element (InsuranceApplication) to form a document. • There are just two operations: calculateQuote and processApplication. • Both of the operations take the entire InsuranceApplication as input. • The responses are returned to the caller asynchronously, via a callback API (InsuranceServiceCallBack). In light of the aforementioned example, the following guidelines should be applied to WSDL files as part of evaluating whether they can be described as loosely coupled. New WSDL files should be created where necessary. • portTypes should have a small number of operations that reflect the business tasks.
Implementing the New Service An ESB provides tools to generate Java skeleton implementations for web services. This new WSDL can be used to generate such a Java skeleton, providing the basis for the implementation of the new service. This service can act as a client of the autogenerated service. Of course, the equivalent changes could be made to the original Java API. However, there are benefits to working at the XML level. An important part of an SOA is the definition of data types within the XML Schema that can be reused across different services. The use of common data types defined using XML Schema should be considered as early as possible when working with an SOA. This greatly reduces the amount of data transformation that must be performed later when reusing these services. Implementing Service Policies By now, you have created business services that can be easily integrated and reused. However, exposure to business partners or deployment within an enterprise requires additional functionality. This is functionality that can be provided by the ESB itself and is typically common to all services. These features can also be described by the WSDL or by policy statements included within the WSDL; see Figure 2. This step in the project lifecycle can be considered the point at which the WSDL changes from being a description of the API to becoming a contract between the service and its clients. The additional features provided by the ESB, such as security, management capabilities, auditing, logging, or even custom features can be added. These can be added across the different services implemented using an interceptor or message-handler framework. This is an area where you can expect an ESB to provide components that can be easily combined with the services you have built. Building Business Processes With services that reflect business functions in place on the ESB, new services that model complete business processes can be created. Business Process Execution Language (BPEL) is a language
Figure 2: Implementing the service policy. 28
• Each operation should have a single parameter defined as a document. • The API should not include the concept of a session, or operations that depend on data passed to previous operations. • The API should support asynchronous interactions. A separate interface (portType) should be provided for responses that the server needs to send back to the client.
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
designed for just this purpose. A BPEL editor should be provided to enable the existing services to be combined into processes. The ease with which new business processes can be developed depends in part on how successful the design of the business services has been. BPEL is a language designed to model business processes — it is not a place to
solve integration issues. These should be solved in the lower layers. Conclusion An ESB provides effective tools and services that can be used to construct an SOA. However, careful thought and planning is required in order to ensure that the resulting system is truly an SOA. This
Listing One public class CustomerDetails { public String name; public String street; public String city; public String state; } public class CarDetails { public String make; public String model; } public class BillingInfo { public String creditCardNumber; public String exprityDate; } public interface InsurancePortType { public String createApplication( CustomerDetails customer ); public void setCarDetails( String reference, CarDetails details ); public Decimal calculateQuote( String reference ); public void processApplication( String reference, BillingInfo info ); }
Listing Two
http://www.ddj.com
requires a good understanding of the principals of SOA to evaluate the output of the tools provided by the ESB. The effort put into the design of individual services exposed on the ESB results in benefits when the task of business-process modeling is performed. DDJ
Listing Three
Dr. Dobb’s Journal, February 2005
DDJ 29
Ruby/Amazon & Amazon Web Services Automating e-commerce with a convenient interface to Amazon’s AWS API IAN MACDONALD
I
t used to be that if you wanted to access the vast wealth of data on the Web, you fired up a web browser and hopped from page to page, selectively reading as you went. That was fine for quickly looking something up or whiling away the time, but the Web has become an increasingly valuable research and business tool. Research is often defined in practical terms as the need to process and categorize large quantities of data, while business transactions are usually more efficiently handled by computers than humans, so the desire to automate these processes is natural and inevitable. That’s where a browser ceases to be the right tool for the job. Various technologies enable programmatic access to the Web without the need for tedious and error-prone screen scraping — REST, XML-RPC, and SOAP are the best known amongst these. These technologies have encouraged information providers to make their data available not just in old-fashioned HTML for interactive consumption, but also as structured data that can be programmatically queried, parsed, and manipulated. The net result is that large web sites — especially those that conduct commerce on the Web — have developed APIs through which their users and affiliates can search Ian is a a senior system administrator for Google. He can be contacted at ian@ caliban.org. 30
for information and conduct business. eBay, Google, salesforce.com, and Amazon, for example, all offer web APIs via which users may mine databases and transact business. These are usually, but not always, free to use, as they encourage the development of third-party products, and in turn, lead to the growth of a wider community around the company’s business. In this article, I examine Amazon Web Services (AWS), one of the more comprehensive APIs offered by web commerce companies. There are a number of different ways of using AWS. For example, Amazon offers both a REST- and a SOAPbased API. The choice of which to use is up to developers and is largely a question of taste, which is why Amazon chose to offer both. At this point, there’s nothing to stop you from reading the API documentation and immediately starting to code an application in your favorite programming language, but you’ll soon find yourself spending time on the mechanics of the API, rather than just dealing with the data you want to extract. Happily, a number of people have written wrapper libraries around AWS, removing the need for you to parse XML, deal with common error conditions, and program around limitations in the underlying API. Here, I look at the Ruby/Amazon toolkit, which is a high-level abstraction of AWS I wrote for the Ruby programming language (http://www.caliban .org/ruby/ruby-amazon.shtml). Ruby has been rapidly gaining ground as a highly viable alternative to Perl, Python, and Java for general scripting, system administration, and applications programming (http://ruby-lang.org/). Coming equipped with strong XML, HTTP(S), XML-RPC, and SOAP libraries, it fortunately also lends itself very well to web-services programming. Ruby/Amazon has some convenient, time-saving features, such as the ability to cache search results for later recall and the option to find all results for a given Dr. Dobb’s Journal, February 2005
search, rather than obtain just the single page of 10 results that the underlying API returns. Without a higher level wrapper library acting as an intermediary, achieving the same result would entail retrieving each consecutive page yourself, until no more results were returned. AWS has undergone a number of revisions since its inception in 2002 and is
“One of the more advanced features offered by Ruby/Amazon is the option of customer geolocation” currently at Version 3.1. At the time of writing, AWS 4.0 is in beta testing and is a complete rewrite of the API. I focus here on 3.1, as that is the version in the most widespread use. Before starting, you should get a developer token by registering at http:// www.amazon.com/gp/aws/registration/ registration-form.html/. While there, it is a good idea to apply for an Associates account (https://associates.amazon.com/ exec/panama/associates/apply/). This lets you earn referral fees for any customers you bring to Amazon via applications you create that take advantage of AWS. You’re ready to roll. If you need to use an HTTP proxy, define it now in the UNIX environment variable $http_proxy; see Listing One. As you can see, I am choosing to display information about a product’s name, author, price, and availability. Each property of the product is available as an instance variable of the corresponding object. In Listing One, the object is passed into the http://www.ddj.com
(continued from page 30) search block as an iterator. To display a list of available properties for any given product, you can call the properties method of the corresponding product object. Not every product has every type of property. For example, a book will have authors, but no directors, while a DVD is the opposite. Even within a single product category, some products will have properties that others are missing. Some properties, however, such as the price, are logically common to all products; everything has a price at Amazon. At this writing, Listing One yields the results in Figure 1. Rather than use the iterator block form of the search, this would also have worked as well: response = r.keyword_search ('ruby programming', 'books')
Using this approach, response now contains the literal REST response returned by AWS; you can print or puts it and see. This is sometimes useful for the purpose of debugging. You can still iterate over
the products by calling each iterator on the products array: response.products.each do |p| # Process product data end
Amazon Standard Item Numbers (ASINs, see Figure 1) are at the heart of the Amazon product catalog, as they provide every product in the catalog with a unique identifier. For a book, the ASIN is the same as the ISBN number, but for other items, it’s an arbitrary alphanumeric string concocted by Amazon. Many of the AWS API calls deal directly with ASINs to uniquely identify the product in question, so you’ll likely find yourself passing them around a lot. If you didn’t elect to use caching at the time of request object instantiation, you can elect to do so now by establishing a cache object for future searches. The cache directory given is created, if it does not already exist. r.cache = Cache.new( '/tmp/amazon' )
Programming Ruby: The Pragmatic Programmers' Guide by Dave Thomas, Chad Fowler, Andy Hunt (ASIN 0974514055) for $44.95 Not yet released. Ruby Developer's Guide by Michael Neumann, Lyle Johnson, Robert Feldt, Lyle Johnson, Jonothon Ortiz (ASIN 1928994644) for $32.97 Usually ships in 11 to 12 days Sams Teach Yourself Ruby in 21 Days by Mark Slagell (ASIN 0672322528) for $28.39 Usually ships in 1 to 3 weeks Ruby In A Nutshell by Yukihiro Matsumoto (ASIN 0596002149) for $16.47 Usually ships in 24 hours The Ruby Way by Hal Fulton (ASIN 0672320835) for $27.19 Usually ships in 24 hours Making Use of Ruby by Suresh Mahadevan, Suresh Mahadevan, Rashi Gupta, Shweta Bhasin, Madhushree Ganguly, Ashok Appu (ASIN 047121972X) for $23.80 Usually ships in 24 hours Game Programming with Python, Lua, and Ruby by Tom Gutschmidt (ASIN 1592000770) for $26.39 Usually ships in 24 hours
Figure 1: Output from Listing One. Bullet CD by Trafik (ASIN B0002NRMMY) Only Fools And Horses - Sleepless In Peckham DVD (ASIN B0002PC39Y) Only Fools And Horses - The Complete Series 1 To 7 DVD (ASIN B0002XOZSS) Only Fools And Horses - Christmas Specials DVD (ASIN B0002XOZV0) The Office: Christmas Specials [2003] DVD (ASIN B0001WHUFK) The Office: Complete Box Set (Series 1 - 2 plus Christmas Specials) [2001] DVD (ASIN B0002IAQFY)
Figure 2: Output from Listing Two. 32
Dr. Dobb’s Journal, February 2005
From this point forward, new searches are not repeated if they were already conducted less than 24 hours ago. Amazon allows the caching and reuse of most data for 24 hours, although this changes from time to time, so check the latest terms and conditions (available in the API ZIP file). Using the cache facility can vastly improve the performance of your application, so it is used by default if you do not specify a preference. You can make life easier for yourself by specifying your usual requirements for new Request objects in a configuration file, to be found at either /etc/amazonrc or ~/.amazonrc. For example: dev_token = 'D23XFCO2UKJY82' associate = 'calibanorg-20' cache_dir = '/tmp/amazon/cache' locale = 'de'
Now, any Request objects you create inherit these characteristics. They can still be manually overridden as required, however. The underlying concept of Listing One can be inverted to produce a program that notifies you whenever a new album by your favorite artist appears in the Amazon catalogue for preorder. This works by looking for products whose availability contains the string “not yet released.” In Listing Two, the “favourites” array contains tuples of product names and catalog sections. The program then searches the relevant parts of the UK catalogue for matching products that have not yet been released. It may be useful to mail the output of this program to yourself on a weekly basis. When run, the output looks like Figure 2. As a final demonstration of the usefulness of this kind of monitoring, I present a program that is designed to be run automatically on a daily basis. It tracks the price of an item in the “electronics” section of the catalog and produces output whenever the price changes. Items in Amazon’s catalog can undergo quite a lot of price fluctuation within a relatively short period of time. Sometimes, the price of an expensive item drops $50.00 or more, only to rise again mere hours later. I don’t know why this happens, but knowing that it does and having AWS at your disposal lets you take advantage of these fluctuations to buy at a favorable moment. Have a look at Listing Three. Replace the ASIN with that of a product in which you’re interested and run the program once an hour or so, perhaps from cron. There are many kinds of searches that can be performed via AWS. You’ve already seen #keyword_search and #asin_search in the example programs, but there are other common search types, such as #actor_search, #artist_search, #author_search, http://www.ddj.com
and #director_search, as well as a slew of other, more esoteric searches. For example, if you have an Amazon wishlist, you can look it up with a line like this: r.wishlist_search( '18Y9QEW3A4SRY' )
The parameter to the method call is the alphanumeric wishlist ID. Or, if you want to get more specific and you’re looking for a book, the power search may be what you need. The following would find all books published by O’Reilly & Associates in the Amazon catalog and sort them by publishing date: r.power_search("publisher:o’reilly", 'books', HEAVY, ALL_PAGES, '+daterank')
This can take quite some time to return, as it will have to look up in excess of 1000 products. I won’t go into any of the other esoteric searches here. Read the AWS and Ruby/Amazon documentation, experimenting as you go, to get a feel for what is possible. One thing worth noting is that the AWS search methods can take many more parameters than illustrated by the programs I present here. For example, I’ve been performing “heavy” searches by default, which return all the information available for a given product. You can also specify a “lite”
http://www.ddj.com
search, which returns fewer properties and is, therefore, theoretically faster to execute. Use the Amazon::Search::HEAVY and Amazon::Search::LITE constants to specify the weight of the search. The weight is either the second or the third argument to the method, depending on the particular method call. Another parameter of interest is the page parameter. This specifies which page of up to 10 responses to return for the given search. If you play around with the various search methods without supplying a page number, you’ll quickly notice that you are never returned more than 10 products for a given search. This is because you’re only looking at the first page of results, since the default page to retrieve is the first. Instead of using the default of 1 (or any other integer), try passing the constant Amazon::Search::ALL_PAGES for a search that you suspect would yield a high number of hits, such as the power search. This constant signals to Ruby/Amazon that it should carry out as many background AWS operations as required to return all
hits for a given search query. If you chose a generic search, it could be quite some time before you get back your results. I now leave searches to concentrate on a different area of AWS where Ruby/Amazon can be of assistance. If you want to build your own online store, but don’t wish to invest in any stock, you can use the gigantic Amazon catalog as your inventory, have Amazon take care of the postage, packing, invoicing, and dispatch, and simply sit back and earn referral fees on any items sold via your site. One way to achieve this is the so-called “remote shopping cart” feature of AWS. This is one of the areas where Ruby/Amazon can take a lot of the work out of your hands, as the required tracking of shopping cart IDs and security tokens is handled internally by the objects you create. You won’t even know these elements exist, much less need to deal with them manually. Listing Four takes a list of ASINs from the command line, and adds each of the corresponding products to a new remote shopping cart. Finally, it spits out the purchase URL of the new cart; see Figure 3.
http://www.amazon.com/exec/obidos/shopping-basket?cart-id=002-1095509-6210461%26associate-id =calibanorg-20%26hmac=%2BWlRcRNiMD6KJ9UuUeNPMvqnVDY=
Figure 3: Output from Listing Four.
Dr. Dobb’s Journal, February 2005
33
If that URL were subsequently entered into a browser, the contents of the remote cart would be uploaded to the local shopping cart of the browser’s user. This feature lets you tie your users more closely to your online store by seamlessly integrating shopping-cart facilities in your own site. This alleviates the need to link away to Amazon for the actual purchase, which would carry with it the risk of the customer continuing his browsing from there, rather than returning to your site to potentially earn you more referral fees. One of the more advanced features offered by Ruby/Amazon is the option of customer geolocation. This is internally orchestrated via MaxMind’s GeoIP library (http://www.maxmind.com/). For this feature to work, you need to have GeoIP installed, along with the associated Ruby bindings (http://www.rubynet .org/modules/net/geoip/). The idea here is that you capture the host name or IP address of the customer connecting to your web site and then use that information to determine the Amazon site at which the customer is most likely to want to shop.
For example, a CGI (or mod_ruby, http://modruby.net/en/) script might contain something like this: require 'amazon/locale' require 'amazon/search' include Amazon::Search locale = Amazon::Locale::get_locale_by_addr ( ENV['REMOTE_ADDR'] ) r = Request.new( 'D23XFCO2UKJY82', 'calibanorg-20', locale )
If, for example, the customer were coming from 217.110.207.55, the locale of the Request object would now be set to de, which corresponds with amazon.de. All future searches would now take place within the German catalogue. You can use this to try to intelligently and transparently connect the customer with the product catalogue in which he or she is most likely to be interested. Your customers don’t have to be physically located in one of the actual countries that Amazon serves via AWS for geolocation of the locale to succeed. For
Listing One
example, Irish customers would be localized to the UK site, Swiss customers to the German site, and Canadians would find themselves localized to the U.S. amazon.com site. What else could you do with Ruby/ Amazon? Perhaps you maintain a database of your book collection, just to keep track of everything. Wouldn’t it be nice if you could associate each book in your collection with a picture of its cover? Well, one of the properties you can retrieve for books in Amazon’s vast catalog is image_url_large, which is a URL to a large image of the book’s cover (there are also small and medium variants). Alexandria (http://alexandria.rubyforge.org/) is a GNOME application written with this exact purpose in mind. It lets you enter and manage your book collection and calls upon Ruby/Amazon to fetch images of the items in it. Ruby/Amazon is a convenient, timesaving interface to Amazon’s AWS API. By extension, it demonstrates that Ruby is a serious contender for general webservice programming. DDJ
#!/usr/bin/ruby -w # Use the Ruby/Amazon library require 'amazon/search'
printf( "%s DVD\n(ASIN %s)\n\n", p.product_name, p.asin ) end end end end
# Include the constants, classes and methods from the Amazon::Search module include Amazon::Search
Listing Three
# # # # # # # # r
Create a new request object, passing in my AWS developer's token, my Associates ID, the locale in which I wish to work (the US or amazon.com in this case) and the fact that I do not wish to use a results cache.
#!/usr/bin/ruby -w require 'amazon/search' include Amazon::Search
Only the first parameter is required. The locale defaults to US and a cache is used unless specified otherwise. Even the need for the first parameter can be avoided by entering it in ~/.amazonrc.
# The iRiver iHP140 jukebox asin = 'B0001G6UAW'
= Request.new( 'D23XFCO2UKJY82', 'calibanorg-20', 'us', false )
# Search for for items related to Ruby programming in the books section of the # Amazon catalogue. Pass each product found into a block, calling the product # object p on each iteration. # r.keyword_search( 'ruby programming', 'books' ) do |p| authors = p.authors.join( ', ' ) # Information for books that are currently in print or soon to be available. unless p.availability =~ /not available/ printf( "%s by %s (ASIN %s) for %s\n%s\n\n", p.product_name, authors, p.asin, p.our_price, p.availability ) end end
Listing Two
# Quit if we didn't get a response exit unless resp new_price = resp.products[0].our_price begin # Read the price from the last check cur_price = File::readlines( '/tmp/price' ).to_s.chomp rescue Errno::ENOENT # File won't exist the first time we are run cur_price = '0' end # Quit if there's no price change exit if new_price == cur_price # Write new price out to file and stdout File.open( '/tmp/price', 'w') { |file| file.puts new_price }
#!/usr/bin/ruby -w require 'amazon/search'
printf( "Price of %s has changed from %s to %s\n", resp.products[0].product_name, cur_price, new_price )
# Construct an array of search strings and product catalog in which to search. favourites = [ [ 'global underground', 'music' ], [ 'the divine comedy', 'music' ], [ 'only fools horses', 'dvd' ], [ 'the office', 'dvd' ], [ 'minder', 'dvd' ] ] q = Amazon::Search::Request.new( 'D23XFCO2UKJY82' ) # I wish to search the British (amazon.co.uk) catalogue. q.locale = 'uk' favourites.each do |search_string, category| r = q.keyword_search( search_string, category ) r.products.each do |p| if p.availability =~ /not yet released/i case p.catalogue when 'Music' printf( "%s CD by %s\n(ASIN %s)\n\n", p.product_name, p.artists, p.asin ) when 'DVD'
34
# Look it up r = Request.new( 'D23XFCO2UKJY82' ) resp = r.asin_search( asin )
Listing Four #!/usr/bin/ruby -w require 'amazon/shoppingcart' include Amazon DEV_TOKEN = 'D23XFCO2UKJY82' sc = ShoppingCart.new( DEV_TOKEN ) ARGV.each { |asin| sc.add_items( asin, 1 ) } puts sc.purchase_url
Dr. Dobb’s Journal, February 2005
DDJ
http://www.ddj.com
GIS Web Services and Microsoft Word VBA and macros carry the load KEITH BUGG
G
eographical Information Systems (GIS) are making inroads into applications ranging from utilities and facility management, to city planning and security, among others. In this article, I present a GIS application that lets you embed a map into a Microsoft Word document simply by highlighting an address in the text. As such, this article is a convergence of several distinct technologies: GIS, Visual Basic for Applications (VBA), web services, and word processing. To build the sample program (actually, a Word macro), you need to have installed on your PC Microsoft Word 2002 (or higher), Internet Explorer 5.x (or higher), and the Microsoft SOAP Toolkit 3.0. However, to use the code I present here, you need to set up a web service on a protected intranet. You can easily replace my web service with your own or use a third-party subscription web service such as Microsoft’s MapPoint (http://www .mappoint.com/). Listing One is the code for the Word macro. The Word document that accompanies this article is available electronically; see “Resource Center,” page 5. To view and open the Word document, press ALT+F11. The execution of the sample program is a two-part operation: Keith is a software developer for Staff IT in Oak Ridge, TN. He can be contacted at
[email protected]. 36
1. Highlight a street address for Knoxville, Tennessee. 2. Right click your mouse and choose “Validate address…” from the context menu. If an invalid address is highlighted, a message box appears informing you that the address is invalid; otherwise, you are presented with a dialog box that lets you select various map “themes” (an aerial photo, zoning restrictions, and so on) as well as specifying a map size and scale. Nearly all the heavy lifting in this program is performed by a web service that lives on the Internet (or an intranet) and performs discrete operations; in this example, the web service first validates an address, then returns a map at the request of users. The beauty of web services is that you don’t have to know anything about their inner workings — you simply use them. Web services expose their methods, which makes it easy to know what data you have to pass in (in this example, an address as a string), and what you get back (again, a string representing the URL of a map). The other nice thing about web services is that they can be written in one language, but called from any other. My web service is written in Visual Basic; however, you could call it from a C# or C++ application. In short, web services go a long way toward delivering on the promise of seamless, heterogeneous, distributed computing. The KGIS Address Web Service The web service used by my Word macro is a product of Knoxville Geographic Information Systems (KGIS) of Knoxville, Tennessee. KGIS is responsible for maintaining all the geographical data for Knox County, and is a consortium of the city of Knoxville, Knox County, and the Knoxville Utilities Board (KUB). KGIS is thus responsible for supporting a wide Dr. Dobb’s Journal, February 2005
user base with diverse data needs. The primary GIS tool is ArcIMS from ESRI (http://www.esri.com/). ArcIMS generates the map that gets inserted into a Word document when an address is validated. The web service I use in this article does
“Nearly all the heavy lifting in this program is performed by a web service”
not actually return a map image, but rather, the URL of the map image. The main point here is that a web service exists that validates an address (for Knox County, Tennessee) and returns you a reference to a map of that address. The web service itself is integrated in a Microsoft Word document as a macro. This entire problem of inserting a map into Word can be broken down into three steps: 1. Create the macro that calls the web service. 2. Provide a way for the user to call the web service. 3. Do something with the result. http://www.ddj.com
To create a Word macro, select Tools-> Macro->Macros… from the toolbar. This launches a dialog that lists all current macros, plus, displays a Create button. Enter the name of your macro and click this button. This launches the VBA editor, which defaults to editing a subroutine with the same name as your macro from the previous screen. The code for my macro (available electronically; see Validate_Address( )) is responsible for reading the address that users select (highlight), and configuring the structures needed by the web service. This is done by using the Selection object; see Example 1. Saving the selected string is necessary for two reasons. First, you need to read the string containing the address. Second, if the address turns out to be valid and users want to see a map, I insert a blank line immediately below the highlighted address for inserting the map. I then rehighlight the address. Saving the starting/ending locations of the address string lets me do this. It also gives users visual feedback as to which address they selected. One final caveat regarding macros — you may need to adjust your security settings in Word before this macro executes. Select Tools->Macros->Security… on the toolbar and enable macros. Calling a Web Service The next step is to provide a way for users to call the web service from inside Word. For my sample program, I modified the context menu that appears when you right-click in Word; see Figure 1. Begin by clicking Customize under Tools on the main menubar of Word. Select the Toolbars tab and check the box next to Shortcut Menus. That opens the shortcut menu bar editor; see Figure 2. Next, select Text from the Shortcut Menus toolbar, and add the text for the
Figure 1: Modifying the context menu. http://www.ddj.com
context menu (Validate Address…). Then, on the Customize dialog, select Commands and then Macros. Add the name of the macro here; see Figure 3. The Code The code exists in a macro called Test( ). In the sample Word document (available electronically), you can access the code by pressing ALT+F11 to open the macro editor. The code declares many different variables; some of these are related to calling my map service. The generic variables are concerned with getting the text that users have highlighted, namely, sSelection. The contents of this variable must be checked for nonvisible ASCII characters. The only thing you want to pass to the web service is a valid street address. Example 2 does just that. sSelection = Selection iStart = Selection.Range.Start iEnd = Selection.Range.End
Once you have the address, you have to validate it against the KGIS database (again, your project would use your database). If valid, the dialog box in Figure 4 appears, letting users select the properties of the map for the given address. This dialog is called UserForm1 in the project. The Theme combobox is populated with the values Topographic, Aerial Photo, Case Parcel Map, and Zoning. The web service uses this information to return the type of map. Scale and Size are self explanatory. The Owner Data combobox is an additional criterion peculiar to the KGIS web service. Its values are Map ONLY, Owner Info., and Owner Info./ Parcel ID. Normally just a map would be returned, but the other options allow information about the owner of the address and the parcel to be returned as well. ' save the highlighted address
Example 1: Using the Selection object. sSelection = Selection iStart = Selection.Range.Start iEnd = Selection.Range.End iLastChar = Asc(Mid(sSelection, (iEnd - iStart), 1)) If iLastChar = 32 Or iLastChar = 116 Then iEnd = iEnd - 1 sSelection = Left(sSelection, (iEnd - iStart) + 1) End If
Example 2: Pass a valid street address to the web service.
Figure 2: The shortcut menu editor. Dr. Dobb’s Journal, February 2005
37
Dim sMapParams() As String sMapParams = Split(sMapURL, "|") Dim obj As InlineShape Dim sNewImage As String ' szTempDir = Environ("TEMP") ' get temp. folder szTempFileName = fso.GetTempName() ' get a temp. filename ' ' remove the .tmp extension and change to .jpg ' szTempFileName = Left(szTempFileName, 9) szTempFileName = szTempDir & "\" & szTempFileName & "jpg" sNewImage = sMapParams(0) ' err = URLDownloadToFile(0, sNewImage, szTempFileName, 0, 0) Selection.InlineShapes.AddPicture FileName:=szTempFileName,
Figure 3: Adding the macro.
SaveWithDocument:=True
Example 3: Inserting a map into a Word document.
Figure 4: Selecting the map properties. After users have made their selections on this dialog and clicked the OK button, the real fun starts. The structures used by the web service are completed and the web service is called using this line: sMapURL = myMap.wsm_GetMapByAddressMSLink (CLng(Val (myAddrInfo.AddrMSLINK)), myMapProps)
The web service returns the URL of a map that lives on the server. The next programming chore is to get this image into the Word document. This is done by first creating a temporary file in the user’s TEMP directory, then copying the JPEG map image to this file. Once on the local machine, it’s a snap to insert it into Word; see Example 3. The line in bold is a littleknown API call that does all the work. According to the documentation, its function is to “Download bits from the Internet and save them to a file”— exactly what you want. Finally, a Selection object is created for an InLineShape, and the map/JPEG is assigned to the AddPicture method. You now have a map in your Word document. Conclusion There are a number of Internet sites that provides maps identifying stores, points of interests, and the like. With the merging of GIS with the Office tools, you can greatly extend and enhance the utility of your GIS-related applications.
DDJ 38
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
Listing One Private Declare Function URLDownloadToFile Lib "urlmon" Alias _ "URLDownloadToFileA" (ByVal pCaller As Long, ByVal szURL As String, _ ByVal szFileName As String, ByVal dwReserved As Long, _ ByVal lpfnCB As Long) As Long ' This is the prototype for calling a Web service from inside a Word doc Sub Test() Dim sSelection As String Dim sValidAddr, sTheme, sScale, sSize As String Dim iStart, iEnd, iResult, iLastChar, k As Integer Dim selOrig As Selection Dim sMapURL As String Dim bLoop As Boolean Dim myMap As clsws_GetMap Dim myMapProps As struct_MapSettings Dim localFileName, szTempDir, szTempFileName As String Dim fso As New FileSystemObject Dim err As Long ' Set selOrig = Selection sSelection = Selection iStart = Selection.Range.Start iEnd = Selection.Range.End iLastChar = Asc(Mid(sSelection, (iEnd - iStart), 1)) ' Code to strip off a trailing , comma, space, or period bLoop = True k = 0 Do While bLoop ' iLastChar = Asc(Mid(sSelection, ((iEnd - k) - iStart), 1)) If ((iLastChar >= 65 And iLastChar = 97 And iLastChar \s∗([A-Z][A-Za-z0-9 .,:/\(\)\-'\"&;!]+)\s∗” doesn’t count. Then there may be some whitespace, followed by our text (enclosed in parentheses so that we can pull it out and do something with it), and finally more possible whitespace and the beginning of another HTML tag. The text itself must begin with a capital letter and may contain letters, numbers, and reasonable punctuation. The first regular expressions that I tried did well. Encouraged by these early results, I started adding more REs to cover more cases. The HTML, server-side JavaScript, client-side JavaScript, and C code all contained different kinds of text strings. So, for example, I wrote an RE to grab all the client-side JavaScript for further processing by other REs. The localizer was taking shape. The localizer filter made two kinds of mistakes: It might grab some code as a text string (a false positive), or it might ignore a text string as code (a false negative). False positives are easier to clean up than false negatives: You’re removing a key from the language table, rather than inventing a new key and inserting it into the language table in the right place. False positives are also easier to find. To test the filter by hand, I tweaked the localizer to replace each text string with a “marked” version in brackets, such as “{{My Favorite Device}}.” Then I ran the web interface and looked carefully at the text strings. A false negative showed up as an unmarked text string. But a false positive usually showed up dramatically; for example, as a blank page when the JavaScript code choked on a marked string. So in my testing, I was more lenient about false positives, and I wound up with about 50 false positives and only two false negatives. The false negatives began with “11b” and “11g” (as in the 802.11 wireless standards), which look more like code than text. Converter.java, the localizer code, is available electronically (see “Resource Center,” page 5). Testing Very soon I realized that this project was ideal for test-driven development (also known as “test-first development”). In this approach, you maintain a test suite together with the application. To add some functionality, you add a test, check that the test fails with the current code, and — only then — add some code to make the test pass. Although I’ve often included automated testing in my work informally, this was my first serious attempt to follow this approach, which is one of the core practices of Extreme Programming. I was inspired by Roger Lipscombe’s Roman numeral example (see http://www .differentpla.net/node/view/58), in which 48
some beautiful code grows organically before your eyes. The schedule on this project was somewhat relaxed, which allowed me to experiment a bit. (Or maybe it was the success of the experiments that kept the schedule relaxed; it’s hard to tell.) Here the test suite was a small set of web source files, together with localized versions that I prepared by hand. I periodically ran the localizer in a special mode, which processed the test files and compared the current output with the correct output. Whenever this comparison choked, I fixed the problem, rewriting the test or the code depending on which looked right. The test suite became a distilled version of the web codebase, containing all the interesting cases. The localizer turned out to be a strange application. First of all, it didn’t have to be perfect. If it could make the right decision 99 percent of the time, I could clean up the other cases by hand in a day or so. I figured there would be a cleanup stage. For example, I was assuming that brand names such as “AppleTalk” should not be translated, but the client might later decide that they should be. On the other hand, although perfection was not a goal, performance was important. The localizer was designed to run just once for real. Although I ran it many times to test it during development, once the official localization was done and I started editing the web files by hand, running the localizer again would wipe out that work. To justify writing it, it had to perform well on its single run. In fact, that’s how I decided when the localizer was done: when refining it further would take longer than the cleanup. But performance in terms of running time was not very important in this application. It ran the test suite in about eight seconds and chewed through the whole 2-MB codebase in about 15 seconds, which was a bit slow, but acceptable. Sometimes I combed through the localized output looking for bugs to add to the test suite. Sometimes a small change to an RE would trigger an unexpected problem, and I would thrash around changing REs until they worked again. Sometimes my code started to look ugly and I stopped to refactor it, relying on the test suite and the compiler to tell me if I’d broken anything. Sometimes I cheated on the test-driven process. I would find myself writing the code before the test, especially if both were trivial. Or I would fix a localizer bug by rewriting the web code to avoid some exceptional case. It might seem easier just to leave those exceptions to the cleanup, but doing even a small amount of localization by hand is tedious. Also, when I was writing the simple testing framework, Dr. Dobb’s Journal, February 2005
I didn’t bother adding a test for the framework itself. Actually, the step in the process where you break the old test, although it may seem pointless, provides a good check on the integrity of the test suite and framework. It guarantees that the test can catch bugs and hasn’t been accidentally disabled. But in the end, I was confident that the localizer was reliable — and written in nice clean code, too. It should be easy to reuse large parts of the localizer code. I doubt this will happen, though, because I imagine the requirements will be too different next time. We’ll probably just throw away the test suite along with the code. This is a shame because a test suite really shines in maintenance. But the general approach, as documented in the code and here, is still useful. The localization ended up taking about three programmer-weeks. This was well within our budget, but still seemed slow to me, although it was fun. I had hoped that test-driven development would magically turn me into a fast developer, which did not happen. But it fit well with an evolutionary style of development, in which you stay close to running code all the time and add features gradually. I find this philosophy comfortable, and it’s perfect for a small project such as this one. Translation The translation got off to a shaky start. I had only a narrow channel to communicate with the Japanese translator because of the time difference: I could post a message for him and get a terse response the next day. At first he was unclear about his part in the project, but I explained the project’s background in detail, and that seemed to help. I handed over the language table, which had about 1500 entries, and the translation came back surprisingly soon. However, it had the wrong number of lines. Although I could only read the keys, it seemed clear that some lines had been duplicated, merged, and generally mangled. This was not too surprising for such a long file. I wrote a little script to compare the keys in the English and Japanese versions, and that made it easy to fix the problems. What I was slow to realize was that I could keep this script around as part of the build process. It caught more mistakes as I continued to tweak the language tables. I was still absorbing a key idea of test-driven development: Automate anything you can. Programs are much better at repetitive checking than you are, so use them. The obvious character encoding to try was UTF-8, the 8-bit Unicode Transformation Format created by Rob Pike and http://www.ddj.com
Ken Thompson (http://www.ietf.org/rfc/ rfc3629.txt/). Among other things, it’s designed to work well with C code, and it’s well supported by web browsers. In short, it worked beautifully. The Japanese strings passed successfully through the web server C code and the client-side JavaScript, and — at least on my machine — were correctly displayed by the browser using the character set already installed in Windows. I was not able to type Japanese text into the input controls, but I copied and pasted some from my browser, and that seemed to work fine, too. At one point, I noticed some suspicious characters in the Japanese text in the browser — something like “?A?” Looking at the language table in a hex editor and using Markus Kuhn’s reference (http://www .cl.cam.ac.uk/~mgk25/unicode.html), there did seem to be some bad UTF-8 characters. I found out that I had corrupted the language file text strings somehow while editing. (What do you expect when different people are using Word and Emacs to edit the same large file?) Because the keys were not affected, the checking script I had written did not catch the problem. I decided to integrate some UTF-8 verification into the language build. First, I looked around for existing Java support but was disappointed. Java supports UTF8, but apparently does not support debugging UTF-8 text. For example, if b is a byte array, you can convert it into a string using either
ough coverage of the various kinds of characters. Conclusion Next time, I might try different tools because mine had some problems. For example, for efficiency I had to use little scripts to build and run the code, which might have been easier with an interactive scripting language like Python. A formal build tool such as Ant might also have helped. Sun’s free Java SDK is nice, but at one point the Java Virtual Machine began crashing with weird exceptions, which I fixed by upgrading. The Java code wound up at around 600 lines, which seems long, and as I mentioned, it was kind of slow. The Apache Regexp documentation was sketchy (what’s the prece-
dence of the various operators?). But overall, I was happy with my choices. I would try something new mostly for my own education. I would definitely try the test-driven approach again. I won’t use it all the time (for example, I’m still frustrated by the difficulties of writing automated tests for GUI applications), but I’ll use it whenever I can. And I’ll recommend UTF-8 to all my friends. Finally, when you’re facing weeks of tedious coding, consider automating it somehow. Programming should be interesting; if it’s repetitive, something’s wrong. A quick experiment could save you a lot of time. DDJ
new String(b, "UTF-8");
or (new DataInputStream (new ByteArrayInputStream(b))).readUTF();
But neither of these techniques identify the location of a bad character, and in any case, Java has its own, slightly incompatible version of UTF-8. So I wrote my own UTF-8 verifier, taking this as an opportunity to try out the popular JUnit testing framework (http://www.junit.org/). JUnit was lightweight — I hate a giant download — and easy to incorporate. My favorite feature was the assert methods built into the TestCase class, which were handy for expressing what’s supposed to be true. Also, it was reassuring to see the brief report of tests passed and time elapsed. JUnit has a bunch of features that only kick in for larger projects, such as a nice way to group test cases into suites, but already I could see the appeal. (Check Utf8.java; the UTF-8 checker is available electronically.) Again, I found that the test-driven approach produced satisfying, clean code — slowly. The basic algorithm came out in three short methods. I spent much more time on the test code, putting in thorhttp://www.ddj.com
Dr. Dobb’s Journal, February 2005
49
Algorithms for Dynamic Shadows Creating geometrically correct dynamic graphic images SERGEI SAVCHENKO
S
hadows are among the most noticeable visual cues and are important for human perception. Consequently, accurate rendering of shadows is paramount for realistic computer-generated images. In this article, I examine five algorithms that generate dynamic shadows in 3D graphics applications. I’ve implemented these algorithms in the Small Dynamic Shadows Library (available electronically; see Resource Center, page 5), a freely available OpenGL-based library. Finding regions of space shadowed from a given light source implies solving the visibility problem for that light source. In other words, if a viewing camera is to replace the light source, all points in space visible for that camera should be illuminated, whereas all points hidden from this imaginary camera are shadowed. In general, generating shadows for moving or animating objects is more difficult than producing static shadows. A common, albeit simplistic, approach is to draw a dark spot underneath moving objects. Even when approximate or geometrically incorrect, this is visually better than not having any shadows at all. The dark spot depicting the shadow can have soft edges, further improving its appearance. Many popular video games (from “Return to the Castle of Wolfenshtein” to “Grand Theft Auto 3”) rely on this technique. One apSergei is the author of 3D Graphics Programming: Games and Beyond and can be contacted at sergei_savchenko@ hotmail.com. 50
proach used to improve the appearance of this spot shadow is to elongate it into the direction opposite to the predominant light source. Another approach is to draw multiple intersecting dark spots. This works well in many instances, especially low-light intensity environments. Although faking dynamic dark spot shadows is still in use, it is hardly enough to draw a compelling picture of the virtual world. Luckily, with the improvements in computer performance, it is now possible to consider more advanced approaches producing geometrically correct dynamic shadows. Planar Shadows One popular solution for a constrained situation where dynamic shadows are always cast onto a plane is to find a projection matrix that squishes the object to that plane and draws the entire model the second time with the found transform and an ambient dark, near black color (see Figure 1). Although constrained, such a situation often occurs in sport games that represent playing fields and players casting shadows. It is not too difficult to find the required transformation matrix. If a normal to a plane where the projection is sought is n and some point on that plane is P, then the plane equation could be represented as n(X – P)= 0, where X is any point on that plane. Given a light source L and an arbitrary point on the model M (Figure 2), Example 1(a) holds because the distance from the light source to the projection point X – L relates to the distance from the light source to point M in the same way the distance from the light source to the plane relates to the distance from the light source to the point M projected onto the direction of the normal n(L – M) (Figure 2). Thus, the projection point relates to the point on the model as in Example 1(b), or in vector notation, represented as in Example 1(c). Thus, this transformation, with some simple arithmetic, can be described as a 4×4 matrix transforming Dr. Dobb’s Journal, February 2005
any point on the model M into a point on the plane X; see Example 1(d). However, there are several potential complications with this algorithm, starting with the possibility of Z fighting. After all, the shadow model is drawn on top of the previously drawn plane. Thus, Z values of the two will be near equal, resulting in some pixels from the shadow appearing
“To improve performance, you could use separate shadowing and drawing models” intermittent with the pixels from the plane. A common solution is to push the projection plane slightly above the shadowreceiving plane. A second problem exists due to the matrix, which you have derived; it projects onto an infinite plane, where what you usually want is more limited (a table top rather than an infinite ground plane, for example). Stenciling techniques can help. When the surface receiving the shadow is drawn, you also simultaneously draw into the stencil buffer, whereas when the shadow model is drawn, the stencil test is performed to check if the current pixel also belongs to the previously drawn limited surface. In many situations, a destination alpha test can be used as a substitute for a stenciling test. Finally, to improve performance, you could use separate shadowing and drawing models. Because shadows (as important as they are) are not what catch the eye http://www.ddj.com
first, shadow models could be considerably simpler and use a smaller number of triangles than the rendering models. Projective Textures The main drawback of the planar shadows is, of course, the severe limitation that is placed onto shadow receivers. Quite often, it is necessary to cast shadows onto objects of an arbitrary shape. Texture mapping and procedural generation of texture coordinates can help. In most graphics pipelines, it is possible to specify a texturecoordinate generation matrix that is applied to spatial coordinates. The result of this multiplication is used as texture coordinates. Thus, you can produce a shadow by first computing a black-on-white image of the shadow caster (see Figure 3) as seen from the position of the light source, further using it as a texture for shadow receivers whose texture coordinates are generated procedurally based on the view transformation as observed from the light-source position. The first step of the algorithm is to draw the black-on-white image of the shadow from the position of the light source. Assume that the light source is positioned at point L, and some point on the model M is projected into the texel T (see Figure 4). The first step of the transformation is the rotation of the object-space point into the light-space coordinate system. The latter is chosen so that its k axis (Figure 4) points along the direction from the center of the shadow caster to the position of the light source. Directions of the other two axes are not particularly relevant (since the orientation of the image in the
plane is not that relevant), so you can arbitrarily choose the i axis to be perpendicular to the Y axis of the object space. The last axis j is mutually perpendicular to the first two. Thus, the axes of the light-source coordinate system can be computed as in Example 2(a). Knowing the axes, the rotation part of the transformation from the object space into the light-source space can be represented by Example 2(b). To project the point, however, you also need to translate the result of the application of Example 2(b) along the view direction by the distance from the light source to the shadow caster. Furthermore, you apply the perspective transform that determines how big the object is in the image, and finally you potentially need a view-port transform (not represented in the formula) to position the shadow caster’s projection in the middle of the image; see Example 2(c). The perspective transform lets you adapt the size of the projected shadow caster to occupy the entire texture. This transform thus depends on the size of the texture buffer and the bounding radius of the shadow caster. For the second step of the algorithm, you use a rendered image of the shadow caster as a texture map on shadowreceiving objects. The same transform as that used in the projection is also used for procedural texture-coordinate generation. Thus, a vertex on the shadow receiver obtains a proper texture coordinate corresponding to the view from the light source. However, you may have to alter the view-port transform since the range
of texture coordinates may be limited and different from the view-port transform used in the projection. As Figure 5 illustrates, this algorithm lets you cast shadows on objects of an arbitrary shape. There are several drawbacks to this algorithm. First, a shadow is obtained on both front- (light-source facing) and backfacing sides of the shadow receiver. It is because there are points on both sides of a closed object that map to the same shadow map texel according to the transform that we have found. You can deal with this problem by choosing very low-ambient illumination so that the back-facing polygons are very dark. It is also somewhat difficult to take care of the concave shadow receivers with this method, particularly when the shadow caster happens to be placed in one of the cavities of the receiver. Suppose a single
Figure 1: Planar shadows. L M
(a)
(X—L) n(L—P) = (M—L) n(L—M)
X n
P
(b)
X=L + nL—nP (M—L)=(nL—nM)L+(nL—nP)M—(nL—nP)L = nPL—nML+(nL—nP)M nL—nM nL—nM nL—nM
Figure 2: Projection onto a plane.
(c)
nPLx—(nxMx+nyMy+nzMz)Lx+(nL—nP)Mx nL—nxMx—nyMy—nzMz) x y = nPLy—(nxMx+nyMy+nzMz)Ly+(nL—nP)My nL—nxMx—nyMy—nzMz) z nPLz—(nxMx+nyMy+nzMz)Lz+(nL—nP)Mz nL—nxMx—nyMy—nzMz) (d) nL—nP—nxLx —nyLx —nzLx nPLx x nPLy —nxLy nL—nP—nyLy —nzLy y = —nxLz —nyLz nL—nP—nzLznPLz z 1 —nx —ny —nz nL
Mx My Mz 1
Example 1: Planar shadows. http://www.ddj.com
Figure 3: Shadow texture. Dr. Dobb’s Journal, February 2005
51
Despite these drawbacks, this is one of the more popular algorithms. A large number of excellent games (from “SSX” to “MGS2”) use variations of this approach.
model is representing a tunnel through which a shadow caster is moving — the shadow may potentially be mapped onto both the floor and ceiling of this tunnel. Because both the floor and ceiling are modeled by the same object (and thus, use the same texture-coordinate generation matrix), it is impossible to suppress the drawing of an extra shadow. Consequently, it may be necessary to subdivide the scene further and use more complex line- of-sight calculations to determine which objects are casting shadows onto which other objects. Generally, extra calculations of which objects cast shadows onto which objects complicate practical implementations of this algorithm considerably. The algorithm is also sensitive to the resolution of the texture where the shadow image is rendered. If it is too small, the shadow is blocky. If that is not enough, with this algorithm, it is hard to model multiple objects casting shadows onto the same receiver. However, there is an elegant solution that falls somewhat in between this algorithm and that of shadow Z buffer.
Shadow Volumes Besides the drawbacks of projective textures just described, there are effects that are simply impossible to do with their help alone; for instance, self shadowing is quite problematic. The shadow volumes algorithm overcomes this difficulty. A shadow volume is an extrusion of the silhouette of a shadow caster that delimits all points in space shadowed by this caster. You can build the volume by finding the silhouette edges of a model and extruding them away from the light source. The shadow volumes algorithm draws shadows as a postprocessing step. It works by first drawing the scene without any shadows and then drawing (in a particular way) the shadow volumes into a separate image while using the Z buffer of the original image. It, thus, becomes possible to find all pixels that should have been shadowed and then we can adjust the image of the scene by j i
k
L T M y x z
Figure 4: Projective texture. (a)
k=
(0 1 0)×j L—M i= (0 1 0)×j L—M
j=
k ×i k ×i
blending with, or stenciling against, the found shadow mask. The key idea of the algorithm is to first borrow the Z buffer from the drawn image of the scene. When shadow volumes (which are assumed to be closed polyhedrons) are drawn, their pixels are drawn an odd number of times for the shadowed regions and an even number of times for the illuminated regions (Figure 6). Indeed, because object B is shadowed, only the front side of the shadow volume is drawn (the backside is prevented from drawing by the Z test, since you have borrowed the Z buffer from the original scene where object B was drawn). This is not the case with object A and the other shadow volume. This observation lets you formulate the following algorithm: First you draw the scene as is, without any shadow volumes. Following that, you continue using the Z buffer from the first image, but switch off Z buffer updates. If you have a stencil buffer permitting positive and negative values, you draw the front faces of the shadow volumes with value 1 into the stencil buffer (initialized with all zeroes) and draw the back faces with the value of –1. Consequently, you obtain a stencil where illuminated pixels are marked with a 0 and shadowed pixels with a positive integer. You can now apply the stencil to the original image so that only illuminated pixels remain in it. A variation of this approach is possible if the graphics pipeline supports flexible accumulation of colors. In such a case, you can create a binary image where shadowed pixels are black, and blend such an image over the original scene image. Note that you need to paint front- and back-facing sides of the shadow volumes in different colors to account for a situation where two shadow volumes intersect. The difficulty of this algorithm is, of course, in the necessity to build the shadow volumes. Even with static light, you may have to recalculate the shadow volumes for dynamic objects whose silhouette from the position of a light
(b)
ix jx kx 0
iy jy ky 0
iz jz kz 0
0 0 0 1
(c)
Tx Ty = Tz 1
ix iy iz 0
1 0 0 0
1 0 0 0
0 1 0 0 0 0 1 0 0 0 1f 0
0 1 0 0 jx jy jz 0 0 0 1 —d kx ky kz 0 0 0 0 0 0 0 0 1
Mx My Mz 1
Figure 5: Shadow texture projected onto a sphere and a plane.
Example 2: Projective textures. 52
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
source may change every frame. To build a shadow volume, you have to extrude the silhouette edges in the direction away from the light source (Figure 7). An edge in the model is a silhouette edge if one of the triangles joined at that edge faces towards the light source, whereas the other triangle faces away. This fact can be easily tested by computing the dot product of the triangle’s normal and the direction of view for this triangle. Thus, by traversing all edges and computing two dot products per edge, we find all silhouette edges. To do that efficiently, you can precompute edge-based representation of the model where two triangles are given for every edge. In real life, many models have some artifacts that complicate the process just described. Some edges may hang so that they have only a single triangle. Other edges may be joins of more than two triangles. The most reasonable solution is to edit models designated for use with this algorithm in such a way that they are closed polyhedrons and don’t contain the situations just described. At the very least, you can use special, refined shadow models for generation of the shadow volumes, whereas less strict drawing models can be used for rendering. Such strategies are more reasonable in practice than trying to complicate a silhouette-detection algorithm by searching closed loops of edges and so forth. The length of the volume is also difficult to compute. If the length is too long, much of the fill rate of the graphics accelerator is wasted. You may also get into a situation when a shadow volume is projected onto a thin wall and you observe a shadow on an opposite side of that wall just because the volumes are too long. If the length is short (and the volumes are open), this can produce nasty artifacts. Indeed, it is common not to close the volumes from above (the model itself fills that gap) nor from the bottom that likely intersects with shadow receivers of the world. There are instances, however, when closing the volumes is still necessary. If the camera moves inside a shadow volume, the clipping plane opens the volume and interferes with the algorithm, and you can’t determine shadowed pixels correctly because the algorithm is based on the assumption that the volumes are closed (thus, every illuminated pixel is drawn at least twice — once from the front-facing surface of the volume and once from the back-facing surface). One solution is to introduce an extra shadow-drawing pass (at least for the volumes that are likely to be clipped by the front plane). In this pass, you disable the Z test and reverse the integers used to paint front and back faces. The effect of this operation is that you compute the http://www.ddj.com
caps on the shadow volumes and can proceed with the actual drawing of the volumes without worrying about a situation where the camera may be inside. Shadow Maps Shadow volumes have several significant shortcomings — one is the necessity to construct volumes for dynamic objects onthe-fly every frame. There is an old alternative algorithm that requires neither building triangles dynamically nor severe limitations on an object’s shape — the shadow maps algorithm. In its classical form, the shadow maps algorithm requires some special hardware support. Nonetheless, there are variations of it (or approximations) that can be done in software only.
The classical shadow-maps algorithm requires access to the Z buffer. It first computes the image of the scene from the position of the light source. The Z buffer of this image is saved as a texture and mapped onto the scene to be viewed by the camera. This could be done in a manner described for projective textures. Of course, Z values stored in the texture correspond to points visible from the light source. If for any point, you can compute the distance to the light source directly and compare it with the texel value that is mapped onto this point, you can determine if the point is illuminated by the light source. If the distances are the same, it is illuminated; if the distances are different (notably, the distance computed directly is larger than that stored in the
L
V A B
Figure 6: Crossing shadow volumes.
Figure 7: Wire frames of constructed shadow volumes and the resulting image. B
C
A
L V
Figure 8: Shadowed and illuminated points. Dr. Dobb’s Journal, February 2005
53
corresponding texel), it implies that something else was closer to the light source and the given point is shadowed. In Figure 8, point B is shadowed because its distance to the light source is larger than the distance that was recorded in the texture (notably, the distance to point C). This is not the case for point A, which is illuminated. One step of this algorithm (notably, computing the distance to the light source from some point when viewing the scene I(d)=Ilight
1
Cconst+Clineard+Cquadraticd 2
Example 3: Shadow maps.
Figure 9: Shadow map texture.
by the camera) normally requires some hardware assistance, or at least flexibility in the graphics pipeline. However, it is still possible to approximate this algorithm by using only commonly available operations. Illumination calculations for attenuatedpoint light sources allow coloring objects based on the distance to the light sources. Blending several images together permits you to mark the pixels that should have been shadowed. Consider this algorithm: You first draw the image of the scene from the position of the light source. You only use ambient white light for this light source that is linearly attenuated with distance. Normally, three attenuation coefficients are present in most renderers and the intensity of the attenuated light is computed as in Example 3. For this algorithm, you only need a linear attenuation coefficient set to a value dependent on how large the scene is — that is, dependent on where this light should stop producing any contribution. Thus, with such illumination set up, pixels that are closer will appear brighter, whereas pixels that are further away from the light source will be dimmer (see Figure 9). As the next step of the algorithm, you compute two images from the position of the camera. In one image, you project onto all objects the texture computed during the previous step using projection from
Figure 10: Obtaining shadow mask.
Figure 11: Shadow mask and resulting image. 54
Dr. Dobb’s Journal, February 2005
the light source. It is done with the help of the mechanism described for projective textures. For the second image, compute the scene with the same attenuated light source as was used on the first step. Figure 10 demonstrates the two images, which contain the approximations to distances to the light source. The color of the point closest to the light source also colors all pixels further along the way in the left image. In the right image, every pixel is colored depending on its own distance only. By subtracting from the value of every pixel in the left image the value of the corresponding pixel in the right image, you can mark shadowed pixels. If the result of subtraction is positive, the pixel is shadowed; if it is zero, it must have been illuminated (see Figure 11). By scaling the values of the shadow mask up and clamping from above, you can obtain a binary shadow mask where all shadowed pixels are white and illuminated pixels are black. Note also that negative values indicate regions that were not mapped with the computed texture in the left image of Figure 10. These points could be considered as illuminated. As the last step, you draw the scene with normal illumination and subtract from this image the shadow mask. That is, from every pixel of the image, you will subtract from its value the value of the corresponding pixel of the shadow mask. The right image in Figure 11 illustrates the result. Note that in the case of OpenGL, the subtraction of images can be done with the help of the accumulation buffer. A significant drawback to this algorithm is due to potential artifacts that may appear on open models illuminated from the inside. A pixel illuminated from the back face of a polygon is not distinguished from the pixel illuminated from the front. Thus, if a light shines into an opening on an object, you may observe a light spot on the side of the object opposite to the opening. The algorithm is not sensitive to these situations. Better modeling of objects ensures that there is no one polygon. Thick surfaces will help avoid these problems. The most significant drawback, however, is the necessity to do multiple rendering passes per every light source producing shadows. On the positive side, the algorithm is very general and inherently produces self shadowing without placing significant limits on the shape of the models in the scene. Finally, there is no need to construct geometry on the fly— an expensive process. Priority Maps The main difference between shadow maps and priority maps is that shadow maps compute approximations of distances per pixel, while priority maps do http://www.ddj.com
it on a per-object basis. By sorting all objects based on their distance to the light source, you can assign ambient color to every object so that the closest is the brightest, and the farthest is the dimmest. The following steps are similar to the shadow maps algorithm. The texture map is drawn from the position of the light source using assigned ambient colors. Furthermore, the camera image is drawn twice — once using a previously computed texture map projected onto all objects, and the second time using the object’s assigned colors. The two images are subtracted to obtain the binary shadow map that is then subtracted from the actual image drawn using proper illumination of the scene. As the resulting image illustrates (see Figure 12), the ability to produce self shadows has been lost. Indeed, the entire object is drawn with the same color, and thus, only the shadows produced by other objects can be computed. The use of sorting in this algorithm also assumes that the objects are not particularly elongated and placed in complex configurations. Otherwise, sorting may not be able to place all objects into an unambiguous order. The greatest advantage of this algorithm is the simplicity with which it handles mul-
http://www.ddj.com
Figure 12: Priority map texture and the resulting image. tiple shadow casters on multiple shadow receivers, something quite complicated in the case of projected textures alone. Conclusion All of the algorithms I’ve described have strengths and weaknesses. Planar shadows limit the shape of shadow receivers to planes only. Projective textures have difficulties with multiple casters on the same receiver and erroneously produced shadows. Shadow volumes require generating additional geometry and need multiple drawing passes. Shadow maps need
Dr. Dobb’s Journal, February 2005
too many passes and tend to produce artifacts as do priority maps. On the other hand, planar shadows are trivial to implement, as are projective textures. Shadow volumes permit self shadowing and easily allow for multiple casters on the same receivers. Shadow maps don’t require building any data structures dynamically and place only moderate limitations on the models. Priority maps inherit many good features of projective textures and shadow maps. DDJ
55
PROGRAMMER’S TOOLCHEST
Extending UML Automatic production of user interfaces TIMOTHY E. MEEHAN AND NORMAN CARR
W
e develop complex scientific applications and integration between different instrument, inventory, and financial systems. To get a better handle on scope and requirements, we began using UML. However, the more we used Activity Diagrams to model the end-user tasks and user interaction at the screen, the more we recognized a gap in UML — there is no way to actually model user interaction that illustrates the steps and required actions users perform to complete their tasks. Upon examination, we identified a pattern of what users needed to do on screen — enter this, read that, select next, and submit all. We then saw a direct correlation between user function and the type of GUI elements we were using to perform that function. Consequently, we devised what we call “Extended Activity Semantics” (XAS) to model the business tasks and required user interactions to complete those tasks. From this mapping, we can automatically generate GUI eleTim is a principal software engineer and Norman a web- site designer for Nuvotec and coinventors of Extended Activity Semantics. They can be contacted at
[email protected] and
[email protected], respectively. 56
ments and simultaneously develop more accurate requirements. In this article, we present XAS notation and show how you can use it to design user activity. We then describe how the notation can be directly mapped to GUI components to simultaneously construct user interfaces. By applying a simple heuristic between the notation and UI components, we generate a construct for the forward and reverse engineering of user interfaces and activity diagrams. To illustrate how XAS notation can be used, we present Guibot, a CASE tool we developed. Background UML provides a good foundation for forward and reverse engineering of class diagrams, and there are numerous tools that produce lots of software based on UML class diagrams. However, for user activity and accompanying user interfaces, class diagrams leave a flat, incomplete view of the user activity. Sequence diagrams and collaboration diagrams produce a dynamic view coupling user activity to the GUI functions and middleware, but they don’t translate well to end users during elucidation of use cases and functional requirements. In short, GUI designers find UML cumbersome and complain that it produces unnecessary work rather than eliminating it. Compounding this problem are the user stories that analysts typically produce as verbose written narratives. Furthermore, user storyboards implemented through written narratives leave too much interpretation and inference by code developers because a narrative offers no inherent mechanism for verifying its own completeness. This deficiency is underscored Dr. Dobb’s Journal, February 2005
by industry practices, which are still plagued by problems of bridging the communication gap between those who consume software and the technologists who create it. Often, prototype user interfaces are used to elicit end-user requirements, but they suffer from only providing a static view of user activity (a “snapshot” view), and fail to express a dynamic model of
“GUI designers find UML cumbersome” the end-user’s workflow. By modeling the task-based steps necessary for the end user to achieve the goal/purpose of a use case, UML activity diagrams overcome many of these problems. They succeed in illustrating the scenarios of the tasks performed by end users to achieve their goals, yet to stakeholders unfamiliar with activity diagrams, they are insufficient to provide a clear picture of the software system “to be.” From our experience of elaborating software requirements for complex systems, domain experts and stakeholders are hungry for such visualization early in the specification of a new software project. Using existing tools, we have found that the best http://www.ddj.com
way to provide such vision to stakeholders is via an approach employing both static UI prototypes and activity diagrams to complement one another. This method yields greater end-user involvement in the design and specification of user activity, provides better user satisfaction, and results in a more accurate and detailed definition of end-user requirements early in the project lifecycle. Subsequently, through such a process of drafting user prototypes in parallel with activity diagrams, we have determined that user interaction at the interface could be directly represented by a notational supplement to activity diagrams, which in turn can be mapped back to GUI components, which we employ in our Guibot tools. UI Behavior Users perform defined sets of irreducible actions with respect to user interfaces — input, review, select, and command. We represent these actions as inputters, outputters, selectors, and action invokers. User actions such as drag-and-drop are composite actions of these irreducible actions, where drag-and- drop would be select-and-input. In addition, there is a multiplicity— formatting of the information in to and out of the system and conditions surrounding the information. The Multiplicity denotes whether the action is required or optional, or whether it requires multiple actions, such as one or many selections to be made. Formatting describes the presentation of the information, such as the format of a date input or output. The Conditions for an action define what is allowed or needed, such as a “date entry must be earlier than today,” or a “password must be longer than some number of characters.” In the use of activity diagrams, prose in the business language of the end user is generally written to describe these actions and constraints. Although the UML specification does state that pseudocode and programming language code can be used in developing activity diagrams, in the context of eliciting requirements from end users, we have found their own business language is the most effective. As an extension to activity diagrams, we added the following XAS notation to translate such “business actions” into user interactions:
• Multiplicity: m..n defines the multiplicity of each user interaction. • Label : DataType, which specifies the plain-language label of the data and the data type, or the command for the action invoker. • Mask/Filter: |, where the data format is represented. • Condition: [ ], where any conditions applied to the interaction are specified. The complete grammar for inputters, outputters, and selectors, therefore, is respectively expressed as: >>,, where the user provides information to the system. • Outputter: Outputters ToString()); } }; void main() { Test^ t1 = gcnew Test(10); Console::WriteLine(t1); Test^ t2 = gcnew Test("hello");
S10
Console::WriteLine (typeid< Dictionary >);
prints this at the command line: System.Collections.Generic.Dictionary`2 [System.Int32,System.Int32]
In other words, the Dictionary type has two type parameters (indicated by `2), and the constructed type has provided Int32 for both of these parameters. In the current beta, it is not possible to get the type object for a generic type: You must provide type parameters. DDJ
Console::WriteLine(t2); Test^ t3 = gcnew Test(gcnew ArrayList); Console::WriteLine(t3); }
Listing Two generic where T : IDisposable ref class DisposableType { T t; public: DisposableType(T param) {t=param;} void Close() { t->Dispose(); } };
Dr. Dobb’s Journal Windows/.NET Supplement, February 2005
http://www.ddj.com
Listing Three interface class IPrintable { void Print(); }; ref class Printer { public: generic where T : IPrintable void PrintDoc(T doc) { doc->Print(); } };
Listing Four // Old Syntax String* Create(int i) __gc[] { return new String __gc*[i]; } // New Syntax array^ create(int i) { return gcnew array(i); }
Listing Five using namespace System; using namespace stdcli::language; double average([ParamArray] array^ arr) { double av = 0.0; for (int j = 0 ; j < arr->Length ; j++) av += arr[j]; return av / arr->Length; } void main() { Console::WriteLine("average is {0}", average(9, 3, 4, 2)); }
Listing Six const int SIZE = 10; array^ a = gcnew array(SIZE); interior_ptr p = &a[0]; for (int i = 0; i < SIZE; ++I, ++p) *p = i;
Listing Seven String^ s = "hello"; interior_ptr ptr = PtrToStringChars(s); interior_ptr p = const_cast< interior_ptr >(ptr); Console::WriteLine(s); // prints hello *p = 'H'; Console::WriteLine(s); // prints Hello
Listing Eight // Illustrating the danger in interior_ptr using namespace System; using namespace stdcli::language; ref class Bad { __int64 x; __int64 y; String^ s; public: Bad() { x = 1LL; y = 2LL; s = "test"; } void KillMe() { Dump(); interior_ptr p = &x; *p = 3LL; // Change x Dump(); p++; *p = 4LL; // Change y Dump(); p++; *p = 5LL; // Change s // Dump will kill the process when it tries to access s Dump(); } void Dump() { Console::WriteLine("{0} {1} {2}", x, y, s); } }; void main() { Bad^ bad = gcnew Bad; bad->KillMe(); }
DDJ http://www.ddj.com
Dr. Dobb’s Journal Windows/.NET Supplement, February 2005
S11
WINDOWS/.NET DEVELOPER
Enhancing .NET Web Services Extending and debugging web services ERIC BERGMAN-TERRELL
C
alling a web service from a .NET application couldn’t be easier. Just select the Project/Add Web Reference menu item. Add a URL to the WSDL file that describes the web service. Press Add Reference and a proxy class is automatically created. This proxy class includes methods corresponding to each of the web service’s methods. Calling any of these proxy class methods generates a SOAP (XML) request, sends it to the server, retrieves the SOAP response, and converts it into a response object. But what if you want to request compressed responses, or access or even change the requests and responses? In this article, I show how to ask for compressed responses, capture and optionally modify SOAP requests/responses, and debug webservice calls with a network packet analyzer program. I’ve included SoapEx, a sample application (available electronically; see “Resource Center,” page 5) that demonstrates these techniques. SoapEx (see Figure 1) is a .NET Windows Forms program written in C#. It requires the .NET 1.1 framework and Windows 2000/XP or later. Build it in Visual Studio .NET 2003 by opening SoapEx\SoapEx.sln. You can also Eric has developed everything from data reduction software for particle bombardment experiments to software for travel agencies. He can be contacted at ericterrell@ comcast.net. S12
install the app by running setup.exe in the setup folder. When you run SoapEx, it calls the Google web service. Check the Compress Response checkbox to request a compressed response. Check the Modify Soap Request and Response checkbox to change the SOAP requests and responses. You’ll need a license key to use the Google web service. Press the Get License Key button to get one, or go to http:// www.google.com/apis/. In either case, it will take only a minute or two to apply for a license key and receive it via e-mail. Press Call Web Service to send the webservice request and retrieve the response. The SOAP request/responses are displayed in the Results text box. Pressing the IP Addresses button displays the IP address of your machine and the server hosting the web service. These IP addresses are useful for troubleshooting web-service calls with Packetyzer. Many web services can return results in compressed format to conserve bandwidth. Given the verbosity of XML, compression can drastically reduce bandwidth consumption. Unfortunately, .NET web-service proxies do not ask for compressed results by default. But it’s easy to correct this situation. The first step is to create a new class derived from the web-service proxy class. (For example, SoapEx has the CompressedGoogleSearch class, which is derived from the GoogleSearchService proxy class.) Then override the derived class’s GetWebRequest method to specify an HTTP header Accept-Encoding value of gzip, deflate (Listing One). The Accept-Encoding header specifies the compression formats that the client can cope with. The SoapEx application can decompress both gzipped and deflated responses. The second step is to override your derived class’s GetWebRe-
sponse method to return a CompressedWebResponse object. The CompressedWebResponse object will be able to decompress web-service responses. When the server returns the results, they are in gzip format, deflated format, or uncompressed. Bear in mind that the
“Many web services can return results in compressed format to conserve bandwidth” Accept-Encoding header is only a suggestion, not a demand. The CompressedWebResponse.GetResponseStream method decompresses the response Stream if it is actually compressed (Listing Two). GetResponseStream determines the response format by inspecting webResponse.ContentEncoding (see the code inside the try block). If the response is compressed, the correct inputStream is selected: GZipInputStream for gzip, InflaterInputStream for deflate. A BinaryReader object is created to read bytes from the inputStream. As the inputSource is read, the uncompressed data is written to the result Stream using a BinaryWriter. If the response was not compressed, the result Stream is set to the original, uncompressed Stream. In
Dr. Dobb’s Journal Windows/.NET Supplement, February 2005
http://www.ddj.com
(continued from page S12) any case, the result Stream is returned at the end of GetResponseStream. The CompressedWebResponse class uses the #ziplib library (available at http://www .icsharpcode.net/OpenSource/SharpZipLib/) to decompress responses. .NET 2.0 will include built-in libraries for gzip compression and decompression. Strictly speaking, you can enable compressed responses without deriving a class from the proxy class. You could override GetWebRequest and GetWebResponse in the original proxy class. But changes to the proxy class will be undone if you ever need to regenerate the proxy in the future. Compressing responses on the server and decompressing them on the client consumes additional CPU cycles on both server and client, but the savings in bandwidth is likely to justify the processing cost. At least at the moment, the Google web service does not return compressed responses, perhaps because the responses tend to be small.
Figure 1: SoapEx sample app.
SoapEx requests compressed responses, and is ready to decompress them if they are returned in the future. SoapExtensions A SoapExtension enables your program to wiretap web-service calls and even change the SOAP requests and responses. SoapExtensions are not tied to a particular web service. If your application calls multiple web services, they will all be manipulated by a SoapExtension. .NET’s SoapExtension mechanism lets custom code be called at various stages of the web-service call process. If you derive a class from SoapExtension, your class’s ProcessMessage method (Listing Three) will be called at the stages of the web-service call process listed in Table 1. If you intend to capture and/or change SOAP requests and responses, override the ChainStream method in Listing Three. The argument to ChainStream is a Stream containing the current SOAP request or response. In your overridden ChainStream, save the Stream argument and create, save, and return another Stream argument. For example, the sample application saves the argument as oldStream, and creates and returns a new Stream named “newStream.” After .NET has created a SOAP request by serializing the request proxy object, you can capture the SOAP request, and even modify it. Just take the original request, which is stored in newStream, change the XML, and store it in oldStream
Figure 2: Packetyzer. Stage
Description
BeforeSerialize
SOAP request has not yet been created from the request proxy object.
AfterSerialize
SOAP request has been created but has not yet been sent to the server.
BeforeDeserialize
SOAP response has been received, but has not yet been converted to a response proxy object.
AfterDeserialize
Data has been converted to a response proxy object.
Table 1: Stages of web-service calls. S14
(see ProcessRequest in Listing Four). When the client receives a SOAP response, the XML can be captured and changed before it’s deserialized into a response proxy object. To modify a response, take the original response stored in oldStream, change the XML, and store it in newStream (see ProcessResponse). You can see SoapEx capture and display SOAP requests and responses by pressing the Call Web Service button and looking in the Results text box. To see SoapEx modify requests and responses, check the Modify Soap Request and Response checkbox, and press the Call Web Service button. Then look at the Results text box. The request and response will be modified by “pretty-printing” them with indentation. I confess: SoapEx’s modifications to SOAP requests/responses are somewhat contrived. But if you have control over both the client and server, you can use SoapExtensions to modify requests and responses to implement proprietary compression, encryption, and so on. Configuring SoapExtensions A SoapExtension can be wired into an application using a configuration file. For Windows Forms applications, put the configuration settings in a file with the same name as the executable, with an extra .config on the end. For example, the SoapEx’s configuration file is named SoapEx.exe.config (Listing Five). For ASP.NET applications, store the same configuration information in a Web.config file. The element is where the SoapExtension is specified. The type attribute specifies the full name of the SoapEx class in namespace.class format, followed by the name of the assembly containing the SoapExtension. If there are multiple SoapExtensions, the priority attribute determines the order in which they are applied. SoapExtensions with higher priorities are applied before ones with lower priorities. Priority 0 is the highest, 1 is the next highest, and so on. SoapExtensions with a group attribute of 0 have the highest relative priority. Debugging Web Services Sometimes, the Visual Studio debugger is not sufficient to debug web services. It’s often important to see the exact bytes going over the wire during a web-service call. For example, when a web-service response isn’t well- formed XML, the response proxy object will be null, and you’ll want to inspect the actual response that the server returned. In these sorts of situations, I recommend Network Chemistry’s Packetyzer program (Figure 2), available at http://www.networkchemistry .com/products/packetyzer/. When you
Dr. Dobb’s Journal Windows/.NET Supplement, February 2005
http://www.ddj.com
launch Packetyzer 3.0.1, the Capture Options dialog lets you “Capture packets in promiscuous mode,” which I don’t recommend. If your machine has a promiscuous network card, Packetyzer can display traffic from your machine and other machines on the network. Unless you need to monitor other machines’ traffic, promiscuous mode just slows down the debugging process. Launch Packetyzer and then launch SoapEx. Click on the IP Addresses button, which will launch a dialog box (Figure 3). Select the line below the “Packetyzer Advanced Filter” and press Ctrl+C to copy it into the clipboard. Then go to Packetyzer and click the Display Filter drop- down combobox. Press Ctrl-V to paste the filter and press Apply to apply it. This filter specifies that Packetyzer will display HTTP network traffic between your machine and the web server that implements the web service. I recommend taking the time to learn Packetyzer’s filtering mechanism — filters can make Packetyzer much more convenient and efficient. Select Session/Start Capture in Packetyzer’s main menu. Then press SoapEx’s Call Web Service button. Give Packetyzer a few seconds to capture the traffic. Select Session/Stop capture. Then click on
http://www.ddj.com
Figure 3: IP addresses dialog. the rows in the list control in the top, right corner of the screen (Figure 2). As you click on the rows, you’ll see the entire conversation that SoapEx had with the web server. To see the entire conversation in one stream, right-click any row in the list control and select Follow TCP Flow. Then select the Decode tab and you’ll see the web-service request and response in a convenient textual format. Conclusion The next time you use Visual Studio’s Add Web Reference feature, consider creating
derived proxy classes to specify compressed responses. The bandwidth that you save may pay for the modest effort of creating a derived class that requests compressed responses. And consider using a SoapExtension to capture your application’s SOAP requests and responses for debugging purposes. Finally, download Packetyzer and install it. You’ll want to have it handy the next time you need to do any serious web-service debugging. DDJ (Listings begin on page S16.)
Dr. Dobb’s Journal Windows/.NET Supplement, February 2005
S15
Listing One protected override WebRequest GetWebRequest(Uri uri) // Update the request's HTTP headers to specify that // we can accept compressed (gzipped, deflated) responses { WebRequest request = base.GetWebRequest(uri); // If user checked the Compressed Response checkbox if (compressResponse) { request.Headers.Add("Accept-Encoding", "gzip, deflate"); } return request; } protected override WebResponse GetWebResponse(WebRequest request) // If we've requested compressed responses, return a WebResponse // derivative that's capable of uncompressing it. { WebResponse result; // If user checked the Compressed Response checkbox if (compressResponse) { result = new CompressedWebResponse((HttpWebRequest) request); } else // no compression requested, return stock WebResponse object { result = base.GetWebResponse(request); } // Keep track of content length to measure bandwidth savings. responseContentLength = result.ContentLength; return result; }
Listing Two public override Stream GetResponseStream() // Decompress the web service response and return it in a stream. { Stream result = null; // Clean up previously-used BinaryReader and BinaryWriter. if (reader != null) { reader.Close(); reader = null; } if (writer != null) { writer.Close(); writer = null; } try { // Get response. HttpWebResponse webResponse = (HttpWebResponse) request.GetResponse(); Stream inputStream = null; bool decompress = true; // Get an input stream based on the type of compression, if any. if (webResponse.ContentEncoding == "gzip") { inputStream = new GZipInputStream(webResponse.GetResponseStream()); } else if (webResponse.ContentEncoding == "deflate") { inputStream = new InflaterInputStream(webResponse.GetResponseStream()); } else { // Response wasn't compressed, return the original, uncompressed stream. result = webResponse.GetResponseStream(); decompress = false; } // If response was actually compressed, decompress it. if (decompress) { // Decompress the input stream. reader = new BinaryReader(inputStream); result = new MemoryStream(); writer = new BinaryWriter(result); int bytesRead; byte[] buffer = new byte[1024]; do { // Read from compressed buffer and store the decompressed // bytes in the buffer. bytesRead = reader.Read(buffer, 0, buffer.Length); // Write decompressed buffer to the result stream. writer.Write(buffer, 0, bytesRead); } while (bytesRead > 0); writer.Flush(); result.Position = 0; } } catch (Exception ex) { MessageBox.Show(ex.Message, "Exception", MessageBoxButtons.OK, MessageBoxIcon.Exclamation); } // Returned decompressed response. return result; }
Listing Three public override void ProcessMessage(SoapMessage message)
S16
// Hook into the web service process at various stages. { switch (message.Stage) { case SoapMessageStage.BeforeSerialize: break; case SoapMessageStage.AfterSerialize: // Capture and optionally modify SOAP request. ProcessRequest(message); break; case SoapMessageStage.BeforeDeserialize: // Capture and optionally modify SOAP response. ProcessResponse(message); break; case SoapMessageStage.AfterDeserialize: break; default: throw new Exception("invalid stage"); } } public override Stream ChainStream(Stream stream) // Save the stream representing the SOAP request or response { oldStream = stream; newStream = new MemoryStream(); return newStream; }
Listing Four private void ProcessRequest(SoapMessage message) // Capture and optionally modify the SOAP request. { newStream.Position = 0; using (MemoryStream memoryStream = new MemoryStream()) { CopyStream(newStream, memoryStream); // Capture original SOAP request. OriginalSoapRequest = GetStreamText(memoryStream); // If user has checked the Modify SOAP Request and Response checkbox... if (modify) { // "Pretty-print" SOAP request XML with indentation. ModifiedSoapRequest = ModifySOAP(memoryStream, oldStream); } else { CopyStream(memoryStream, oldStream); } } } private void ProcessResponse(SoapMessage message) // Capture and optionally modify the SOAP response. { using (MemoryStream memoryStream = new MemoryStream()) { CopyStream(oldStream, memoryStream); OriginalSoapResponse = GetStreamText(memoryStream); // If user has checked the Modify SOAP Request and Response checkbox if (modify) { ModifiedSoapResponse = ModifySOAP(memoryStream, newStream); } else { CopyStream(memoryStream, newStream); } } newStream.Position = 0; }
Listing Five
DDJ
More .NET on DDJ.com ASP.NET2theMax: Skin Your Pages ASP.NET’s themes let you alter the appearance of all controls in an app. Themes, or “skins,” can be applied at various levels—to an individual page, to all pages within a site, to an entire web server. Windows Security: Scripting Patch Deployment with WUA API Perform patch updates from the command line using VBScript and the Windows Update Agent API—no Internet Explorer required. Available online at http://www.ddj.com/topics/dotnet/
Dr. Dobb’s Journal Windows/.NET Supplement, February 2005
http://www.ddj.com
EMBEDDED SYSTEMS
Inside the uIP Stack Adding network support to a DSP-based embedded system DREW BARNETT AND ANTHONY J. MASSA
O
n one hand, adding networking support to embedded devices can be a daunting task, primarily because network stacks are so resource intensive. On the other hand, going to the trouble of adding networking support makes standardized access to devices possible because these protocols are portable across many platforms. TCP/IP, for instance, makes it possible for a wireless sensor node to communicate using the native protocol of most networking infrastructures, where each node in the sensor network can be accessed by a desktop PC, Tablet PC, PDA, or even cellphones. Networking support also improves the basic feature set each node is capable of supporting. For example, if an alarm condition occurs in the network, the device with the fault can generate e-mail and send it off to instantly notify the network
Drew is a wireless-communications/DSP systems engineer and president of Elintrix (http://www.elintrix.com/) where he is currently engaged in the development of networked radio and sensor products. He can be contacted at dbarnett@elintrix .com. Anthony is cofounder and chief engineer of software development at Elintrix and author of Embedded Software Development with eCos. He can be contacted at
[email protected]. http://www.ddj.com
operator about the problem. Another benefit is that web-based management can be easily incorporated into all devices with network support. Technicians can then connect directly to a sensor node for configuration and monitoring using a standard PDA equipped with a standard web browser. In effect, the sensor node’s web server and HTML pages are the new user interface for the device. There are a number of available network stacks that target deeply embedded systems where processor cycles and memory are limited. In this article, we take an in-depth look at one of these — the uIP network stack— and then port it to a DSPbased wireless sensor board. Figure 1, for instance, is a sensor network made up of nodes that incorporate a network stack and a web server to enable web-based management. Individual network nodes can then be controlled and monitored through HTML pages sent by the nodes’ web server, as in Figure 2. The File Transfer Protocol (FTP) can be used in a networked sensor node to update existing software for feature upgrades or bug fixes. Dynamic sensor node configuration information can also be sent to a particular sensor node. Networking & Wireless Sensor Nodes One of the keys in deciding which stack best fits your device is determining the resource requirements for the software. The amount of data each node in your system needs to transmit and receive during communication sequences should also dictate which solution is right for your design. For example, if your sensor node wakes up every hour to transmit a few bytes of data, a compact network stack implementation is ideal. On the other hand, systems that are constantly transmitting large packets of data might need a stack implementation that offers better packet buffer management. It is important when evaluating and selecting a lightweight network stack implementation that it allows communicaDr. Dobb’s Journal, February 2005
tion with standard, full-scale TCP/IP devices. An implementation that is specialized for a particular device and network might cause problems by limiting the ability to extend the nodes’ network capabilities in other generic networks. It is always possible to implement your own network stack, and this might be necessary depending on your particular device’s needs. However, with the assortment of
“The uIP stack was designed specifically for resource-constrained embedded devices” solutions available in the open-source community, leveraging existing technology lets you quickly move development ahead. The following software network stacks are open-source solutions for smaller sensor nodes that must take into account resource usage. This list is not comprehensive, but rather a starting point for further investigation: • lwIP (http://www.sics.se/~adam/lwip/). The “lightweight IP” stack is a simplified but full-scale TCP/IP implementation. lwIP is designed to be run in a multithreaded system with applications executing in concurrent threads; however, it can also be implemented on a system with no real-time operating system (RTOS). The protocols include Internet Control Message Protocol (ICMP), 61
Dynamic Host Configuration Protocol (DHCP), Address Resolution Protocol (ARP), and User Datagram Protocol (UDP). It provides support for multiple local network interfaces and is flexible enough to let it be easily used on a wide variety of devices and scaled to fit different resource requirements. • OpenTCP (http://www.opentcp.org/). Tailored to 8- and 16-bit microcontrollers, OpenTCP incorporates the ICMP, DHCP, BOOTP (Bootstrap Protocol), ARP, and UDP protocols. There are also several applications included with OpenTCP, such as a Trivial File Transfer Protocol (TFTP) server, a Post Office Protocol Version 3 (POP3) client to retrieve e-mail, Simple Mail Transfer Protocol (SMTP) support to send e-mail, and a Hypertext Transfer Protocol (HTTP) server for web-based access.
Figure 1: The wireless sensor network map gives an overview of the individual nodes that make up the entire sensor network.
Figure 2: Clicking on each networkenabled node lets users examine and control the parameters associated with an individual sensor node. Module ARP IP/ICMP/TCP HTTP Server Checksum Functions Packet Buffer TOTAL
• TinyTCP (http://www.unusualresearch .com/tinytcp/tinytcp.htm). TinyTCP is designed to be modular and only include the software required by the system. For example, different protocols can be compiled in/out based on configuration. It provides a BSD-compatible socket library and includes the ARP, ICMP, UDP, DHCP, BOOTP, and Internet Group Management Protocol (IGMP) protocols. • uC/IP (http://ucip.sourceforge.net/). Pronounced “mew-kip,” uC/IP is designed for microcontrollers and based on BSD. Intended to be used with the uC/OS RTOS. Protocol support includes ICMP and Point-to-Point Protocol (PPP). • uIP (http://www.sics.se/~adam/uip/). The “micro IP” stack is designed to incorporate only the absolute minimal set of components necessary for a full TCP/IP stack. There is support for only a single network interface. Application examples included with uIP are SMTP for sending e-mail, telnet server/client, an HTTP server and web client, and Domain Name System (DNS) resolution. In addition, some RTOSs include or have network stacks ported to them. One such example is eCos (http://ecos.sourceware .org/), an open-source RTOS that includes both the OpenBSD and FreeBSD network stacks as well as application layer support with an embedded web server (for more information, see Embedded Software Development with eCos, by Anthony Massa, Prentice Hall PTR, 2003). If a network stack is included or already exists in your device, several embedded web servers are available to incorporate web-based control. One such open-source solution is the GoAhead WebServer (http:// www.goahead.com/; see “Integrating the GoAhead WebServer & eCos,” by Anthony Massa, DDJ, November 2002). There are also resources that detail a ground-up approach to developing a small TCP/IP stack intended for embedded systems (TCP/IP Lean: Web Servers for Embedded Systems, by Jeremy Bentham, CMP Books, 2002). The uIP Network Stack The uIP (pronounced “micro IP”) stack was designed specifically for resourceconstrained embedded devices (http:// www.sics.se/~adam/uip/). Nevertheless, it is a full TCP/IP stack implementation,
Code Size
RAM Usage
1324 3304 994 636 0 6258
118 360 110 0 400 988
Table 1: uIP memory usage, in bytes, for a typical system. 62
Dr. Dobb’s Journal, February 2005
and its size and resource requirements make it ideal for applications such as wireless sensor nodes. Any processor that contains a reasonable amount of internal memory can support the entire network stack on board, eliminating the added expense of external RAM; see Figure 3. uIP utilizes a single global buffer, which has a configurable size based on the maximum packet length supported, for incoming and outgoing packets. Because of this, the application must process the incoming packet before the next packet arrives, or the application can copy the data into its own buffer for processing later. Incoming data is not stored into the packet buffer until after the application has processed the previous data. The standard socket interface is the BSD socket API. This interface relies on an underlying multitasking operating system to function properly. Due to this requirement and needed overhead, uIP does not support the socket interface. Instead uIP uses events. Therefore, data arriving on a connection (or an incoming connection request) causes an event to be sent to the application for processing. To avoid buffering outgoing data before it has been acknowledged, uIP requires the application to assist in packet retransmissions. When the uIP stack determines that a retransmission must occur, an event is sent to the application. The application reproduces the data previously sent. Sending data the first time and during a retransmission is the same from the application’s perspective; therefore, the same code can be used. The complexity for determining when the retransmission must take place is contained in the uIP stack itself, eliminating this from the application. To examine the resource requirements for the uIP stack implementation, the Atmel AVR processor architecture and the GNU compiler Version 3.3 were used to perform measurements. Table 1 gives resource usage for a uIP configuration with one listening TCP port, 10 TCP connection slots, 10 ARP table entries, a packet buffer of 400 bytes, and the simple HTTP server. uIP includes a Perl script for converting web pages into static memory arrays so they can be incorporated in the compiled code. This eliminates the need for any type of filesystem when using the web server. The total RAM usage is dependent on various configurable components within uIP. These components are easily set up in the uIP options include file. Things you can configure include the number of TCP connections supported to save RAM and the amount of memory allocated for the receive/transmit buffer, especially if packet sizes are typically smaller on the network. http://www.ddj.com
(continued from page 62) Certain code modules can be eliminated altogether if you’re using a serial network interface. Serial protocols are available, such as the Serial Line Interface Protocol (SLIP) or the more robust PPP protocol. Opting for a serial interface instead of Ethernet allows ARP support code and the ARP table to be removed. Not using UDP can also reduce the code size. These options are easily removed, via macros, in the uIP stack. The flow of incoming packets can be throttled by adjusting the maximum segment size of incoming data allowed by the sensor node. Other system optimizations that are ideal for wireless sensor networks are available, but require more in-depth configuration. Areas for further investigation include TCP/IP header compression techniques to reduce overhead of the various protocol headers. This is best for sensor systems comprised of nodes that typically transmit smaller data packets. For TCP, packet retransmission schemes are available that move the resending of lost packets closer to the destination of the data rather than relying on the source point. This can also reduce the length that an acknowledgement needs to travel through the system. Porting the uIP Stack to the TMS320C5509 DSP To further examine the uIP stack, we built a prototype wireless radio sensor board that was based on the TMS320C5509 digital-signal processor from Texas Instruments. For most ports of the uIP stack, the TCP/IP code doesn’t need to be modified. However, unique features of the TMS320C5509 DSP made it necessary for us to change uIP core files. Figure 4 illustrates the uIP directory structure. A new architecture-specific directory,
Figure 3: uIP network stack block diagram showing supported protocols and included applications. 64
with files copied from the UNIX directory, can be created for the new port. The radio sensor board does not have a typical Ethernet port used with network stacks; this is an add-on module that is currently not present on the main board. This system uses a software-based serial
“One of the major differences between the C5509 and most processors is the treatment of the char type” port for network connectivity. Some host configuration is necessary in order to set up the host PC to communicate using SLIP over the serial port. The uIP stack includes an example SLIP driver, which we modified to accommodate the radio sensor board hardware. The original SLIP driver polls the serial port for incoming data and stores each character, applying the SLIP data translation changes where necessary, as it arrives. The sensor board code uses an interrupt for serial port, allowing the ISR to store the incoming data. When a complete network packet is received, a flag is set to start the processing of the data. One of the major differences between the C5509 and most processors is the treatment of the char type. On the C5509, a char is 16 bits rather than 8 bits, as found on typical processors. This adds some challenges to porting the uIP code that is designed to accommodate an 8-bit char type. In the C5509 world, a char, short, and int are all at 16 bits and data space memory is addressed 16 bits at a time. uip-0.9 - main directory for uIP stack version 0.9 apps - application modules httpd - web server module fs - web server file system files resolv - DNS resolution module smtp - SMTP e-mail module telnet - Telnet client module telnetd - Telnet server module webclient - HTTP client module doc - uIP documentation uip - main uIP stack source files unix - architecture specific files
Figure 4: uIP directory structure. Dr. Dobb’s Journal, February 2005
As data comes in to the serial driver, it is stored into a buffer. The data in the buffer in a typical system (with an 8-bit char type) would look similar to Figure 5(a), while Figure 5(b) shows the buffer in the C5509 system. When a structure is overlaid on this data buffer for parsing, the members of the structure do not contain the correct data values. Because of this difference, the main structure for the TCP/IP header, uip_tcpip_hdr (located in the uip.h file), must be modified. Furthermore, the code that uses this structure must also be modified. Listing One(a) is the original uip_ tcpip_hdr header structure. Structure members that are of type u8_t do not need to be modified, because the only difference on the C5509 platform is that these types will occupy 16 bits of memory each instead of the typical 8 bits. However, the u16_t type members do need to be changed. Listing One(b) is the modified structure to accommodate the C5509based prototype board. Next, these structure changes are incorporated into the uIP networking code. An example of code using the new structure is in uip.c, where the TCP source and destination port numbers are swapped to format the header of an outgoing packet. Listing Two(a) is the original code, and Listing Two(b) the modified code. Other variables that pose the same issue between the C5509 platform and a standard processor are also modified in a similar way. In this example, BUF is a uip_tcpip_hdr pointer type and points to the start of the incoming network packet. Because we changed the original srcport from a 16-bit unsigned short into an array of two 16-bit unsigned short elements, we need to store the srcport array elements properly. This example also shows the destport code modification. These modifications are done throughout the uIP source code. Another file that needs attention is uip_arch.c, which is located in the architecture-specific directory. This file contains the TCP and IP checksum routines. After porting the uIP networking code to the C5509, the main processing loop is (a) Address
Data
0x0000 0x0008
45 00 00 30 7D 64 40 00 80 06 d3 fc c0 a8 14 0a
(b) 0x0000 0x0008
0045 0000 0000 0030 007D 0064 0040 0000 0080 0006 00d3 00fc 00c0 00a8 0014 000a
Figure 5: (a) Data buffer in 8-bit char system; (b) data buffer in 16-bit char C5509 system. http://www.ddj.com
implemented. Listing Three shows a slightly modified version of the processing loop included in the uIP source code. The initialization is performed for all of the necessary drivers and uIP modules including the serial port (serial_init), timer (timer_init), and SLIP drivers (slipdev_init), as well as the uIP stack (uip_init), and the HTTP web server (httpd_init). The slipdev_poll function checks to see if the serial driver has received a packet. If a packet has arrived, the uip_len is set to the packet length. The data is then processed by calling uip_input, which is a macro that calls the main uIP processing routine uip_ process for handling the received packet. If a data packet has not been received, a timer count is checked to see if it is time to call the periodic function (aptly named uip_ periodic). The timer counter, ulTimer0Count, is incremented every millisecond in the timer interrupt service routine. The periodic routine also calls the main processing routine, uip_ process, to maintain any open connections in the system.
There are a few things to keep in mind during the porting process:
“Embedding a networking stack is no longer an arduous task” • If a timer is not available, a variable can be used to count when a second has occurred and, therefore, when to call the periodic function. However, the code will have to be calibrated based on processor clock, and any modifications to the code affect this timing. • It might be helpful to enable debugging in the uIP stack to get output from the code. Setting the macro UIP_LOGGING
to 1 in the file uipopt.h located in the architecture-specific directory can do this. • Also included with the web-server code is a Perl script that generates C code from HTML files. This can be found under the httpd\fs directory. • Various uIP stack parameters, such as the number of simultaneous network connections and packet buffer size, can be customized in the file uipopt.h. Conclusion Embedding a networking stack is no longer an arduous task that requires an enormous amount of resources. As we’ve shown, there are a number of solutions that use the TCP/IP protocol suite for communication. Tailoring one of the network stack solutions to the specific characteristics of the sensor node device ensures the system will operate at its most optimum level and utilize system resources best. We would like to thank Adam Dunkels, developer of the uIP stack, for his support during the porting process.
Listing One (a)
tmp16 = BUF->srcport; BUF->srcport = BUF->destport; BUF->destport = tmp16;
typedef struct { /* IP header. */ u8_t vhl, tos, len[2], ipid[2], ipoffset[2], ttl, proto; u16_t ipchksum; u16_t srcipaddr[2], destipaddr[2]; /* TCP header. */ u16_t srcport, destport; u8_t seqno[4], ackno[4], tcpoffset, flags, wnd[2]; u16_t tcpchksum; u8_t urgp[2]; u8_t optdata[4]; } uip_tcpip_hdr;
(b)
(b) typedef struct { /* IP header. */ u8_t vhl, tos, len[2], ipid[2], ipoffset[2], ttl, proto; u16_t ipchksum[2]; u16_t srcipaddr[4], destipaddr[4]; /* TCP header. */ u16_t srcport[2], destport[2]; u8_t seqno[4], ackno[4], tcpoffset, flags, wnd[2]; u16_t tcpchksum[2]; u8_t urgp[2]; u8_t optdata[4]; } uip_tcpip_hdr;
Listing Two (a)
/* Swap port numbers. */ tmp16 = ((BUF->srcport[1] srcport[0]); BUF->srcport[0] = BUF->destport[0]; BUF->srcport[1] = BUF->destport[1]; BUF->destport[0] = (tmp16 & 0x00FF); BUF->destport[1] = ((tmp16 >> 8) & 0x00FF);
Listing Three // Initialize and start the uIP timer to fire every 1 ms. timer_init(); // Initialize the serial port for use with the uIP stack. serial_init(); // uIP networking stack related initialization. slipdev_init(); uip_init(); httpd_init(); // Main processing loop. while (1) { // If we get a length greater than zero, we know a packet is ready for // processing. Otherwise, we call the periodic function once per second. uip_len = slipdev_poll(); if (uip_len == 0) { // Call the periodic function once per second and loop through all // connections. if (ulTimer0Count >= 1000) { ulTimer0Count = 0; for (uiConn = 0; uiConn < UIP_CONNS; uiConn++) { uip_periodic( uiConn ); // If the periodic function resulted in data that needs to // to be sent out, length is set to a value greater than zero. if (uip_len > 0) slipdev_send(); } } } else { // Process the incoming data. uip_input(); // If the input function resulted in data that needs to // to be sent out, the length is set to a value greater than zero. if (uip_len > 0) slipdev_send(); } } // End of main loop.
DDJ
/* Swap port numbers. */
http://www.ddj.com
DDJ
Dr. Dobb’s Journal, February 2005
65
PROGRAMMING PARADIGMS
Loners and Guerrillas Michael Swaine
erhaps the title this month should be “Programmer Paradigms.” Several things I’ve read recently have got me thinking about different kinds of programmers: the lone coder versus the connector, the troubleshooter versus the architect, the guerrilla versus the soldier. Few of us, perhaps, would fit unambiguously into one category or another, but we can probably all see what label fits us best. The challenging question, though, is: How well does the kind of programmer you are fit into the time and situation in which you currently find yourself? And if the fit is not ideal, can you change yourself? Can you change your environment? (I’m just asking; I don’t have the answers.)
P
variations on bogus news reports and autopsies of the Lone Coder’s death (death by overdose of Jolt, and so on); arguments that he didn’t die, he just moved overseas; and rejection of the whole Lone Coder theory in favor of a Second Coder on the Grassy Knoll. To a comedian, everything looks like a straight line. I still think that the fate of the Lone Coder is an interesting question, but if you want to send me your thoughts on it, I just wanted you to know that I’ve read all the obvious jokes. We all are Lone Coders, though, when it comes to programming for that very special client — ourselves. And we all keep looking for better tools for our homemadesoftware toolkits.
The Lone Coder A Slashdotter asked a disturbing question last November. “Is the lone coder dead?” CyNRG asked. “The little guy. The oneperson software company. Can it still exist today?” CyNRG’s corn wasn’t economics or code complexity or even Microsoft’s dominance of the software market, but software patents. Will the need to ensure that no idea in your code is covered by some obscene obvious-tweak patent soon mean that the Lone Coder can no longer afford to sell software? “The amount of money required to perform the due diligence research seems like it would be greater than the amount of money needed to develop the software, or even the total revenues that the software could ever generate,” CyNRG moaned. “Please someone tell me I’m wrong!” CyNRG can’t have been satisfied with the discussion that this plea generated on Slashdot. Mostly it was
Mathematica I really want Mathematica to be a tool in my toolkit. Whenever a new version comes out, I hack away for a weekend trying to get a handle on it, but the learning curve rises ahead of me and I only advance a small step, while I’ve fallen back one and a half since my last effort. The learning curve has so far kept me from exploiting the serious power that’s in this product. The new string-handling capabilities in the latest release could change that. Wolfram Research suggests using the new features for web site manipulations, data mining, or DNA processing. While I’m not heavily into DNA processing, manipulating text is something I do a lot of — whether for something as mundane as preprocessing text for web publishing or as frivolous as trying to write a program to generate crossword puzzles for my own amusement. Mathematica’s analytical power combined with the new string-handling capabilities immediately suggested some
Michael is editor-at-large for DDJ. He can be contacted at
[email protected]. 66
Dr. Dobb’s Journal, February 2005
uses. Such as, “how many unique words are there in a sample of my DDJ columns?” “What are the most common words?” “How about a bar graph showing the frequencies of words in my columns?” This time I found myself getting farther up that curve. Mathematica supports different programming paradigms and modes of operating, but it is above all designed to make it easy to play around with variables and calculations in an interactive way. You work in a workbook, which remembers values of previous calculations so that you can build up very elaborate but flexible what-if structures. I wrote the following code to read all my 2004 “Programming Paradigms” columns and combine their texts into one long string. Nothing startling, but note the specification of multiple files with Greplike notation in the second line. This makes myColumns be a list with entries like “PP0401.txt,” “PP0402 revised.rtf,” and so on: SetDirectory["../Documents/Paradigms/"] myColumns = FileNames["PP04*"] For[rawText = ""; i = 1, i 2 Length[myColumns], i++, rawText = StringJoin[rawText, Import[myColumns[[i]], "Text"]]]
The 32K-word string rawtext needed some massaging before I could do statistics on it. I appreciated the interactive nature of Mathematica when it came to identifying the punctuation and other characters that I needed to strip out to have just words left. It took several passes with different word lists to come up with a list of 25 characters to ignore. http://www.ddj.com
Mathematica is big on treating actions as data; the first line below (which I’ve abbreviated) defines a list of replacement operations (replace “.” by “ ”), which is then used by the built- in function StringReplace. Other built-in functions turn letters to lowercase and return a list of the individual words. (Lists in Mathematica are denoted by braces: {}.) ignoreChars = {"." -> " ", "?" -> " "...} wordList = StringSplit[ToLowerCase [StringReplace[rawText, ignoreChars]]]
Mathematica’s programming paradigm of choice is functional programming, and like that classic functional language Lisp, Mathematica loves lists. And because Mathematica was created to do symbolic math, it knows about operations like Union, which treats lists as sets. When applied to a single list of words, Union eliminates all duplicate words. uniqueWords = Union[wordList]
Now I had my vocabulary list for my 2004 columns and could start doing statistics on it.
little more time plumbing the language’s depths for other ways to do the calculations. That wasn’t necessary in this case, because Mathematica is well tuned under the hood. Operations that look wildly inefficient generally run faster than you’d guess. On my 1.8-GHz iMac G5, these wasteful string manipulations ran too fast to bother optimizing. And I didn’t even exploit all the new string manipulation features.
“Spheres has come out with a new programming language that is event based”
Length[uniqueWords]
That one-liner told me that I used 4766 unique words in the 12 columns of 2004. Most of the other statistical manipulations that I’d like to do, like computing the most common words or looking at word sequences, are also one-liners. But graphing the results wasn’t much more difficult, after I loaded the Graphics module. For[favoriteWords = {}; i = 1, i 2 Length[uniqueWords], i++, AppendTo[favoriteWords, Length [Cases[wordList, uniqueWords[[i]]]]] ] Horizontal, BarLabels -> uniqueWords]
A glance at that graph shouted “Too Much Information,” so I went back a step and trimmed down the favoriteWords list to just the most common ones, testing different parameters to get a readable graph. And so on. Before the weekend was over, I had built a statistical “Programming Paradigms” column generator. I was relieved to see that my Mathematica skills aren’t good enough that I can replace myself as a columnist. There’s nothing here that couldn’t be done in any other language, and in some languages it could doubtless be done just as succinctly. But Mathematica is powerfully interactive and operates at a symbolic level that can be exploited to do surprising things. And its own vocabulary is very deep: If the quick-and-dirty code I wrote proved inefficient, I could have spent a http://www.ddj.com
EPL This one won’t go in my own toolkit, but it might find a place in yours. Spheres, a spin-off from Caltech, has come out with a new programming language that is event based. That doesn’t sound too revolutionary, but the kinds of events they have in mind are financial transactions and attempts to break into a computer system. A programming language designed from the ground up to deal with such events is certainly worth a look. An early version, called EPL (Event Programming Language) Server, is available and can be used royalty free. EPL Server isn’t intended to replace other programming languages, but to interface with them and to simplify a small set of tasks. The model is designed to: • Monitor information across multiple applications. • Filter irrelevant data. • Detect actionable changes in patterns and trends. • Notify applications when events occur. What’s new about EPL is not the technology, which is familiar. Other companies and industry groups are working on standards and technologies for business event processing. As reported on Cnet’s Builder.com.com, KnowNow has developed similar software to send data to many destinations based on an event. EfDr. Dobb’s Journal, February 2005
forts are afoot to create an industry-wide language for automating business processes, called Business Process Execution Language (BPEL), for which languages or rule engines to handle events will presumably be required. The fact that we will probably soon have billions of RFID tags broadcasting their meagre bits of information everywhere we go suggests that we need a model of programming that deals opportunistically with the events that impinge on it. And so on. EPL Server is an interesting tool that may become an important tool for swimming in that sea of digital events. But what I find interesting about EPL Server, in the context of programmer paradigms, is the philosophical model. The EPL programmer is less an architect, more a troubleshooter. The language encourages a worldview that accepts that things happen on their own schedule and need to be dealt with. It is a philosophy of dealing with what is thrown at you rather than trying to engineer top-down control. EPL Server is certainly not a Lone Coder’s programming language. But there is an aspect of it that at least feels incompatible with the top-down control philosophy. Them Here is a different psychological profile of a programmer: One who possesses social skills, enjoys working in groups, and is a born networker. Such a programmer would hardly mourn the death of the Lone Coder. This is the kind of programmer that I like to call “Them.” Occasionally, I have to pretend — to myself, especially — to be one of Them. I found a book that helps. Fearless Change, by design pattern experts Mary Lynn Manns and Linda Rising (Addison-Wesley 2005; ISBN 0201741571), is not a book that loners would naturally gravitate toward. The authors unabashedly admit to being connectors, and the whole thrust of their book (and of the patterns they present) is connecting. It’s all about design patterns for effecting change in groups. To a comedian, everything looks like a straight line, and if all you have is a hammer, every problem looks like a nail. So naturally, these born networkers argue for solving the problem of introducing change into groups by their preferred method of working to recruit others to your cause, rather than by the time-honored technique of core- dumping every detail of your brainstorm onto everyone within three feet of you and stomping out of the room in a huff when the clueless losers don’t immediately get it. To each his or her own? Yes, but sometimes their own works better than our own; and those least like us have the most to teach us, if we are open to learning 67
from them. As a natural loner, I find that this book speaks only to that underdeveloped social lobe of my brain, but the question is: How well does it do that? Pretty well, I think, on balance, but for those of us whose need is great, there is a design pattern to help. The authors reference it on page 253: Teach yourself to play a role so that observers believe you are extroverted, bold, and outgoing. Teach yourself to recognize the situations in which this role is appropriate and to gather your resources and play the role. http://csis.pace.edu/~bergin/ patterns/introvertExtrovert.html.
It’s just one of the places in the book where a subversive, Machiavellian tone comes through. I like that. Book-Reading Patterns I don’t know why no one has yet come up with a set of design patterns for reading books. It’s such a common activity, and so often poorly done. Think how much time you would have saved by now if you’d had a few good book-reading patterns like the following: • Ignore Pages Numbered in Roman Numerals. That’s a good pattern to follow with this book. If you already know what design patterns are, skip the Preface. Unless you’re in it, skip the Acknowledgments. And especially skip the Foreword: It’s all buzzphrases in bad grammar. There; I’ve saved you xxii pages of reading. • The Typical Nonfiction Book Would Make a Good Article. In reading the patterns that make up half of this book, just read the boldface sections. That may be all you need, but if you need more, read just the italicized sections. If you need still more, go ahead and read the full text and don’t be embarrassed: The design pattern Just Enough may be one that you still haven’t wrapped your mind around. • Dead Trees, Dead Information. In the age of the Internet, any information you glean from a book is at least a year out of date. This does not necessarily make it useless, but it does constrain the kinds of information that it makes sense to try to get from this dead tree medium. Design patterns are a good example of the kind of information that books actually do convey well. Design patterns benefit from having some whiskers. It takes time to know that a candidate really merits being elected to patternhood. Information in books is also information that you may have come across before. But familiarity does not necessarily make information worthless. As the authors point out, if we never needed to be 68
reminded of things we have already learned, we would never make the same mistake twice. Familiar information, presented in a systematic way, can be very useful if that system works for you. Is that the case with this book? You’ll have to decide. The book consists of a collection of patterns, each described in
“Individuals are more powerful than institutions” about three pages, plus 12 chapters discussing a philosophy of organizational change and describing how to use the patterns, plus four case studies, including Sun’s experience in using patterns in introducing J2EE applications. Here are some of the themes that the authors emphasize in the chapters: • Three elements must be considered in effecting change in an organization: the change agent, the culture, and the people. • A pattern is a way of capturing expertise. • Initiating change, maintaining it, overcoming resistance, and converting the masses are all about politics. And here are some of the patterns: • Ask for Help. (This is a tough one for some of us.) • Do Food. (Making food available at a meeting changes the nature of the event.) • External Validation. (I advised you to skip the Foreword, but I wouldn’t advise an author not to include a Foreword.) • Just Enough. (See above.) • Just Say Thanks. (See below.) Guerrilla Action In my humble opinion, the great insight of the early 21st Century will be that individuals are more powerful than institutions; that a dedicated indigenous guerrilla movement can always wear down an invading army; that you can’t defeat terrorism with bombs; that ad hoc, self-organizing task groups work and management-composed committees don’t. The authors of this book, unlike this columnist, avoid such political pronouncements, but they clearly understand Dr. Dobb’s Journal, February 2005
the power of guerrilla movements. Their book shows little respect for the structures of power, except as obstacles to get around or weaknesses to exploit. As I said, Machiavellian. “How do I sell my executive team on doing this stuff?” they ask, and, quoting consultant Jim Highsmith, they answer, “Don’t. Just do it. They don’t know what you’re doing anyway.” That’s also a recipe for resisting change: The authors describe the powerful passive resistance that Carly Fiorina met when she tried to impose top-down changes at HP. The book freely acknowledges that people are not rational, that we use facts to justify emotionally-arrived-at decisions. It then presents techniques for getting things done anyway, despite this fundamental irrationality, by the use of proven patterns. Whether the things you get done are rational or not is another question. Okay, that’s it for this month. Thanks for reading. DDJ
Dr. Ecco Solution Solution to “The Luck of the 4,” DDJ, January 2005. Here are Tyler’s expressions for 11– 40. Twenty of these numbers required only three 4s. I’ll report your solutions to the open problems in a future column. 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
= 44/4 = 4*sqrt(4/.4R) = (4/.4R)+ 4 = 4/.4+ 4 = (4/(0.4*sqrt(.4R)))=4/(0.4*(2/3))=(3/2)*10 = 4*4 = 4*4+ 4/4 = 4*4+sqrt(4) = 4!– sqrt(4)/0.4 = 4*4+ 4 = 4!– sqrt(4/.4R) = 4!– sqrt(4) = 4!– 4/4 = 4! = 4!+ 4/4 = 4!+sqrt(4) = 4!+sqrt(4/.4R) = 4!+ 4 = 4!+sqrt(4)/0.4 = 4!+ 4+sqrt(4) = 4+ 4!+sqrt(4/.4R) = 4!+ 4+ 4 = 4!+ 4/.4R = 4!+ 4/.4 = 4!+sqrt(4)+ 4/.4R = 4!+sqrt(4)+ 4/.4 = 4!+ 4+ 4/.4R = 4!+ 4+ 4/.4 = 44– sqrt(4)/0.4 = 4*(4/.4)
http://www.ddj.com
EMBEDDED SPACE
Security Measures Ed Nisley
T
he National Cyber Security Alliance recently released the results of a study showing how difficult it is to achieve even minimal security on consumer PCs. The disparity between users’ expectations and reality should be particularly scary for folks working on embedded consumer products. As of last October, about 3/4 of the surveyed users felt reasonably safe from the usual assortment of online threats, despite the fact that 2/3 have outdated antivirus software and 2/3 (presumably a different subset) lack firewall protection. Essentially none have any protection against adware or spyware, and in fact, 9/10 haven’t a clue what that means. The result is predictable: 2/3 have experienced at least one virus infection, 1/5 have a current infection, and 4/5 have adware or spyware infestations. The NCSA actually measured those numbers by sending technicians to examine the PCs. The study did not report on worms or backdoor programs, perhaps lumping those with viruses. Many of the talks at last year’s Embedded Systems Conference dealt with security techniques and practices, but the ground truth indicates the battle has little to do with technology. Let’s compare expectations and requirements with the actual results. Boot, But Verify Transactions, pretty much by definition, have at least two sides. In the PC realm, we’ve been mostly concerned with verifying that the server on the far end is what it claims to be, taking for granted that the PC end is honest. That level of trust is certainly unjustified now, and for distributed embedded systems, it verges on wishful thinking. Ed’s an EE, PE, and author in Poughkeepsie, NY. Contact him at ed.nisley@ieee .org with “Dr Dobbs” in the subject to avoid spam filters. http://www.ddj.com
Several ESC talks described the requirements for a Trusted Computing Platform, one that is known to contain only valid hardware and software. During the power-on sequence, the hardware must verify that it has a correct, trusted configuration, using only known-good software that is either internal to the CPU or protected by a cryptographic hash stored inside the CPU. The verification extends well beyond the familiar BIOS sanity checks that most people disable to reduce boot time. The CPU must ensure that no potentially malicious devices are present, perhaps by preventing further execution if the system contains hardware that’s not on a list of preapproved devices. After the hardware checks itself, it must verify that the main system software is correct. Because that code is generally too large for on-chip storage, the hardware uses more crypto hashes. The entire software image or just key components can be hashed, with the obvious space-versustime tradeoffs. Once you have a system running trusted software atop trusted hardware, you can be reasonably sure that the entire configuration will behave consistently, if not necessarily correctly. From that point onward, however, it must ensure that only trusted hardware is connected and only trusted software is loaded and executed, because a single exposure can lead to a successful exploit. Based on a quick thumb check of the Trusted Computing Group’s numerous papers and documents, I’d say the TCG’s initial focus is on large organizations, both governmental and industrial, which can and must maintain tight control over their IT facilities. The cost of virus and worm episodes has become so great that any alternative seems preferable to the status quo. Unfortunately, high security presents a considerable challenge for IT departments that normally deal with commodity hardware. It’s feasible, if not easy, to keep Dr. Dobb’s Journal, February 2005
thousands of systems running with fungible components. It’s essentially impossible to maintain thousands of unique configurations with mutually incompatible components. The Trusted Platform Module (TPM) specs describe methods to implement nearly any security model you’d like, but getting it right requires considerable upfront planning and rigorous follow through. That level of attention to detail may be feasible for large institutions, but the rest of us simply can’t cope. The NCSA numbers reveal how difficult the challenge will be. Only 6 percent of users thought their PCs had a virus at the moment, 44 percent claimed to have clean PCs, and half simply didn’t know. The NCSA technicians actually found 25 percent of dial-up and 15 percent of broadband PCs had at least one virus. The average infection rate was 2.4 viruses per machine and the maximum was 213 (yes, on a single PC!). Enforcing a security model on unwilling participants is a recipe for disaster. Trouble in Paradise Let’s assume that all hardware must have a known provenance before it can be used by a trusted system. This may be feasible for embedded systems with tight access controls, but is probably impossible in the general case. Suppose, for example, that the system uses a generic PC-style keyboard with a PS/2 interface. Each keyboard could include a unique identification number that would be accepted only by a specific CPU. Alternatively, all approved keyboards could include a common identifier acceptable to any system. The ID must be wrapped in crypto to prevent spoofing or unauthorized duplication. But that’s not enough. To maintain security, your keyboards must be tamper evident, if not tamper proof, to prevent the introduction of unauthorized hardware. Consider what would happen if someone 69
added a hardware recorder either inside the keyboard case or between the keyboard and the system: They would capture all manual data going into the system. There is simply no electrical way to detect a hardware keystroke recorder between the keyboard and the PC, which means once it’s inserted, it won’t be found unless someone specifically looks for it. Commercial recorders monitor the data stream on the PS/2 wires for the keystroke sequence that activates them, but that’s entirely optional. Any decent hardware engineer could cook up a recorder that sniffs data from the wires, depends on external hardware for a data dump, and fits neatly on your thumbnail. Here’s an example of how much damage a simple logger can cause: In 2001
and 2002, Juju Jiang installed a software keystroke logger on PCs in various Kinko’s copy centers, then retrieved the logs using the PC in his apartment. Several Kinko’s customers had accessed their home PCs using a commercial remote-access program and Jiang’s logger snagged their user IDs and passwords. He later used that information to access their PCs, open new accounts using their personal information, and transfer money from their existing accounts. Jiang was only caught because one of the victims noticed his PC creating an account on its own, acting “as if by remote control” (which was, in fact, the case). There was enough evidence to trace the intrusion to Jiang, indict, prosecute, and convict him.
The same reasoning applies to any peripheral: If it’s not mechanically secured and physically suborned, you must assume its data stream can be monitored, either instantly or stored for later recovery. Keyboards are the obvious weak spot, but USB, serial, and parallel ports provide nice access points. While a hardware exploit requires two intrusions, one to plant the device and a second to recover it, the actual data collection doesn’t depend on easily discovered software hacks. The specs for a Trusted Platform Module require tamper-resistant or tamperevident packaging, but do not define the physical security requirements of the entire system. For widely distributed systems, whether PCs or embedded devices, it’s safe to assume they have no physical security at all and that the hardware can be compromised at will. That makes internal security measures moot, as they can be bypassed with enough time and patience. The Jiang case and the NCSA study both show that most people simply don’t notice unexpected behavior, never mind surreptitious information leakage. Only half of the users said that their machines had adware or spyware programs, while NCSA technicians found 9/10 of dial-up and 3/4 of broadband PCs were infected. The average infected machine had 93 “adware/spyware components,” whatever those are, and the worst-hit machine had 1059 (!) components. Although the price of freedom may be eternal vigilance, it’s obvious that we cannot depend on ordinary folks for sufficient attention to the details of secure computing. Business Opportunities My friend Aitch recently described an embedded-system business opportunity based on the convergence of two trends. While it’s not exactly legal, there’s excellent upside potential. Send me a suitcase of small bills if it works for you. It seems there’s a growing underground of automotive hotrodders who trick out their cars with all manner of performanceenhancing gadgets — turbochargers, superchargers, water injectors, you name it. They also modify the engine control computer firmware to further improve performance. As you might expect, these modified engines tend to poot out somewhat more unwanted emissions than their factory-stock brethren. Many states or urban areas now include pollution measurements as part of their annual automobile inspections. Those measurements require complex dynamometers, expensive buildings, and trained technicians, but, now that all vehicles have on-board engine controllers, why not just take advantage of the information available through the standard OBD (On-Board
70 WIBUbDDJalt.indd 1
Dr. Dobb’s Journal, February 2005 10/7/04 5:31:58 PM
http://www.ddj.com
Diagnostic) connectors in every car? If the engine controller reports no faults or unusual situations, the car passes the inspection and taxpayers save the cost of all that infrastructure. What’s not to like? Here’s the deal. Stick a cleverly coded microcontroller on the back of an OBD connector and sell it to the hotrodders, who tuck the official OBD connector up inside the dashboard, put the gimmicked OBD connector in its place, and drive off to their car inspection. The microcontroller responds to the inspection station’s inquiries with “Nothing to see here. Move along, move along.” Aitch suggests that $100 retail would move OBD spoofers off the loading dock at warp speed. Many ESC speakers observed that security must be included in the overall system design and that it cannot be grafted on after the first exploit goes public. The OBD system started as a simple diagnostic tap but is becoming part of a legislated system with legal force behind it. If you think Windows has security flaws, think about that situation for a moment. Pop quiz: Estimate the number of test instruments that must be replaced in order to accommodate trusted engine computers and inspection stations. Extra credit: Estimate a lower bound for the half-life of the existing engine population. Back at home, it’s widely agreed that Microsoft sells XBox gaming consoles at a loss to gain market share. Because MS also derives royalties from each unit of game software to offset the hardware loss, the XBox includes rather stringent controls on just what software it will accept, to the extent that it spits out unapproved CDs or DVDs. Consider it as a first approximation to a trusted computing platform. Days after the XBox first appeared on the market, Bunnie Huang figured out how to bypass the boot-code protection hardware. His methodology, presented at the 2002 Cryptographic Hardware and Embedded Systems conference, should be required reading for anybody trying to distribute trusted hardware in a consumer device. Summary: It won’t work. During the ensuing arms race, someone discovered a security hole in the “save game” feature of several XBox programs: You could save a game to a memory card, replace the file with something completely different, and reload it. The contents of the saved file, perhaps a Linux boot image, then runs without catching the attention of any of the security gadgetry. Of course, that hole isn’t present in current XBoxes. Isn’t it interesting that Microsoft patches economic leaks immediately, while leaving some security holes to languish for years? http://www.ddj.com
Hash Happens Most security measures depend on cryptographic hashes to ensure the integrity of large messages (or files or documents or whatever). A good hash algorithm has two key attributes: The probability that two messages will produce the same hash value (“collide”) is low and the difficulty of finding such a colliding message is high. Although their achievement didn’t achieve mainstream headline status, a team
“Enforcing a security model on unwilling participants is a recipe for disaster” of researchers recently published an algorithm that, given a message and its corresponding MD5 hash, produces a modified message with the same hash value. The fact that two messages can have the same MD5 hash value is not surprising. The trouble arises from being able to create a message with the same hash as the first. Moore’s Law tells us that once something can be done, it will be done much faster in only a few years. While this is not yet a catastrophic problem, it does show how you can get everything right and still have a security failure. If you’ve ever used an MD5 hash to verify a downloaded file, there’s now the possibility (admittedly, a small one) that the file you get may not be the file you expect, even though the hash is bit-forbit correct. The Trusted Computing Group chose SHA1 as the TPM hashing algorithm long before news of the MD5 compromise appeared. There are speculations about cracking SHA1, but, for the present, it seems secure. Pop quiz: Assume you had sold a few hundred thousand gizmos that validate new firmware downloads using MD5 hashes and signatures. How would you convert the units to use SHA1, given that attackers can spoof your files? Essay: Describe how you’ll get your users to care. Reentry Checklist Two things are quite clear: Consumers can’t be trusted with security decisions and hardDr. Dobb’s Journal, February 2005
ware cannot be secured from customers. The former implies that any security measures must not require user intervention; the latter implies that you cannot depend on trusted hardware in the field. Therefore, your overall system must be resilient enough to survive the failure of any defense and any unit. The XBox exploit shows how even a fairly complex security system can be penetrated with surprising ease. If all the units are identical, then one exploit is all it takes. Scary thought, eh? The NCSA study is at http://www .staysafeonline.info/news/safety_study_v04 .pdf with the summarizing press release at http://www.staysafeonline.info/news/ NCSA-AOLIn-HomeStudyRelease.pdf. The Trusted Computing Group is at https://www.trustedcomputinggroup.org/ home. The architecture document at https://www.trustedcomputinggroup.org/ downloads/TCG_1_0_Architecture_Overview .pdf mentions that keyboards and mice must include validation credentials. Details on a typical keystroke logger are at http://www.4hiddenspycameras .com/memkeystrok.html. The documents for the Juju Jiang case are at http://www .cybercrime.gov/jiangIndict.htm and http:// www.cybercrime.gov/jiangPlea.htm. It seems Kinko’s now restores the drive images each week, which is a step in the right direction. A quick overview of OBD-III can be found at http://lobby.la.psu.edu/_107th/ 093_OBD_Service_Info/Organizational_ Statements/SEMA/SEMA_OBD_frequent_ questions.htm. The usual searches will produce hotrodding information. Andrew “Bunnie” Huang performed the initial Xbox hacks and reported his findings at http://www.xenatera.com/bunnie/ proj/anatak/xboxmod.html. Pay particular attention to the trusted-computing part of his presentation at http://ece.gmu.edu/ crypto/ches02/talks_files/Huang.pdf. Microsoft countered the saved-game bug in short order with an automatic update, as reported at http://www.theinquirer .net/?article=11548. More on MD5 and SHA1 hashing is at http://www.secure-hash-algorithm-md5sha-1.co.uk/. The original paper describing the MD5 break is at http://eprint.iacr.org/ 2004/199.pdf. “Trust, but verify” was President Reagan’s take on nuclear disarmament. If you don’t recognize “Nothing to see here,” get a Star Wars DVD set and discover how much movie special effects have improved in recent years. “And Now for Something Completely Different” was a 1971 Monty Python video compilation. DDJ 71
CHAOS MANOR
The Google Revolution Jerry Pournelle
T
he first time I ever heard of Google was when Darnell Gadberry came over in some excitement to alert me to a new search engine. He called it “google” but, having read Mathematics and the Imagination, by Edward Kasner et al., I thought he said “googol,” which is the number 1010^10. It took me some time to get my head straight. Since that time, Google has created a number of revolutions. As a company, it has gone public in a way that enriched early stock buyers rather than investment bankers, while its search engines have changed scholarship the world over. The Internet always did contain a great deal of information and had the potential to let you do a lot of research without having to go down to libraries, but without Google, it was so hard to find anything that many of us went to the library anyway. Now I rarely visit one of those sepulchers of knowledge. There’s money in a good search engine, and Google has found a way to extract some profits from it, which may or may not get it into Microsoft’s sights. But Google has a huge head start. For one thing, Google already has the infrastructure: A server farm reportedly of more than 100,000 Intel boxes running Linux. And it is adding boxes so fast that if one dies, it’s just abandoned in place, with no attempt to fix or even remove it. If Microsoft is serious about getting into the search game, it needs comparable facilities. Google has another advantage. With most companies, Microsoft can lose often but only has to win once before the battle is over and The Borg absorbs its enemy. Google, however, is now big enough to lose a couple of battles and still be around. That is, if they lose at all. Microsoft was betting heavily on the Windows File System in Longhorn to help with its search problems — the desktop Search engine is notoriously slow and clunky — but now WFS is apparently not to be in the first release of Longhorn. Jerry is a science-fiction writer and senior contributing editor to BYTE.com. You can contact him at
[email protected]. 72
Microsoft doesn’t get into fights it can’t win, and Google has considerable advantages in the search engine wars. A pitched battle between these giants would harm both, probably to the advantage of Apple. After all, almost a year ago, Steve Jobs showed live a demo of something as good as Windows File System, and that may be shipping in a couple of months. My guess is that the war won’t happen, and Google president Eric Schmidt’s flat statement that Google isn’t building a browser (http:// www.marketingvox.com/archives/2004/ 10/25/google_ceo_no_browser_effort/) was a public signal to that effect. Some Lessons in Troubleshooting Chaos Manor is now pretty thoroughly converted to gigabit Ethernet switches. The general product line we have used is D-Link. As I have noted before, D-Link equipment Just Works, and the documentation is sufficient and in good English — not that gigabit Ethernet switches need documentation. You may recall from last month that I first went to gigabit when Fry’s had a blowout sale on AirLink gigabit switches, and I bought several of them. In particular, I bought a 5-port 1000-base-T switch, which went up at “Ethernet Central.” Alas, although the AirLink 5-port was plugged into a powerstrip that supposedly has surge protection, it wasn’t connected to a UPS. This turned out to be a Big Mistake, for several reasons. First, of course, if there’s a power failure while you are using your Ethernet connection, either for connection to the Internet or for file transfers, you won’t be able to finish those jobs and shut down gracefully. Secondly, a UPS — at least, the Falcon online UPS systems I use — offers far more surge protection than any powerstrip. About a week after I installed the AirLink 5-port gigabit switch, it rained in Los Angeles. Rain here is usually either nothing or a big deal — and this time it was a big deal, with storms and lightning and thunder. It got bad enough that Roberta shut everything down in her office, but I was working on something and couldn’t be bothered. Sure enough, a few minutes later we had a short power failure. My computers kept working due to the FalDr. Dobb’s Journal, February 2005
con UPS systems, but of course I lost Internet communications. I began shutting systems down. There are enough of them that this took some time, and before I was done, the power came back on. I went back to work, but found I still didn’t have communications. My first move was to go reset the cable modem by powering it off, then back on. That’s on a UPS, so it stayed on until I reset it. When it came back up, all the lights came on — steady green for power, rapidly blinking cable connection light quickly going to steady green to indicate lockon, and green for the “PC” Ethernet connection light. The data light blinked a couple of times indicating traffic. All was well. Only it wasn’t. When I got back to my desk, I didn’t have any Internet connections with my main communications system. Time to do some troubleshooting. The first thing was to look at the system’s Ethernet port. Green light on, yellow light blinking, just what I expect. Trace the Ethernet cable to the AirLink Gigabit switch, note what port I’m connected to, and look at the lights for that port. All the port lights for the switch are blinking furiously; once again, just what I’d expect. Next, go to the cable room and see if everything there is connected. All seems well. The problem isn’t the cable modem, and it’s not out there in the Internet. All that is working, so the difficulty is between my computer and the cable modem. Hmm. Pournelle’s Law says that 90 percent of the time, it’s a cable or cable connector. Could rats have gnawed the cable? They certainly could have. The last time we had any work done here, the construction people didn’t seal the access ways to the area under the house. Rats got in, and gnawed through the signal cable for my air conditioner. Another got a telephone line, and yet another managed to get into the washing machine motor, bringing both himself and the clothes washer motor to a spectacularly messy end, as a very surprised Maytag man found when called in. We have sealed the access ports — some of them were very obscure, and we only found one when Sable, our dog, trapped a rat in the barbeque pit, and when the rat was released it made a bee line for a part of the foundation we http://www.ddj.com
hadn’t looked at. But now, all the ways under the house are sealed off and we’ve been trapping rats ever since. I think we have them all. Sable hasn’t been excited about anything skittering around the house for a couple of weeks. And even if it was a rat, there was hardly time to gnaw through an Ethernet cable in the 10 minutes or so the power was off. This had to be a different problem. Failure without Grace The next step was to do some serious troubleshooting work isolating the problem. Back to Anastasia, the main communications system. Could she ping anything? Yes. No. Yes. Hmm. That’s decidedly odd. Mostly I can’t get DNS resolution for anything, but if I type in an actual numerical URL, I can get maybe one ping out of 10 returned. Do ipconfig /all and look at the result. Seems normal, but I can’t ping Imperator (the Active Directory Server) by name or by URL address number. On a whim, I did ipconfig /release, then ipconfig /renew. That had a result. Now I wasn’t connected to anything. I couldn’t get an address from Imperator. No DHCP service. Another machine in here had the same problem. In fact, all the machines connected to the AirLink 5-port gigabit Ethernet switch had the same problem. No, or very intermittent, communications, even though the lights on the AirLink blinked furiously. I got out a different 5-port switch, connected it, and everything worked fine. The problem had been the AirLink all along, and I would have spotted it instantly if the darned thing had just failed. Instead, it failed in the worst possible way, with all its lights blinking as if it were still working, when in fact, it was as dead as a doornail. D-Link Gigabit The next day, I found myself down at the Shrine Auditorium for the Microsoft Digital Entertainment Anywhere show, where D-Link had an exhibition on the show floor. It didn’t take long to negotiate one of their Wireless Media Players and to tell them my tale of woe about the AirLink switch. D-Link’s Brad Morse asked why I hadn’t used D-Link, and I had to explain that our request had probably been lost, and I don’t like to look like I am grubbing for equipment. A couple of days later, I got the D-Link 24-Port gigabit switch as well as several 5-Port switches. I replaced the dead AirLink 5-port (actually replaced the older 10/100 switch that temporarily took its place) with one of the D-Link 5-ports, and put the 24-port system in the cable room where it replaced three different switchhttp://www.ddj.com
es — two of them older 10/100, and a 5port gigabit switch. Then I went around replacing all the AirLink switches with DLink. I can’t blame the AirLink for succumbing to a power spike, but I fault it most grievously for failing so ungracefully. When things die, they ought to look dead, not cheerfully continue to blink lights as if nothing were wrong. The bottom line here is that all the DLink gigabit equipment worked as I expected it to work, and whenever I plug in a machine capable of gigabit communications, it just works. Gigabit file transfers are some three- to five-times faster than doing the same thing at 100 megabits. I can’t call it any closer than that: That is, I have transferred the same file between the same two machines, the only difference being the Ethernet switch (D-Link gigabit versus a Netgear 100). And while the transfer times at 100 megabits are consistent, at 1000 megabits they vary in no discernible pattern, with 3× being about the slowest and 5× about the fastest. I had the same results with the AirLink switch when it was working, so it’s not a D-Link problem, and is probably caused by overloading Windows. I’m about to add a couple of Linux boxes capable of gigabit speeds, and I’ll do the same test with those. More when I know more. D-Link gigabit switches Just Work, and the time saved is significant. Building Your Ethernet Alex notes that when he is called in to troubleshoot a small business network, a good part of the time the trouble is a failed hub. Murphy’s Law dictates that the hub is behind someone’s desk, covered with dust, and its location isn’t recorded on any document. Moreover, it cascades to another hub, rather than to a switch, and it’s not at all obvious whether it needs to be connected as an uplink or downlink (on hubs that makes a difference; modern switches may not make a distinction). There are rules about how many stages of cascade you can have for hubs. It’s called the “5-4-3 rule,” and it mandates that between any two nodes on the network, there can only be a maximum of five segments, connected through four repeaters or concentrators, and only three of the five segments may contain user connections. There are reasons for this. The Ethernet protocol requires that a signal sent out over the LAN reach every part of the network within a specified length of time. The 5-4-3 rule ensures this, and thinking about it can make your head explode. If you are planning a network there’s an easier way: Get a D-Link 24-port gigabit switch, put that in a central place, Dr. Dobb’s Journal, February 2005
73
and when you wire your establishment, have enough cables pulled so that you have an outlet everywhere you might want a computer. This will likely turn out to be more lines than computers, and so what? It may even turn out to be more than 24 lines, and again, so what? Just connect up the ones you’ll use, and if you move a computer, you just plug it into a new Ethernet outlet. Of course, most of us don’t have that luxury, and when I designed this part of Chaos Manor I wired it for ARCnet, and when we later abandoned the ARCNet cables in place and pulled Ethernet wires, I didn’t plan enough outlets, so I need to cascade switches in several places. I’m giving you advice I haven’t followed, but then at Chaos Manor, we do a lot of silly things so you don’t have to. Upgrading to Gigabit Now that I have replaced all the switches with D-Link gigabit (24-port at the central location, 5-port out on the periphery), I could, in theory, put gigabit Ethernet boards in the machines that don’t have it on the motherboard. I am not going to do that, and I don’t advise you to do it. In the first place, if you put a gigabit board into a PCI slot, the PCI bus itself slows things to below a gigabit. The best I have ever
74
been able to do in upgrading from 100 megabit by adding a gigabit board is to about double the throughput. Now that isn’t anything to sneeze at, but it’s not the Earth, either. Note that gigabit-to-gigabit throughputs tend to be three- to five-times faster than 100 megabit (the best I ever got was about 6×, and that was only once). Note also that motherboards with a faster slot bus like PCI-Express will very likely have gigabit Ethernet on board. It seems reasonable, then, to just let nature take its course. Replace the switches, and as you upgrade your systems, you’ll automatically upgrade to gigabit Ethernet. It may be worth putting a gigabit card in a server, since that’s used a lot and doubling its throughput is probably worth the effort, but don’t be surprised if there’s no effect whatsoever. The best way to speed up server operations is to replace the motherboard with one that has gigabit Ethernet on a bus designed to handle it. Winding Down The game of the month is Rome: Total War. It takes an hour to install, what with all the options, and the online updating, and the rest of it. I have some complaints, but I sure like it. Fair warning: It eats time. I wouldn’t bother with the Prima Game Guide book, which gives no more information than the
Dr. Dobb’s Journal, February 2005
excellent manual that comes with the game. If you want some strategy guidance, Google “smackus maximus.” I have two books of the month, both on the same theme. First. Cowboy Capitalism: European Myths, American Reality, by Olaf Gersemann (Cato Institute, 2004). The publisher is Cato Institute, which should be warning enough of the viewpoint, but the arguments are sound and there is a lot of data, unlike the usual tome on economic policy. The second book is Bill Blunden’s Offshoring It: The Good, The Bad, and The Ugly (Apress, 2004), which once again is long on data, and intelligently discusses an important subject. The Computer Book of the Month is Getting Permission: How to License & Clear Copyrighted Materials Online and Off, by Richard Stim (Nolo Press, 2000). This is comprehensive and useful for anyone compiling materials for publication. The other computer book of the month is Mac OS X Power Hound, Panther Edition, by Rob Griffiths (O’Reilly & Associates, 2004). It’s a compilation of tricks and procedures for Mac users, and I suspect it will be as useful to experienced Mac users as it has been to me. There’s just a lot of good stuff in there. DDJ
http://www.ddj.com
PROGRAMMER’S BOOKSHELF
Enterprise Patterns Martin Heller
T
ransmitting wisdom and experience can be difficult in any sphere of activity. Probably the closest that software architects have come to a reliable way of transmitting their wisdom and experience is by abstracting patterns from their applications. Patterns are a form of “chunking,” which provides both a higher level way of thinking about a subject, and a common vocabulary. If I recognize the opening moves in a chess game as the Queen’s Gambit, I know the choices before me and their most likely consequences. If I tell another chess player I played Queen’s Gambit Declined, my meaning is clear without having to detail the moves made. Similarly, if I recognize that I have an application that needs to load and display what could turn out to be a very large database, I know I can avoid bringing the entire database into memory with a Lazy Load pattern, and I can mention the pattern to another software developer without having to draw diagrams to explain what I mean. Patterns don’t generally represent new inventions: What they reflect are old inventions that could be used for other things. Wheels, threads, and gears are all common patterns for mechanical engineers. Patterns are for things that are done over and over, so that you don’t have to reinvent them each time you need them, even if the application varies each time. There have been a number of books on software patterns over the last 9 or 10 years. Probably the best known is the seminal Design Patterns by Gamma, Helm, Johnson, and Vlissides (Addison-Wesley, 1995), also known as the “Gang of Four” or “GoF.” The Gang of Four patterns are fairly low level but universal: creational patterns, such as Factory Method and Singleton; structural patterns, such as ComMartin is a web and Windows programming consultant and Senior Contributing Editor at BYTE.com. He can be contacted at http://www.mheller.com/. http://www.ddj.com
Patterns of Enterprise Application Architecture Martin Fowler Addison-Wesley, 2003 533 pp., $49.99 ISBN 0321127420 posite and Proxy; and behavioral patterns, such as Iterator and State. Since the publication of Design Patterns, higher level and more applied patterns, such as the aforementioned Lazy Load, have become fodder for books. Martin Fowler is well known as an OOD/OOP guru (or “object bigot,” as he calls himself), and is the author of Analysis Patterns, UML Distilled, Planning Extreme Programming, and Refactoring. In his fifth book, Patterns of Enterprise Application Architecture, Fowler discusses 40 design patterns that are appropriate for enterprise applications. As Fowler explains in the preface: Enterprise applications are about the display, manipulation, and storage of large amounts of often complex data and the support or automation of business processes with that data. Examples include reservation systems, financial systems, supply chain systems, and many others that run modern business. Enterprise applications have their own particular challenges and solutions, and they are different from embedded systems, control systems, telecoms, or desktop productivity software.
The book has two major sections: the narratives and the patterns. Fowler expects you to read through all the narratives and then use the patterns for reference as the need arises. The narratives introduce some common problems encountered in enterprise applications and discuss their solutions at a high level, so that you can get a feel for which patterns solve what problems, and for how the patterns interrelate. The chapters in Part 2 are detailed references to the patterns themselves, covering how they Dr. Dobb’s Journal, February 2005
work, when to use them, and sample applications and implementations. The sample code is written in Java or C#, and illustrated with UML diagrams. The front inside cover of the book lists the patterns alphabetically; the back inside cover is a “cheat sheet” for finding the correct pattern to solve a problem. Patterns of Enterprise Application Architecture covers domain logic patterns, like Domain Model and Table Module; data source architectural patterns, like Table Data Gateway and Data Mapper; objectrelational behavioral patterns, such as Unit of Work and Lazy Load; object-relational structural patterns, such as Foreign Key Mapping and Concrete Table Inheritance; object-relational metadata mapping patterns, such as Query Object and Repository; web presentation patterns, such as Model View Controller and Front Controller; distribution patterns; offline concurrency patterns; session state patterns; and base patterns, such as Gateway, Layer Supertype, and Registry. The book does not cover asynchronous messaging, richclient interfaces, validation, error handling, security, or refactoring. Although he wrote the bulk of the exposition and sample code himself, Fowler had five contributors: David Rice, Matthew Foemmel, Edward Hieatt, Robert Mee, and Randy Stafford. Rice assisted with the excellent chapter on concurrency, and provided several patterns; Foemmel assisted with several examples and patterns; Hieatt and Mee provided the Repository pattern; Stafford provided the Service Layer pattern. Overall, Patterns of Enterprise Application Architecture is an excellent treatment of its subject matter. If you are an architect or developer working on Enterprise or Enterprise-style applications — that is, applications that work with large amounts of data— you’ll find the time spent learning these patterns more than worthwhile, whether they are new to you or familiar. DDJ 77
OF INTEREST Tech-X Corporation is offering OptSolve++ Version 2.1, a library providing implementations of algorithms for solving and optimization of nonlinear multidimensional user-defined merit functions. Nonlinear optimization algorithms include Powell, Conjugate Gradient, Nonlinear Simplex, and Levenberg-Marquardt. OptSolve++ libraries are available for Windows, Linux, or Mac OS X, and OptSolve++ Pro provides library and cross-platform source code for Linux, Mac OS X, and Windows. Tech-X Corp. 5621 Arapahoe Avenue, Suite A Boulder, CO 80303 303-448-0727 http://www.txcorp.com/ Macrovision has updated its installation software to InstallShield 10.5. New features include a network repository to share common elements, Assemblies that reuse components across different products, and the ability to edit any XML file. The new Trialware feature lets you easily build evaluation versions of software. InstallShield 10.5 also supports the latest version of Microsoft Software Installer, MSI 3.0. Macrovision Corp. 2830 De La Cruz Boulevard Santa Clara, CA 95050 408-743-8600 http://www.macrovision.com/ Absoft has announced Cluster Builder’s Kit 2.0 to replace its Beowulf Tool Kit 1.0. Absoft is also selling and supporting a new High Performance Computing Software Developers’ Kit for IBM Linux on POWER clusters and servers, and is preparing to release its IBM XL Fortran Advanced Edition compiler Version 8.1 for Mac OS bundled with Visual Numerics’ IMSL Fortran Numerical Library 5.0. Absoft Corp. 2781 Bond Street Rochester Hills, MI 48309 248-853-0050 http://www.absoft.com/ Elegance Technologies has developed CSharpener for VB, an add-in to Microsoft Visual Studio that automatically converts Visual Basic .NET code to C#. C-Sharpener for VB is integrated with Visual Studio .NET, and can be launched from within Visual Studio; a wizard guides you through the translation process. Symbol table technology is used to perform the conversion. C-Sharpener for VB does a syntactic and semantic analysis, including a full symbol tree with referenced assemblies. Elegance Technologies 1721 Green Valley Road Havertown, PA 19083 484-431-1775 http://www.elegancetech.com/ 78
Zope X3 is the next major Zope release and has been written from scratch based on the latest software-design patterns and the experiences of Zope 2. The “X” in the name stands for “experimental” because this release does not try to provide any backward compatibility to Zope 2. Zope is an open-source web-application server primarily written in Python. It features a transactional object database that can store content, custom data, dynamic HTML templates, scripts, a search engine, relational database (RDBMS) connections, and code. Zope Corp. Lafayette Technology Center 513 Prince Edward Street Fredericksburg, VA 22401 540-361-1700 http://zope.org/ Klocwork has updated its static analysis tools, which extend support for Java across the entire family to include application security testing. inForce for Java, inSight for Java, and inSpect for Java let you understand and analyze large Java architectures with very complex relationships along multiple dimensions. The new versions include system-level analysis and defect detection and prevention capabilities you as a plugin for Eclipse, a plug-in for the Rational Application Developer for WebSphere, or as a standalone capability. Klocwork 35 Corporate Drive, 4th Floor Burlington, MA 01803 866-556-2967 http://www.klocwork.com/ Windward Studios has made available Windward Reports 3.0, its Microsoft Wordbased report-layout process. With 3.0, Windward Reports is now available for use in .NET programs. A new AutoTag add-in for Microsoft Word simplifies data layout and data-source mapping, and reduces the size of tags on the template page. Users can also write JavaBeans that Dr. Dobb’s Journal, February 2005
perform final processing on the output of a tag; map to SQL, XML or custom data sources; and call Windward Reports directly from Java. Windward Studios Inc. PMB 115, 637 B South Broadway Boulder, CO 80305 303-499-2544 http://www.windwardreports.com/ GridinSoft Notepad 2.7 is designed to keep the simplicity Notepad offers while adding code-editing features you need, such as customizable syntax highlighting, a spell checker that works for 10 different languages simultaneously, console command support, a hex editor, a math expression evaluator, support for the LaTeX format, bookmarks, drag-and-drop support, and 10 other options lacking in the original Notepad. GridinSoft Proletarskaya 54/56 39617 Kremenchuk Ukraine http://www.gridinsoft.com/ Afalina has released Version 2.1 of Add-in Express .NET, a .NET component library for developing MS Office COM add-ins, SmartTags, and Excel RTD Servers. Add-in Express .NET installs technology-specific wizards that generate projects in Visual Basic, C#, Visual C++, J#, and Delphi. Afalina Co. Ltd. 149 Barykina Street Gomel, BY Belarus +375-232-473-466 http://www.add-in-express.com/ AppForge Crossfire 5.5 provides application development for Palm OS, Symbian, and Pocket PC mobile devices using the Microsoft Visual C# development environment. Other features include built-in support for radio frequency identification (RFID), support for Microsoft Windows Mobile-Based Smartphone, Motorola A1000 devices and support for Pocket PC 2003 second edition extended features. AppForge Inc. 3348 Peachtree Road NE Tower Place 200, Suite 625 Atlanta, GA 30326 678-686-9000 http://www.appforge.com/ DDJ Dr. Dobb’s Software Tools Newsletter What’s the fastest way of keeping up with new developer products and version updates? Dr. Dobb’s Software Tools e-mail newsletter, delivered once a month to your mailbox. This unique newsletter keeps you up-to-date on the latest in SDKs, libraries, components, compilers, and the like. To sign up now for this free service, go to http://www.ddj.com/maillists/.
http://www.ddj.com
SWAINE’S FLAMES
The Capital Gang
F
rom the weekly AwareSoft SoftWare Small Successes Pattern and Buzzword Brown Bag:
Barney: Well, you’ll all be happy to hear that I’ve been doing Corridor Politics with our Corporate Angel, as well as a bit of Whispering in the General’s Ear and Bridge Building, and yesterday, I got a Royal Audience to Test the Waters. Chas: That’s great, Barney, because we need to Plant the Seeds, and the best way to do that is to Stay in Touch and Involve Everyone. Annie: Not to mention getting a Guru on Our Side. Barney: We need to do all of that if we’re going to get Corporate Buy-In and build an Early Majority. But in today’s Brown Bag, I think it’s Time for Reflection. Would anyone care to Shine the Spotlight on our Strategic Vision? Chas: I’ll give it a shot, Barney. Basically, it’s that all Stakeholder Communities within our Corporate Culture should embrace a Fear Less pattern regarding Digital Rights Management of our Intellectual Property. Annie: Which means that we should Just Do It and Open-Source our Core Capabilities. Chas: It all comes down to Total Cost of Ownership up and down the Value Chain, doesn’t it, Barney? Barney: You’ve got my Buy-In on that, Chas, but what do the rest of you Early Adopters think? Rollie: I mean, like, for our Software Ecosystem, isn’t Digital Rights Management basically an instance of You Aren’t Gonna Need It? Chas: Actually, it’s YouAren’tGonnaNeedIt. Extreme Programming concepts are pronounced as one word with InterCaps. Patterns are pronounced as separate words, capitalized. Fred: If this is a Brown Bag, shouldn’t we adopt the Do Food pattern? Barney: So in the spirit of Sustained Momentum and proceeding Step by Step, could someone articulate the proposed Business Model? Chas: I believe the model is Give Away the Software and Charge for Support, Barney. Barney: I’m going to invoke the Just Say Thanks pattern, Chas, because that’s exactly right. Fred: If we can afford to Give Away the Software, why can’t we Do Food? Barney: But as we all know, every Business Model serves the Core Values articulated in a Mission Statement. Anybody want to PowerPoint our proposed Mission Statement? Chas: It’s Zero-Prerequisites Software for Clueless Losers, Barney. Annie: Uh, actually, I think that was a joke. Didn’t we change it to Extremism in Ease of Use is No Vice? Rollie: Uh-uh. That one failed to get External Validation. Red-State political references are okay, but Barry Goldwater is by current standards Too Liberal. The official proposed Mission Statement is Empowering Users through Transparent Functionality. Barney: Right, Rollie. So I think we have Group Consensus. And if it’s The Right Time, we ought to be able to get Buy-In from all the Stakeholders in the Corporate Family. Fred: Well, you’re not getting my Buy-In unless I get something to eat. Barney: So in summary: Give Away the Software and Charge for Support, Zero-Prerequisites Software, Extreme Ease of Use, Transparent Functionality. Chas: That’s it, Barney. Is it time for Next Steps now? Cyn: If I can just inject a contrarian note, I do see one problem with all this. Barney: Well, that’s why you’re our Champion Skeptic, Cyn. By adopting the Personal Touch and engaging in Bridge Building, we can encompass a whole Greek Chorus of Devil’s Advocates. Because if we’re not open to challenging input, we miss out on the Holistic Diversity that comes from Q Cyn: Here’s what concerns me, Barn. If the business model is Give Away the Software and Charge for Support, then our viability as a company rests entirely on designing the software so that it needs a whole lot of of support. And it’s not really clear to me how you reconcile that with those very nice-sounding Core Values. Barney: Um, yes, well…Is there a pattern that applies to Cyn’s input, Rollie? Rollie: Yeah. Too Much Information.
Disclaimer: To the best of my knowledge or five minutes of Googling, whichever comes first, there is no company named AwareSoft SoftWare, although there is at least one programmer using the name awaresoft in various forum postings. My use of the term is satirical in intent and maybe in effect and has no connection with any actual person or entity. In fact, neither does anything in this column.
Michael Swaine editor-at-large
[email protected] 80
Dr. Dobb’s Journal, February 2005
http://www.ddj.com
