,
Dr. Dobbs J O U R N A L
#361 JUNE 2004
SOFTWARE TOOLS FOR THE PROFESSIONAL PROGRAMMER
http://www.ddj.com
COMMUNICATION & NETWORKING • • • • • •
Remotely Controlling Windows Apps Probing Network Characteristics SALT: The HTTP-Based Speech Anonymous Communications Application Markup Unraveling the Language OpenCable API Identity & Equality in Playing the .NET Secure Shell Game GPS & .NET Wi-Fi Protected Access Ed Nisley Inside the Java Messaging Service Palm OS Security Software Performance & A Heisenberg Compensator High-Integrity Embedded Systems
Inside a Cable Modem
Jerry Pournelle Building an Internet Cafe
FEATURES
C O N T E N T S
REMOTELY CONTROLLING WINDOWS APPLICATIONS 14 by Ruben Patel
When your application is running a mile or so under water, it makes sense to control it remotely.
PROBING NETWORK CHARACTERISTICS 22 by Michael Larson
This framework lets you monitor, record, and act on packet performance.
HTTP-BASED ANONYMOUS COMMUNICATION CHANNELS 30 by Marc Waldman and Stefan Köpsell
Need a general-purpose request-reply anonymous communication channel? Here’s how to build one.
THE OPENCABLE APPLICATION PLATFORM 34 by Linden deCarmo
The OpenCable Application Platform provides Java-based APIs and uses open networking for set-top boxes and HDTV.
THE SECURE SHELL GAME 38 by Glen Matthews
Glen examines the SSH protocol and shows how it can be implemented.
WLAN SECURITY & WI-FI PROTECTED ACCESS 45 by Derek Cheung
Wi-Fi Protected Access is designed to address known WLAN security issues in the original 802.11 specification.
BUILDING A LIGHTWEIGHT JMS PROVIDER 48 by Eric J. Bruno
Here’s a Java Message Service provider that is lightweight both in size and overhead — but not in features.
SECURITY & PALM OS 5.X 57 by Michael Yam
Michael presents techniques for using masked records and encrypting/decrypting data on the Palm OS 5.x platform.
SALT: THE SPEECH APPLICATION MARKUP LANGUAGE S1 by Robert Hartman
Speech Application Language Tags let you integrate speech technologies into a range of user-oriented computing devices.
GPS PROGRAMMING & .NET S6 by Johan Franson
The Global Positioning System is a worldwide navigation system. Johan shows how to use it for .NET applications.
IDENTITY AND EQUALITY IN .NET S11 by Matthew Wilson
C#’s as operator tests an object instance against a type, and returns a reference to the given type.
TECH TIPS S15
edited by George Frazier
Boris Eligulashvili shows how to programmatically invoke the Microsoft Speech API.
STRING-BASED ATTACKS DEMYSTIFIED 61
by Herbert H. Thompson and James A. Whittaker
Far too often, programmers trust string input without checking it for validity. That’s foolish.
A HEISENBERG COMPENSATOR FOR MEASURING SOFTWARE PERFORMANCE 64 by Gary Carleton and Charles Spirakis
Measuring software performance without affecting the performance of the system being measured.
EMBEDDED SYSTEMS CODE GENERATION & HIGH-INTEGRITY EMBEDDED SYSTEMS 68 by Tom Erkkinen
Production-code generation provides a framework for adding software details to the behavioral algorithm model.
COLUMNS PROGRAMMING PARADIGMS 71
JUNE 2004 VOLUME 29, ISSUE 6
FORUM EDITORIAL 6 by Jonathan Erickson LETTERS 8 by you THE NEW ADVENTURES OF VERITY STOB 10 by Verity Stob NEWS & VIEWS 12 by Shannon Cochran OF INTEREST 83 by Shannon Cochran SWAINE’S FLAMES 84 by Michael Swaine
RESOURCE CENTER As a service to our readers, source code, related files, and author guidelines are available at http:// www.ddj.com/. Letters to the editor, article proposals and submissions, and inquiries can be sent to
[email protected], faxed to 650-513-4618, or mailed to Dr. Dobb’s Journal, 2800 Campus Drive, San Mateo CA 94403. For subscription questions, call 800-456-1215 (U.S. or Canada). For all other countries, call 902-563-4753 or fax 902-563-4807. E-mail subscription questions to ddj@neodata .com or write to Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 803226188. If you want to change the information you receive from CMP and others about products and services, go to http://www.cmp .com/feedback/permission.html or contact Customer Service at the address/number noted on this page. Back issues may be purchased for $9.00 per copy (which includes shipping and handling). For issue availability, send e-mail to
[email protected], fax to 785838-7566, or call 800-444-4881 (U.S. and Canada) or 785-8387500 (all other countries). Back issue orders must be prepaid. Please send payment to Dr. Dobb’s Journal, 4601 West 6th Street, Suite B, Lawrence, KS 66049-4189. Individual back articles may be purchased electronically at http://www.ddj.com/.
by Michael Swaine
EMBEDDED SPACE 74 by Ed Nisley
CHAOS MANOR 77
NEXT MONTH: In July, the joint will be jumpin’ with Java.
by Jerry Pournelle
PROGRAMMER’S BOOKSHELF 80 by Gregory V. Wilson
DR. DOBB’S JOURNAL (ISSN 1044-789X) is published monthly by CMP Media LLC., 600 Harrison Street, San Francisco, CA 94017; 415-905-2200. Periodicals Postage Paid at San Francisco and at additional mailing offices. SUBSCRIPTION: $34.95 for 1 year; $69.90 for 2 years. International orders must be prepaid. Payment may be made via Mastercard, Visa, or American Express; or via U.S. funds drawn on a U.S. bank. Canada and Mexico: $45.00 per year. All other foreign: $70.00 per year. U.K. subscribers contact Jill Sutcliffe at Parkway Gordon 01-49-1875-386. POSTMASTER: Send address changes to Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 80328-6188. GST (Canada) #R124771239. Canada Post International Publications Mail Product (Canadian Distribution) Sales Agreement No. 0548677. FOREIGN NEWSSTAND DISTRIBUTOR: Worldwide Media Service Inc., 30 Montgomery St., Jersey City, NJ 07302; 212-332-7100. Entire contents © 2004 CMP Media LLC. Dr. Dobb’s Journal is a registered trademark of CMP Media LLC. All rights reserved.
http://www.ddj.com
Dr. Dobb’s Journal, June 2004
3
EDITORIAL
You & Your Dr. Dobb’s Subscription
S
everal readers have complained that they’ve recently received Dr. Dobb’s Journal subscription renewal notices from a company called “Platinum Publishing Services,” which bills itself as “an independent magazine clearinghouse agent.” Let me make this perfectly clear: Platinum Publishing Services is not an agent for this magazine, and has no legitimate ties to it. Do not send the company any money for Dr. Dobb’s Journal subscriptions. Of course, part of the confusion is that Platinum Publishing Services operates under a bunch of different aliases, including I.C. Marketing, Publishers Services Exchange, Rabor Management, Lakeshore Publishing, and American Consumer Publishing Association, to name a few. You can find a more complete list of nom de plumes at Scientific American (http://www.sciam.com/ page.cfm?section=subscriberalert), whose readers have also been bothered by the company. Likewise, Running Times magazine is warning its subscribers to beware as well (http://www .runningtimes.com/special/subalert.htm). Moreover, Publishers Services Exchange (or whatever it goes by) is reportedly under investigation by the Oregon Department of Justice and is (or has been) embroiled in nasty lawsuits with magazine publishers ranging from Time and Hunting & Fishing to Coin World. To find out more about what the outfit is up to, it seemed fair to give them a hidey-ho. But when I tried calling the company, the corporate phone number turned out to be inoperable. I next did a “whois” on its domain name (http://www.acpai.com/) and dialed the number that popped up there; alas, it was disconnected. Finally, I resorted to the customer service number, which finally put me through to customer service. Remarkably, what I learned was that the company claims to be authorized to cash checks made out to “Dr. Dobb’s Journal” (it is not), that the name of the company president couldn’t be released (“for security reasons”), and that no one knew where the company’s marketing department is located (“maybe somewhere in Texas”). Still, I did learn that I could subscribe to Dr. Dobb’s Journal, but to get a special rate, I’d have to immediately commit via electronic fund transfer — paper checks not accepted. The bottom line: Don’t send this company any money for Dr. Dobb’s Journal— checks, cash, credit cards, debit cards, or wooden nickels (on second thought, go ahead and send those wooden nickels). If you hear anything from the company, forward the information to me. We’ve already notified the Oregon Department of Justice and we’ve sent the company notices barring it from saying it is an agent for DDJ. As for your subscription, if you have any questions, you drop can me a note. I may not know the answer to your question, but I do know who to ask. ••• A few weeks ago, we held a reader roundtable (well, “reader rectangular table” would be more accurate) where I had the great pleasure to meet several people who’ve written for DDJ (Shehrzad Qureshi and John Kanalakis immediately come to mind, for instance) and with whom I’ve enjoyed chatting with online over the years. Participants made a lot of good suggestions on how we can make the magazine better and we’ve already started implementing some of their ideas (and no, the space allotted for this column is not going to be halved). Still, the biggest surprise was that most of the folks around the table missed altogether the introduction of our special Windows/.NET supplement that kicked in with the April issue. I guess we didn’t do a good job explaining it. If you recall, we’re publishing two versions of DDJ— one with extra Windows./NET coverage (see pages S1 through S16 in this issue), and one without. You pick the version you want and signing up for the at-no-extra-charge additional content is simple — just go to http://www.neodata.com/cmp/ddj.html. Starting with next month’s July 2004 issue, you will only receive the extra articles if you ask for them. If you don’t request it, what you’ll likely be missing are articles on debugging .NET apps, building tamper-resistant .NET assemblies, and more. Again, if you have any questions, just drop me note. ••• And speaking of subscriptions and questions — here’s one for you. It’s going on about a year now since we introduced downloadable PDF versions of DDJ, online archives, and a bunch of other stuff. So what do you think? Any suggestions as to how we can improve things? Any ideas on how we can make it less confusing? Anything you’d like us to do differently? We’d really like to hear from you so that we can better serve you. You know the routine — just drop me a note at
[email protected].
Jonathan Erickson editor-in-chief
[email protected] 6
Dr. Dobb’s Journal, June 2004
http://www.ddj.com
LETTERS
, PO S S O BB
281 B.C., King Pyrrhus of Epirus landed on the Italian shore with 20 elephants and 30,000 soldiers to defend the Greeks against the Romans. Pyrrhus won the first battle, but lost half his army, and ultimately the war. The term “Pyrrhic Victory” comes from this battle. We’re here to serve.
T
D
2
N CE
TS
2
E-Voting Dear DDJ, E-voting is a subject touched upon by both Jonathan Erickson (“Editorial,” DDJ, February 2004) and Michael Swaine (“Programming Paradigms, DDJ, January 2004). The software for the Diebold voting machines is allegedly written in C++. To your knowledge, has anyone in the C++ universe examined the software to see what it does besides presenting screen images and “tallying votes”? Supposedly the software runs to almost a million lines of code, which to this nonexpert seems excessive, for what amounts to a “display” and “book keeping” program. David Laning
[email protected] Jonathan responds: Dave, to the best of my knowledge, the code is proprietary and hasn’t been examined by the public. I’m assuming it has been examined to some degree by the government agencies, but none that have ever said anything about it. Maybe fellow readers have a better take on this. How Do You Spell… Dear DDJ, This e-mail has nothing to do with computers. The problem is that I can’t remember how to spell “Pyric” as in a “Pyric victory.” I have asked many people and I can’t find it in the dictionary. You being editors and all, I thought you might know. There was a Greek battle where the price of victory was so high that it was almost not worth fighting at all (almost everybody died). Such victories are called “Pyric Victories.” Unfortunately, I can’t for the life of me remember how to spell it properly (“Pyric”)— and the Microsoft Word spellchecker is no help. Secondly, I am a subscriber to DDJ, so you will be helping a customer if you have an answer for this. Thomas Zammikiel
[email protected] Jonathan responds: Thomas, it is spelled “Pyrrhic” after King Pyrrhus of Epirus. In 8
Professional Programmers Dear DDJ, Andrew Todd’s letter in the April 2004 issue of DDJ reminded me of a prediction I made to a colleague about 20 years ago, probably after reading James Martin’s book on user-written programs. I predicted that foreign language options for American school children would soon include Basic and Cobol as well as Spanish and French. My colleague disagreed, saying that it wouldn’t be necessary because programming languages would become so English-like. We were both partly right. Students are learning to program at an earlier age, and there are tools that let end-users accomplish what used to require programmers. But I don’t think the distinction between users and programmers will disappear, nor should it. Having everyone who can learn a programming language call themselves programmers is just as dangerous as having everyone who can learn anatomy call themselves surgeons. To see what I mean, read the following excerpts from Can Software Kill? by Debbie Gage and John McCormick (http://www .eweek.com/article2/0,1759,1543652,00 .asp?kc=EWNWS030804DTX1K0000599). Victor Garcia considers himself lucky to be alive. Three years ago, a combination of cancer and miscalculation almost killed him. In November of 2000, Garcia and 27 other patients at the National Cancer Institute in Panama were jolted with massive overdoses of gamma rays partly due to limitations of the computer program that guided use of a radiation-therapy machine. The three Panamanian medical physicists who used the software to figure out just how much radiation to apply to patients are scheduled to be tried on May 18 in Panama City on charges of second-degree murder. In 1993, 1995, and 1998, FDA inspectors found the same deficiencies in Multidata’s software-development process coming up time and again: a lack of good software specification and documentation procedures to guide and control the software-development and change process; insufficient documentation to show that the software had been properly tested to see if it worked; and inadequate investigation into customer complaints. Software engineer [Jack] Ganssle, notes that programmers don’t need any form of certification or license to work on commercial software, including life-critical medical device software. Yet, he says, “In Mary-
Dr. Dobb’s Journal, June 2004
land, where I live, if you want to cut hair, you need to be licensed.”
The potential for misbehaving software to kill us is not just in hospitals. It’s in cars, airplanes, and probably places I don’t even know about. Eventually, I think the public will insist on more dependable software, which means programmers as well as doctors will need “special expertise,” and will be expected to “uphold a trust in employing this expertise,” to quote Mr. Todd. Although programming languages are becoming easier to learn and use, acquiring this special expertise will never be easy. Just as not everyone has what it takes to become a doctor, not everyone has the patience to analyze a problem before designing a solution, or to create a model before writing any code. In addition to having patience, expert programmers must be willing to tolerate the imprecise way in which their clients view the world, while being intolerant of any imprecision in their own work. Most importantly, those who pay for software must expect programmers to be professionals, and trust them to take as long as needed to do the right thing. In the short run, software will be more expensive. In the long run, it will be worth it. Jim Wiggins
[email protected] Column Idea Dear DDJ, As a long time reader of DDJ, I’ve enjoyed the articles that show different features of languages. All programmers at one time or another, have cursed a language for some difficulty with syntax. Like others, I’ve always thought I could design a better language syntax. With the wide range of experiences of the DDJ readership, you could have a monthly column that develops a “DDJ Language” based on input from readers. The column would evolve over time from general design goals to specific language syntax. Each month would present a topic that readers would respond to with e-mail comments. The column author, or team, would compile the best comments into a language spec posted on the web site. Some topics could require a survey or vote of readers. If the column becomes popular, continued development could be modeled after the Linux development community. Dan Laughlin
[email protected] Jonathan responds: Dan, that’s a great idea. If anyone would like to follow up on this, please drop me a note at jerickson@ ddj.com. DDJ http://www.ddj.com
The Biographies of Women Mathematicians web site notes that Ada, Countess Lovelace’s social life “in addition to Charles Babbage, included Sir David Brewster (the originator of the kaleidoscope)…[and] Charles Dickens.” Verity Stob thinks this may explain a famous failure.
F
ebruary 3rd 184–. I attended Mr. Babbage at his house to show him my new translation of Signor Menabrea’s paper on the great Analytical Engine, and — dear diary, I whisper it only to you— to mention my ideas about the generation of Bernoulli sequences. To my chagrin, I found Mr. B not at all receptive. He made great pretenses of enthusiasm about “such noble and thoughtful scratchings that ever proceeded from the pen of a member of the weaker sex,” but I was undeceived. There was some matter troubling him. When his man, who served us tea and lit the candles, had left the room, I made to ask what was the cause of his pensiveness. Mr. B said, “Oh, Lady Lovelace! Do look at this — it is wonderful!” From his desk drawer he produced a cylinder fashioned of brass of about ten inches in length and two inches in diameter, with a graspable furled collar that could be twisted. A hole was drilled at one end; the other was covered with some translucent material. “Why Mr. Babbage, for once I have the advantage of you,” I said. “It is Sir David Brewster’s beautiful-form-instrument, and it produces pleasing charming symmetrical patterns when one claps the device to one’s eye. I met Sir David at the greekmonger’s, where he had taken it to get it properly named, and he shewed it me.” “You know already? Then you know that I am too late.” “Too late, Mr. Babbage? Pray tell me, for what?” “Can you really not see it, my lady? This is surely the advance in the art of entertainment that the 19th century has been waiting for. When Brewster’s work falls into the hands of the common people, the pleasure gardens at Vauxhall will be deserted in the evenings and the very ale will go sour in its barrels at the inns for want of customers. They will all be
Verity is the pseudonym of a programmer based in the UK. She can be contacted at
[email protected]. 10
sitting at homes; twirling, twirling their brass tubes.” “Surely you exaggerate Mr. Babbage. Sir David’s invention is very clever, but it becomes quite dull after three or four minutes. I’m sure you could easily devise something twice as amusing.” “Do you really think so?” He turned to me, and I noticed that his face was livid with excitement. “Of course! Yes! You are right, my lady. I shall devise an engine so marvellous that the talk will be of nothing else from Truro to Dundee. I must start at once!” After this, he began feverishly turning the leaves of his notebook and scribbling furiously, and I could get no more sense out of him. I fear this bodes no good. February 21st. Again to the house of Mr B. This time I hoped to present to him some thoughts I had had about backing store where the machine might “remember” its state, and the idea of making it jump backwards in its list of instructions so that it might accomplish complex tasks by repetition of simple ones. Again, I was to be disappointed. I found him in his workshop, where crouched one of his great iron machines, with a ladder leaned against it. I saw that it was an old Difference Engine of the kind that I had supposed Mr. B to have put to one side. He was seated on top of the machine, adjusting the mechanism, and invited me to join him there. I said, “In these shoes? I think not. Mayn’t you join me in the terrestrial sphere?” He came down the ladder. “I am most delighted to see you, my lady, for you are just in time to witness the first trial. Howe: Start turning now.” His manservant began vigorously cranking a handle attached to the machine, which shook and trembled as its mechanism came alive. Mr. B pointed to a small crystal window in the machine’s side, behind which I could see various coloured shapes moving. “You manipulate the dropping shape with these levers, and when you make a Dr. Dobb’s Journal, June 2004
line of the same colour, they all disappear…like this!” As he spoke, there was a terrible screeching, followed by a loud crash, then silence. The servant stopped cranking. A gearwheel pinged to the ground and rolled along the parquet. Mr B. wrote something on a piece of paper affixed to the machine. I walked over and read it: 1st — C.B.— points
“Highest scores table,” explained Mr. Charles Babbage. March 4th. Attended at Mr. Babbage’s late, for dear little Lady Byron had fallen upon her knee while at play. That fool of a nurse had put vinegar on the place and made the child howl fit to shake the foundations, requiring my intercession. I had planned to tell Mr B. about an extended system of instructing (or “programming”) that I have devised for the Analytical Engine, which I am provisionally calling “Generic Haskell.” As is the usual case of recent times, I found Mr B. quite distracted with nonsense. “Good evening, Lady Lovelace. I trust you are feeling sane, tame, and advantageous to know?” This got us off to a bad start as I find it most disagreeable when inconsiderate persons mock me with the memory of Papa. “I am very well, thank you Mr Babbage,” I said coolly. “How are you progressing with the Tetris Engine?” “Oh, I have given up on that — now I have a new idea,” he said impatiently. “I am building a machine that can automatically write out English sentences.” “Yes?” I was taunted by hope; this sounded more like the Mr. B of old. “And I will use it to create my Text Adventure Engine. I am negotiating a crossmedia tie-in with your novelist friend Dickens. Look, I have already thought of an initial scenario.” He gestured to his daybook, wherein he had written: You are in a workhouse with no gruel. “But what about all the mathematical and practical things you were going to do? Your engines were to be a great benefit to the world!” “Don’t worry, sweet Lady Lovelace,” he said complacently. “There will be time enough for that.” Dedicated to the Popcap Games web site, without which this article might have been submitted on time.
DDJ http://www.ddj.com
SECTION
A
MAIN NEWS
Dr. Dobb’s
News & Views
Reign of Rexx The Rexx Language Association (http:// www.rexxla.org/) is celebrating the 25th anniversary of the Rexx programming language with an extended International Rexx Symposium. Created by Mike Cowlishaw of IBM UK Laboratories, Rexx stands for “Restructured Extended Executor Language.” As a procedural language that was designed to be used as a macro language for other applications, Rexx is considered a precursor to modern scripting languages like TCL and Python. That’s not to say that Rexx is obsolete. Currently maintained projects include Regina, a Rexx interpreter that has been ported to most current OSs; Object Rexx, IBM’s object-oriented version of Rexx, which is upwardly compatible with classic Rexx and provides interfaces to DB2, C, and C++ applications; and NetRexx, which compiles Rexx code into Java bytecode. The Rexx Standard was adopted by ANSI in 1996, under the title “Information Technology— Programming Language REXX.”
New High-Speed Internet Protocol Proposed Scientists at North Carolina State University say their Binary Increase Congestion Transmission Control Protocol (BIC-TCP) is thousands of times faster than DSL. Professors Injong Rhee and Khaled Harfoush, along with postdoctoral student Dr. Lisong Xu, presented a paper to the IEEE demonstrating transfer speeds approaching 10 Gbps — roughly 6000 times that of DSL. The problem with TCP, they claim, is that even when vast amounts of bandwidth are available on a network, TCP’s scaling mechanism is too slow to ramp up to full
12
consumption. On a 10-Gbps pipeline with 1500-byte packets, it would take a traditional TCP connection over an hour to go from half utilization to full utilization of the bandwidth. BIC-TCP is much more aggressive about consuming all available bandwidth. The problem then becomes one of “playing fair”— making sure that BIC-TCP can coexist peacefully with old-fashioned TCP traffic. BIC-TCP includes two policies designed to make sure it consumes only its fair share of bandwidth: An “additive increase” strategy keeps BIC-TCP from gobbling up bandwidth too quickly, and a “binary search increase” technique forces it to scale back as the network approaches full capacity. Rhee, Harfoush, and Xu’s paper is published in the Proceedings of IEEE INFOCOM 2004, and available at http://www4.ncsu .edu:8030/~lxu2/xu_INFOCOM_2004.pdf.
Analysts Predict Costs of Offshoring Twenty-five percent of U.S. IT jobs will be lost to offshoring by 2010, according to a new Gartner report. The research firm suggests that most of those jobs will go to India, but China and Russia are also strong contenders, with Malaysia and the Philippines benefiting as well. Gartner also says that European companies are following the outsourcing trend; by next year, the company expects one third of the major European businesses to move some of their IT operations offshore. Roughly 500,000 of the 10.3 million technology jobs in the U.S. will move offshore this year alone, according to Gartner; but the company also predicts that
Dr. Dobb’s Journal, June 2004
DR. DOBB’S JOURNAL June 1, 2004
political maneuverings combined with a few visible business failures — such as Dell’s recently abandoned attempt to direct tech support requests to a call center in India— will produce a short-term backlash against outsourcing. However, Gartner predicts that the cost-cutting imperative will win out in the long term. Other research firms concur that the offshoring trend will intensify over the next few years. Deloitte Research predicts that 5 percent of the telecom industry, and 2 million jobs in the financial sector, will move offshore by 2008, while Forrester Research expects the U.S. to lose 3.3 million jobs over the next 15 years.
Java Community Process Gets Overhaul Responding to charges that the Java Community Process (http://www.jcp.org/) is too closed and proprietary, Sun has implemented changes designed to encourage more public feedback during the JSR process. Version 2.6 of the JCP specifies that the initial review period for each JSR will be open to the public, and changes have been made to the process to encourage JSRs to enter review earlier in their development, with more issues open for feedback. Moreover, the Spec Leads on each JSR must create a “transparency plan,” demonstrating how progress on the JSR will be communicated to the public. Steps are also being taken to allow representatives from smaller companies and individual members to become Spec Leads. On the technical front, Version 2.6 of the JCP simplifies the process of creating reference implementations and Technical Compatibility Kits (required for every JSR).
http://www.ddj.com
Remotely Controlling Windows Applications Autonomous vehicles still need a driver Ruben Patel
A
common way of collecting scientific data involves the use of electronic sampling equipment. Typical scenarios for collecting biological data include long-time surveillance, surveillance near specific biomasses, and surveillance using autonomous platforms. In this article, I describe how I communicate with an EK60 SIMRAD echo sounder embedded in the High-precision Underwater Geosurvey and Inspection System (HUGIN) Autonomous Underwater Vehicle (AUV). Although this autonomous vehicle was originally designed for seabed mapping, its software can accommodate any sensor that can connect to the AUV. As it turns out, the Institute of Marine Research in Norway uses scientific echo sounders for biomass measurements. While the echo sounders work well for remote sensing, they are unfortunately attached to a very large mother ship, which fish tend to react to because of engine and propeller noise. Consequently, the behavior of the fish is altered, introducing bias into the measurements. However, due to its low noise level, the AUV can get closer to schools of fish without introducing fish reaction. Because of this, we decided to embed our sensor in the AUV, thereby letting us move sensors closer to the biomass and collect detailed information and less biased data. AUV Operation and Communication The AUV has the ability to run in autonomous or controlled mode. • In autonomous mode, the AUV conducts a preprogrammed survey with no communication with the mother vessel. This lets us use the mother ship for other activities. When the AUV survey is finished, it is recovered at a predefined time and location. • In controlled mode, the AUV is remotely controlled from the mother ship using acoustic communication links. The pilot controls the trajectory of the AUV using a 55 bps link. Data from the AUV and its onboard sensors is transmitted over a data link running at 2000 bps. Critical data from the AUV is prioritized and occupy 1000 bps, meaning there is 1000 bps remaining to share among all the payload sensors. Since the acoustic communication links have a limited range of 2000 meters vertical, the mother ship has to follow the AUV to maintain communication. Ruben is currently working on his Ph.D. at the Institute Of Marine Research in Norway, focusing on autonomous and stationary remote sensing of marine resources. He can be contacted at
[email protected]. 14
The High Precision Acoustic Positioning system (HiPAP) tracks the position of the AUV. When running the AUV in the beam of the mother ship’s echo sounder, we get detailed information of the biomass location relative to the AUV. This helps us steer the AUV close to the biomass of interest. Figure 1 shows the trajectory of the AUV while approaching and penetrating an enormous school of herring. This biological aggregation is common in some of the fjords in Norway during the herring’s wintering phase. Since no cables are attached between the AUV and mother ship, the AUV can maintain a speed of 4 knots at its maximum depth of 2000 meters. In Figure 1, data was collected by running the mother ship directly above the AUV. The sea bottom is the thick red line and the herring school the biggest red aggregation. Two smaller schools of fish can be seen in the beginning of the image, and one small school directly above the biggest. The AUV trajectory can bee seen as a red line enhanced by blue. Dispersed fish (blue dots) can be seen distributed in the image. The image is contaminated due to acoustic interference form the AUVs sensor and communication system. The depth range is 550 meters and the distance from the start of the image to the end is around 3 kilometers. Initially, the stepwise AUV trajectory was to approach the school gradually. This was not successful as the school turned downward and disappeared. Still, this is an excellent performance compared to towed bodies. The communication protocol between the sensor in the AUV and the control program on the mother vessel is complex. Figure 2 is an overview of the top side and bottom side of the HUGIN system. The horizontal dashed line indicates physical separation of the top side and bottom side. The vertical dashed line is the network connection when the AUV is connected directly to the topside system, this is done only when the AUV is onboard the mother ship. All commands and data sent to or received from the AUV go through the HUGIN Operator Station (HUGIN OS). This computer is also used for navigation, mission
Figure 1: AUV trajectory through a school of fish.
Dr. Dobb’s Journal, June 2004
http://www.ddj.com
Others POS
EK60 POS
SSS POS
HUGIN OS
NOS POS
Ship nav sys
APOS (HiPAP) Serial Lines
Ethernet
Survey Vessel
Vehicle Service Network (When AUV on Survey Vessel)
AUV
AEL/ HIPAP
Cmd Link
Data Link
RF Link
AEL/ HIPAP
Cmd Link
Data Link
RF Link
EK60 Serial Lines
SSS/ SBP
Payload Processor
Control Processor
Vehicle Network (Ethernet)
CTD
Serial Lines Others
Payload Network (Ethernet, Serial Lines)
Navigation Processor
Gyr com
DVL
Depth
DGPS
Altimeter
Collision Avoidance
Serial Line IMU
Figure 2: Communication protocol. CRBI500RemoteDialog
planning, and monitoring AUV performance. Sensors are controlled from Payload Operator Stations (POS); if a POS wants to send a command to its corresponding sensor, it has to first send it to the HUGIN OS. In turn, the command is sent over the acoustic data link to be received by the control processor. The control processor sends the command to the payload processor, which addresses it to the correct sensor plugin. All sensor plugins rely on the payload processor and are implemented as Dynamic Link Libraries (DLLs). From this point on, it is up to the programmer of the DLL to forward the message to its sensor. In our case, we send the command using TCP/IP to a bridge program. This design abstracts much of the complexity of the infrastructure in the AUV, and sensors can be added in a systematic manner. Controlling the Application The Windows-based applications we run for these studies remotely control the EK60 sensor in the AUV from the mother ship. You have several options when remotely controlling Windows applications. Commercial software packages that let you do this include PcAnywhere and Citrix. While these products give total control over the PC, they don’t meet the AUV’s communication speed and protocol needs. Consequently, I developed an alternative approach that, as a side benefit, can be used over any protocol. The basic idea behind the approach I present here is to send events from one application to another using interprocess communication (IPC) based on Windows messages. This lets me send commands to applications from programs on the same machine for opening dialog boxes, pressing buttons, and reading and setting the states of different controls. In other words, I can send commands to applications just as if I were using a keyboard and mouse. This concept opens the possibility to remotely controlling many Windows applications. You can implement complex timers for sampling data at different hours. Sensors can be connected to other sensors and programs. In our case, we wrote a bridge program that translates messages between two communication protocols, letting us communicate with the sensor from different machines. Figure 3 shows the three abstraction layers of the remotecontrol system. The first layer defines the fundamental functions for finding Windows handlers and performing simple commands http://www.ddj.com
CSurfRangeRemoteDlg
CRDialog Basic Functions
Figure 3: Abstraction layers in the bridge design. on controls (Listing One). The next layer is a generic dialog box class, which encapsulates some of the fundamental functions (Listing Two). The last classes are dialog box classes that reflect the dialog boxes in our application. In this example, we need to access the BI500 Dialog and Surface Range Dialog dialog boxes (see Listings Three and Four, respectively). Spying on Applications Before writing the bridge program, I had to decide what commands to send to the application and what data to retrieve. Typical steps for executing a command are to open the correct dialog box, alter one or more of the controls in it, then press the OK button on the opened dialog boxes. This means that I have to map all the necessary events that are generated during the command execution. Since the control of the application lies in different windows, I have to map the events to open the corresponding dialog boxes, then I have to identify events for executing different commands, and finally the event for pressing the OK button. I did this using Microsoft’s Spy++ program to spy on the message loop in the application we want to remotely control while executing the commands manually. Remote Dialog Boxes The Set Surface Range command controls the vertical depth range across the echogram. All the other commands are implemented in a similar manner. To set the surface range to 200 meters, for instance, I have to: 1. Open the BI500 dialog box. 2. Press the Surface Range button to open the Surface Range Dialog Box. 3. Enter the number 200 into the Range text box. 4. Press OK in the Surface Range Dialog Box. 5. Press OK in the BI500 dialog box. (continued on page 18)
Dr. Dobb’s Journal, June 2004
15
//handle GPT transceiver not found dialog box. CGPTTransceiverNotFoundDlg *gptNotFound = new CGPTTransceiverNotFoundDlg("SIMRAD EK60", "GPT Transceiver Not Found"); // Check if GPTTransceiverNotFound dialog is open // if so press the retry button until ok while(gptNotFound->isOpen()) { gptNotFound->PressButtonRetry(); Sleep(2000); }
For(;;) { nBytes=-1; If(test) nBytes=getCmdLineCommand(cmd); // Manually sending commands from cmd line Else if(remote) nBytes=getRemoteCommand(cmd); // Receiving commands over TCP/IP If(nBytes>0) translateCmdToEventAndSen…(cmd) if(lostConnectionWithClien()) waitForClient() sleep(400); }
Example 1: Sending events to press the Retry button.
Example 2: Pseudocode for the bridge program.
(continued from page 15) Using Spy++, I map the events that have to be sent to the application to perform each of the steps mentioned. The events are collected in a header file in Listing Five. Using the defined classes, I can set the range to 200 meters using Listing Six. Line 3 starts the application if it is not started. We then control the range value to see if it is within the valid range using the macro in line 1. A new instance of the Surface Range Dialog box is created in lines 8 and 9. The parent and child window names are set. These names are used to find the windows handler in the window tree. In line 10, the command for setting the range is executed; see Listing Four. Line 41 shows the start of the method for setting the range. This method calls the SetText method in line 33. In line 35 the BI500 dialog is first opened, then the Surface Range Dialog box is opened by pressing the Surface Range button in the BI500 dialog box. Line 36 inserts the range value and closes the dialog boxes by pressing the OK button in each of them. As you see from the code, the PostMessage and SendMessage functions are the core of the communication. These functions send specified messages to a window and call the window procedure of the specified window. SendMessage does not return until the window procedure has processed its message. In contrast, PostMessage returns immediately without waiting for the window procedure to process the message.
The Bridge The bridge program was made for running in two modes. For testing purposes, a command-line interface gives me the ability to send commands to the application. In remote mode, the bridge programs listen on a TCP/IP port. Received messages are translated to events and sent to the application. Example 2 is pseudocode for the bridge program. Between every command translation, I test for potential GPT error boxes. These rarely occur, but would halt the application for further input if not closed. I then test if we are in test or remote mode. These modes receive commands differently and, therefore, we have two different functions for each mode. If a valid command is received, the nBytes variable contains the number of bytes the command occupies. If the command contains a valid command number of bytes, it is translated to events and sent to the application. If the client has lost connection with the bridge, we wait for the client to reconnect.
Error Handling The example works only if no errors are cast by the application. If an error is thrown, a dialog box appears, notifying users of the error. In most cases, the error box blocks the application for further input. We use two methods of dealing with this. Before any command is executed, we search for error or warning boxes and close them. The other method to avoid errors is by checking the commands that are sent to the application. One problem I had with our sensor was that our application sometimes lost connection to the general-purpose transceiver (GPT). This is the piece of hardware that does the actual sampling and signal processing of the raw data collected from the transducer. This caused an error box to appear. After pressing Retry three or four times, it worked fine and data was collected. I solved this problem by sending events to press the Retry button as long as the error dialog box was open; see Example 1. Controls in dialog boxes often have a range limit and, if you try to set some value out of range, an error box appears. I avoided these types of errors by testing the range of each control and checking the range of the value before it was sent to the control, as in Listing Six, line 4. Listing One 1 2 3 4 5 6 7 8 9 10 11 12
18
// Header #ifndef __GLOBAL_HH__ #define __GLOBAL_HH__ #include typedef struct { HWND hwnd; const char *title; } FindWnd; CALLBACK CheckWindowTitle( HWND hwnd, LPARAM lParam );
Conclusion You need to be careful when remotely controlling applications. It is important to map all the potential errors that can appear during application use. One unknown error can halt the whole communication. Dialog boxes can use some time before they appear or close. It is therefore important to halt any commands before we are sure that the dialog box is open or closed. In my case, I poll the window tree to see if we can find the dialog box. Some programs are unstable and occasionally shut down or halt. A good rule is to check whether the sensor program is running before sending any commands. If it is not running, then start it before the command is sent. It was necessary to implement commands for stopping the sensor, restarting the computer, and shutting down the computer. I needed to stop the sensor and shutdown the computer before the AUV was launched and recovered. This was to reduce the risk for damaging the transducer and hard disk. There is always the danger of getting a total machine halt, like the blue screen in Windows. To recover from this, the control processor can recycle power on all the sensors at command from the HUGIN OS. I tested the system during a cruise period over two weeks. My experience during this cruise is that this way of remote controlling and reading data from an application can indeed be used for remote sensors. The method is easy to implement and can be used in many ways. If the application behavior is well investigated a robust communication protocol can be developed. DDJ 13 14 15 16 17 18 19 20 1 2 3 4 5
HWND HWND HWND HWND HWND HWND
FindWinTitle(const char *title); FindWndByTitle(const char *parent,const char *child); WaitForDialogToOpen(char *parent,int timeout); WaitForDialogToClose(char *parent); OnShotOpenDialog(char *parent); LeftClickInAt(const char *parent,const char *child,int x,int y);
#endif // Cpp file #include "Global.h" // Se if the specified window has a specified title CALLBACK CheckWindowTitle( HWND hwnd, LPARAM lParam )
Dr. Dobb’s Journal, June 2004
http://www.ddj.com
6 { 7 char buffer[MAX_PATH]; 8 // Get the window title form window 9 GetWindowText( hwnd, buffer, sizeof( buffer ) ); 10 FindWnd * fw = (FindWnd *)lParam; 11 // Compare window tile with title to be checked. 12 if(strcmp( buffer, fw->title ) == 0 ) 13 { 14 fw->hwnd = hwnd; 15 return FALSE; 16 } 17 return TRUE; 18 } 19 // Find a parent window by it window title 20 HWND FindWinTitle(const char *title) 21 { 22 FindWnd fw; 23 fw.hwnd = 0; 24 fw.title = title; 25 EnumWindows( (WNDENUMPROC) CheckWindowTitle, (LPARAM) &fw ); 26 return fw.hwnd; 27 } 28 // Find a child window by it window title 29 HWND FindWndByTitle(const char *parent,const char *child) 30 { 31 FindWnd fw; 32 fw.hwnd = 0; 33 fw.title = child; 34 HWND hWnd = FindWinTitle(parent); 35 if(child==NULL) return hWnd; 36 else 27 { 28 ::EnumChildWindows(hWnd, (WNDENUMPROC) CheckWindowTitle, (LPARAM) &fw); 29 return fw.hwnd; 40 } 41 42 } 43 // Halt until a dialog is opened 44 HWND WaitForDialogToOpen(char *dlgName,int timeout) 45 { 46 HWND hWndDlg; 47 hWndDlg = NULL; 48 // Loop until the window is opened 49 do 50 { 51 hWndDlg= FindWndByTitle(dlgName,NULL); 52 } while(!hWndDlg); 53 return hWndDlg; 54 } 55 // Halt until a dialog is closed 56 HWND WaitForDialogToClose(char *dlgName) 57 { 58 HWND hWndDlg; 59 hWndDlg = NULL; 60 // Loop until the window is opened 61 do 62 { 63 hWndDlg= FindWndByTitle(dlgName,NULL); 64 } while(hWndDlg); 65 return hWndDlg; 66 } 67 // Open dialog 68 HWND OnShotOpenDialog(char *dlgName) 69 { 70 HWND hWndDlg; 71 hWndDlg = NULL; 72 // Get the handler 73 hWndDlg= FindWndByTitle(dlgName,NULL); 74 return hWndDlg; 75 } 76 // Left click in a child window at position x,y 77 HWND LeftClickInAt(const char *parent,const char *child,int x,int y) 78 { 79 // Get window handler 80 HWND hWnd = FindWndByTitle(parent,child); 81 WPARAM wParam = MK_RBUTTON; 82 LPARAM lParam = MAKELPARAM(x,y); 83 // simulating left mouse click in window 84 if(!::PostMessage(hWnd, WM_RBUTTONDOWN ,wParam,lParam)) 85 return NULL; 86 return hWnd; 87 }
Listing Two 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
// RDialog.h: interface for the CRDialog class. #ifndef __CRDialog_H #define __CRDialog_H #include class CRDialog { public: CRDialog(char *parent,char *child); ~CRDialog(); HWND long DWORD
IsDialogOpen(char *dlgName); // Check if the dialog // with name in dlgName is open SetText(int nIDDlgItem,char *text); // Set text in a control SetCheck(int nIDDlgItem,BOOL checked); // Check or // uncheck a check
box 16 HWND CloseDialog(void); 17 BOOL PressButton(int nIDDlgItem); 18 19 char *m_sParent; 20 char *m_sChild; 21 22 }; 23 #endif 1 2 3 4 5 6 7 8 9 10 11
// Close this dialog box // Press a button // String to parent window // String to this dialog box
// CRDialog class implementation #include #include "CRDialog.h" #include "Global.h" #include "EK60MK1ID.h" // Set string of parent and child window CRDialog::CRDialog(char *parent,char *child) { int len1=strlen(parent)+1; int len2=strlen(child)+1;
(continued on page 20) http://www.ddj.com
Dr. Dobb’s Journal, June 2004
19
(continued from page 19) 12 m_sParent = new char[len1]; 13 m_sChild = new char[len2]; 14 sprintf(m_sParent,"%s",parent); 15 sprintf(m_sChild,"%s",child); 16 } 17 CRDialog::~CRDialog() 18 { 19 delete[] m_sParent; 20 delete[] m_sChild; 21 } 22 // Close dialog box 23 HWND CRDialog::CloseDialog(void) 24 { 25 // Find handler of dialog box form string 26 HWND hWndDlg = FindWinTitle(m_sChild); 27 // Close dialog by pressing the OK button 28 ::SendDlgItemMessage(hWndDlg,RIDC_BUTTON_OK,BM_CLICK,0,0); 29 // wait for dialog to close 30 return WaitForDialogToClose(m_sChild); 31 } 32 // Return handler of dialog specified by window name 33 HWND CRDialog::IsDialogOpen(char *dlgName) 34 { 35 return FindWndByTitle(dlgName,NULL); 36 } 27 // Set text in control in dialog box 28 long CRDialog::SetText(int nIDDlgItem,char *text) 29 { 40 return::SendDlgItemMessage(FindWndByTitle(m_sChild,NULL), nIDDlgItem,WM_SETTEXT,0,(LPARAM)text); 41 } 42 // Check or uncheck a check control 43 DWORD CRDialog::SetCheck(int nIDDlgItem,BOOL checked) 44 { 45 // manipulate control in dialog 46 DWORD wParam ; 47 // Check it 48 wParam = (WPARAM) (checked)?(BST_CHECKED):(BST_UNCHECKED); 49 ::SendDlgItemMessage(FindWndByTitle(m_sChild,NULL),nIDDlgItem, BM_SETCHECK,wParam,0); 50 // Check if success 51 return::SendDlgItemMessage(FindWndByTitle(m_sChild,NULL), nIDDlgItem,BM_GETSTATE,0,0); 52 } 53 // Press a button 54 BOOL CRDialog::PressButton(int nIDDlgItem) 55 { 56 HWND hWndDlg; 57 // Check if the dialog is open 58 hWndDlg = IsDialogOpen(m_sChild); 59 // get handler of button to press 60 HWND hWndCont = ::GetDlgItem(hWndDlg,nIDDlgItem); 61 // Press it 62 ::PostMessage(hWndCont,BM_CLICK,0,0); 63 return TRUE; 64 }
Listing Three 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
// BI500RemoteDlg.h: interface for the BI500RemoteDlg class. #ifndef __BI500RemoteDialog_H #define __BI500RemoteDialog_H #include "EK60MK1ID.h" #include "CRDialog.h" class CBI500RemoteDlg:public CRDialog { public: CBI500RemoteDlg(char *parent,char *child); virtual ~CBI500RemoteDlg(); BOOL BOOL private: HWND BOOL BOOL
PressButtonSurfaceRange(); PressButtonOK(); OpenDialog(void); SetText(int nIDDlgItem,char *text); PressButton(int nIDDlgItem);
}; #endif // BI500RemoteDlg.cpp: implementation of the BI500RemoteDlg class. #include "CBI500RemoteDlg.h" #include "global.h" CBI500RemoteDlg::CBI500RemoteDlg(char *parent,char *child) :CRDialog(parent,child) {} CBI500RemoteDlg::~CBI500RemoteDlg() {} HWND CBI500RemoteDlg::OpenDialog(void) { HWND hWndDlg; // Check if dialog is already open hWndDlg = IsDialogOpen(m_sChild); if(hWndDlg) return hWndDlg; // Open the dialog if(!::PostMessage(FindWinTitle(m_sParent), WM_COMMAND, RID_INSTALL_BI500,0 )) return NULL; // get dialog handler hWndDlg = WaitForDialogToOpen(m_sChild,1000); return hWndDlg; } BOOL CBI500RemoteDlg::SetText(int nIDDlgItem,char *text) { OpenDialog(); CRDialog::SetText(nIDDlgItem,text); CloseDialog(); return TRUE; } BOOL CBI500RemoteDlg::SetSurfVals(char *Surf) { SetText(RIDC_LIST_NOSURFVALS,Surf); return TRUE; }
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 BOOL CBI500RemoteDlg::PressButton(int nIDDlgItem)
20
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
{
}
HWND hWndDlg; hWndDlg = IsDialogOpen(m_sChild); if(!hWndDlg) hWndDlg=OpenDialog(); HWND hWndCont = ::GetDlgItem(hWndDlg,nIDDlgItem); ::PostMessage(hWndCont,BM_CLICK,0,0); return TRUE;
BOOL CBI500RemoteDlg::PressButtonSurfaceRange() { return PressButton(RIDC_BUTTON_SURFRANGE); } BOOL CBI500RemoteDlg::PressButtonOK() { return PressButton(RIDC_BUTTON_OK); }
Listing Four 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
// CSurfRangeRemoteDlg: interface. #ifndef _SURFRANGEREMOTEDLG_H #define _SURFRANGEREMOTEDLG_H #include #include #include #include
"EK60MK1ID.h" "CRDialog.h" "CBI500RemoteDlg.h" "Global.h"
class CSurfRangeRemoteDlg :public CRDialog { public: CSurfRangeRemoteDlg(char *parent,char *child); virtual ~CSurfRangeRemoteDlg(); BOOL SetRange(char *range); BOOL SetStart(char *start); private: HWND OpenDialog(void); HWND CloseDialog(void); BOOL SetText(int nIDDlgItem,char *text); }; #endif
1 // CSurfRangeRemoteDlg: implementation. 2 #include "SurfRangeRemoteDlg.h" 3 4 CSurfRangeRemoteDlg::CSurfRangeRemoteDlg(char *parent,char *child) 5 :CRDialog(parent,child) 6 {} 7 CSurfRangeRemoteDlg::~CSurfRangeRemoteDlg() 8 {} 9 HWND CSurfRangeRemoteDlg::OpenDialog(void) 10 { 11 HWND hWndDlg; 12 // Open BI500 dialog 13 CBI500RemoteDlg BI500RDlg(m_sParent,"BI500 Dialog"); 14 // Press the Surface Range button in the BI500 dialog 15 BI500RDlg.PressButtonSurfaceRange(); 16 hWndDlg = WaitForDialogToOpen(m_sChild,1000); 17 return hWndDlg; 18 } 19 HWND CSurfRangeRemoteDlg::CloseDialog(void) 20 { 21 HWND hWndDlg = FindWinTitle(m_sChild); 22 // Close dialog 23 ::SendDlgItemMessage(hWndDlg,RIDC_BUTTON_OK,BM_CLICK,0,0); 24 // wait for dialog to close 25 WaitForDialogToClose(m_sChild); 26 //CRDialog::CloseDialog(); 27 CBI500RemoteDlg BI500RDlg(m_sParent,"BI500 Dialog"); 28 BI500RDlg.PressButtonOK(); 29 return NULL; 30 } 31 32 33 BOOL CSurfRangeRemoteDlg::SetText(int nIDDlgItem,char *text) 34 { 35 OpenDialog(); 36 CRDialog::SetText(nIDDlgItem,text); 37 CloseDialog(); 38 return TRUE; 39 } 40 41 BOOL CSurfRangeRemoteDlg::SetRange(char *range) 42 { 43 SetText(RIDC_LIST_SRANGE,range) ; 44 return TRUE; 45 } 46 47 BOOL CSurfRangeRemoteDlg::SetStart(char *start) 48 { 49 SetText(RIDC_LIST_STARTSURF,start); 50 return TRUE; 51 };
Listing Five 1 2
#define RID_INSTALL_BI500 #define RIDC_BUTTON_SURFRANGE
32878 0x510
3 4
#define RIDC_LIST_SRANGE #define RIDC_BUTTON_OK
0x3ec 0x01
// ID to activate BI500 dialog // Button to push for activating // Surface Range Dialog box. // ID for surface range text box // Ok button id
Listing Six 1 #define IsInRange(val,min,max) if(val>=min && valSetRange(val); // Set range 11 delete surfRangeDlg; 12 }
DDJ Dr. Dobb’s Journal, June 2004
http://www.ddj.com
Probing Network Characteristics A distributed network performance framework Michael Larson
I
t is one thing to blindly push packets onto the Internet, and quite another to monitor, record, and act on packet performance. For instance, say that the available network bandwidth is of interest and you would like to act on a sudden constriction of a connection’s available bandwidth. Identifying a sustained drop in bandwidth would allow you to terminate lower priority streaming video connections or transfer the connection to a different server. In this article, I present a simple framework that can support this behavior and more. It is flexible enough in utility that the framework is analogous to a network Swiss army knife. It can fit into your network to aid in the diagnosing and resolution of network events as they occur. This framework, which I call “PerfScout,” is based on a message-queuing system where messages are dispatched between different blocks of responsibility. Each block operates independently and dispatches messages in a fire and forget manner. What’s more, PerfScout framework is fairly OS agnostic and can be recompiled to a large number of supported operating systems with minimal work (over 32 different flavors including Mac OS X, most UNIX variants, Windows, plus others). The complete source code for PerfScout is available electronically; see “Resource Center,” page 3.
your network connection to specific destinations. And suppose that bandwidth per connection is an important quantity for customers trying to upload videos to your web site for processing. Low bandwidth would tend to indicate that a customer would give up trying to upload data — which translates to a lost opportunity. That is one scenario; other important network performance scenarios could include Voice over IP and latency, streaming video and the available bandwidth, file copying and the packet loss, and router congestion and low-priority data transmission. In situations such as these, you would want to monitor network performance and act on poorly performing connections. Actions that might be appropriate for poor connections would be to move the client to less loaded servers, to limit the number of connections to servers, delay in sending data, notifying network engineers or automated route control systems, or to simply record your network’s performance. Heck, recording your network’s performance may even be a legal requirement (or you may also want to independently verify your IPS’s records). PerfScout allows for extensions in three areas: • Sources. How data destinations of interest are to be identified. • Tools. What network performance is measured. • Actions. How to respond to network events.
Knowing Your Performance Needs Say that your digital video processing site has certain expectations on the quality of
PerfScout is built on a framework that provides for concurrent processing at each stage of the pipeline with these capabilities. The provided implementation contains two types of network performance tests: network bandwidth estimator and a latency tool. Other sources, tools, and actions can easily be plugged in to provide a more complete picture of your network’s connections.
Michael is a principal software engineer at Proficient Networks. He can be reached at
[email protected].
Efficient Processing of Results A key aspect of the PerfScout design is its flexibility. This model can be split into a
22
Dr. Dobb’s Journal, June 2004
distributed set of components. PerfScout encapsulates communication between each block of work and allows the engineer to focus on just two details: • How to efficiently manage work that is requested of me. • How to perform the work that is being requested in a timely manner. Figure 1 shows a processing block layout of PerfScout in which there are three basic blocks in this design. The “glue” between each block is the messaging framework that provides the conduit for the dispatching and reception of messages. This glue can be upgraded for various distributed implementations (such as TCP/UDP communication or interprocess communication) and is highly portable (through the use of the ACE library; see the accompanying text box entitled “The ACE Library”). It certainly is possible to mix transport mechanisms between these blocks. The relative order of processing is from left to right. A fully distributed PerfScout would be pretty neat. Potential uses could be: • Coordination of multiple streaming servers at various locations. • Notifications of users accessing streams. • Shutdown of streams if bandwidth limitations are reached. Figure 2 shows a distributed PerfScout ensconced in a hypothetical network where the web server dispatches destinations that are of interest (in this case, newly connected clients) to be measured. Finally, these results are recorded in a database, and an action request (such as dropping low-priority clients when bandwidth drops) sent back to the web server. PerfScout pipelines messages through each stage of processing (ownership of a message is the responsibility of the input queue’s owner). Each stage must manage its input queue of work intelligently and in a manner that avoids queue overflow. Relying on this messaging interface http://www.ddj.com
Internet Processing Blocks
Source
Task (threaded)
Task (threaded)
Tool
ProcessTask (interprocess)
ProcessTask (interprocess)
NetworkTask (TCP/UDP)
NetworkTask (TCP/UDP)
Front-End Servers
Action
PerfScout Source
PerfScout Tools
Messaging Transport PerfScout Actions
Messaging Flow
Network Performance DB
Figure 1: PerfScout processing blocks and flow.
Figure 2: Potential deployment of PerfScout within a network.
cessing of messages within each processing block. This work is generally algorithm specific and encapsulated within the component, but involves specific workflow rules. This design scales well and lets you add more processing requirements at any step, provided that queue sizes are managed in a timely manner (so that queue overflow is avoided). Finally, it is worth noting that PerfScout can support additional components without requiring a recompilation of code (with some small modifications to the current design — perhaps a dynamically loading processing block).
.(continued from page 22) cleanly encapsulates the specific processing details for each component, and concretely defines the input/output relationship for each processing block. Correct management of each block’s input queue results in a highly scalable framework. It is worth noting that this type of design is used in several commercial products, such as Microsoft’s Message Queue Center (MSMQ) or IBM’s MQSeries. For our needs, these canned messaging systems are a bit of an overkill. For the task at hand, what is needed is a thin, streamlined implementation that provides a highly flexible and efficient implementation that is portable across a wide variety of platforms. The job of software designers within this framework is to optimize the pro-
PerfScout Design Figure 3 shows the object diagram for PerfScout (the Source, Tool, and Action blocks
Task put(); get(); getAll(); run()=0;
ToolManager ActionManager TestSource CompleteTest
ExternalSource BandwidthImp DB
http
LatencyImp
ToolBase
Sources
Tools
ActionBase
Actions
Figure 3: PerfScout object diagram. 24
Dr. Dobb’s Journal, June 2004
are shown here in relation to Figure 1). The Tool and Action blocks require a manager to effectively handle incoming messages. These messages are subsequently dispatched internally for processing (the Source processing block has no input queue and, therefore, no manager — sources operate independently with no coordination). Operations within the Tool and Action blocks are coordinated through their respective Managers. These blocks contain processing clients that perform the real work within PerfScout. These clients derive from their respective base classes (ToolBase and ActionBase) and provide a common interface for initialization and communication. Each entity that communicates using a message queue within PerfScout derives from the Task base class (Listing One). Task, in turn, derives from the ACE base class Task_Base. The Task class is a simple wrapper around the ACE library that lets PerfScout support a message notification system. Task provides two basic services: a worker thread and a message queue. Task::put( ) and Task::get( ) are used to access the message queue, while the run( ) method services the worker thread. Task::put( ) and Task::get( ) are mutexed to prevent concurrent access, and Task::get( ) blocks when the message queue is empty. Exiting Task::run( ) terminates execution of the thread, which should not occur unless a Task-derived object is to be restarted or shutdown. Task is templatized on the message type, which, for PerfScout, is always the Test object. Derived classes are required to implement their own ::run( ) method. Pretty straightforward stuff. Any modifications specific to the message queue behavior, or transport of messages, needs only to be done within the Task base class. The common communication token dispatched between objects that derive http://www.ddj.com
(continued from page 24) from the Task base class is the Test object (Listing Two). The Test object contains data used to identify target destination, requested tests, current active test, time of test request and collection, and finally the test results (in the Result object). The current active test, m_curTest, is required so that requested test types can be performed sequentially on a Test object within the Tool Manager. The accessor ::getActiveTest( ) returns the value of m_curTest. Encapsulating the current test means that the Tool Manager doesn’t need to track progress of individual Tests. When all tests are complete, the Test object will return ::kEnd via Test::getActiveTest( ) signaling the completion of processing for a specific
Test object. The Test object wraps an additional object — Result. The Result object maps tests to their results. The Result object is contained as a map collection within the Test object and is templatized on the data stored. There are many ways to handle input queue processing. The initial implementation of PerfScout uses a rather straightforward (or simple) approach where the maximum size of the queue is defined at instantiation — any puts into queues that exceed this limit are silently dropped. Therefore, all objects that derive from Task need to manage their queues in a quick and efficient manner, ensuring that messages do not back up. The Tool Manager and Action Manager dispatch messages to their clients quickly; therefore,
the real work at the client level is to efficiently process these tests within the queue of each client. There are two source implementations included within PerfScout: TestSource dispatches a fixed set of tests at five minute intervals and contains the location and types of tests to be performed, and ExternalSource contains a UDP connection that lets requests be remotely dispatched to this component. Other sources can obviously be built and inserted at this level, such as integration with a web server, router, route-control device, and so on. The second processing block, known as the “tool block,” contains the ToolManager (Listing Three) that is derived from the Task base class. Incoming Test requests are received by the ToolManager and dispatched to the tool clients contained within the Tool block. These clients derive from ToolBase, which provides a basic framework to perform specific network tests. The run( ) method in ToolManager receives messages and dispatches these requests to the corresponding tool
The ACE Library
A
daptive Communication Environment (ACE; http://www.cs.wustl .edu/~schmidt/ACE.html) is a C++ open-source software middleware toolkit created to support network programming and performance-driven systems across a wide range of platforms. ACE provides many basic patterns in areas such as: multithreading, networking, memory management, and the like. The ACE library employs a wrapper façade design that allows applications to use ACE interfaces that encapsulate and enhance the native OS concurrency, communication, memory management, event demultiplexing, dynamic linking, and file system APIs. Applications can access this functionality by selectively inheriting, aggregating, and instantiating components. By encapsulating lower level operating system APIs, ACE supports a broad range of platforms. A word of caution: One criticism of the ACE library has been that, given the wide range of platforms, only the lowest common denominator of system capabilities are sometimes supported — do your homework when considering incorporating the ACE library into your future project. For purposes here, ACE provides a framework that enables portability with a minimum of code rework on the implementers part. — M.L.
26
Dr. Dobb’s Journal, June 2004
http://www.ddj.com
as specified by the Test object. There is a completion thread as part of the ToolManager encapsulated within ToolManager (called the CompleteTest class). The CompleteTest::run( ) method receives completed tests and either circulates the Test to the next test or pushes out the finished tests to the next processing block (the Action Block). The determination of when a Test has finished processing within this block is determined by the Test object itself. For tool clients, it often makes sense to allow for processing of multiple tests to occur simultaneously across multiple worker threads, as is the case for the two provided tool implementations. Now to the final processing block, the Action block, which contains the ActionManager. Test results are interpreted here and actions can be performed based on these results. The ActionManager receives a single Test message containing test results for a specific destination. In this block, there again can be multiple clients as in the Tool block. Each client within the ActionManager block receives a copy of the Test object as in method ActionManager::run( ). In PerfScout, there are two client implementations: One is a notification action to drop poorly performing connections, and the other is an interface to writing the data to a SQL database. All kinds of useful action modules are possible at this level, such as a module that shutdowns a stream if the bandwidth reaches a certain lower limit, or an e-mail notification system on certain network events, or a recording/reporting module that lets you capture network health over a long period of time, and so on.
Available bandwidth is not a direct quantity that can be measured but must be inferred from at least two different latency measurements. The algorithm used in this implementation is a rather simple algorithm used to compute bandwidth — remember, it is a simple matter to plug in different versions as they are developed. Bandwidth is inferred from latency values received for packets of differing sizes — the theory is that a larger packet will encounter more latency due to longer send/receive queues at the Layer 2 level (Ethernet)— where the original data packet is broken down into smaller chunks for transmission over the wire. If you were to draw a linear relationship between two measurements of differing packet sizes,
then this equation can be used: Bandwidth=(lat2-lat1)/(bytes2-bytes1) (bytes/ ms), where lat2 and lat1 are the results of two latency measurements, and bytes2 and bytes1 are the results of respective byte counts of packets used in the latency measurements. For the PerfScout implementation, the first packet is 40 bytes and the second packet is 1300 bytes in size. The assumption of a linear relationship between latency and bandwidth is not always true for various reasons such as router input queues, congestion, and so on. Therefore, multiple tests help identify and remove outliers in computing this value. Conclusion PerfScout can be customized and expanded in many directions, as suits the needs
Bandwidth Detection and Latency Two tool clients are provided within PerfScout — a bandwidth detection tool and a latency tool. Latency is a fairly straightforward quantity. Latency measures the roundtrip delay in the transport of the packet. There are several ways to capture latency. This tool uses UDP probes to compute the latency. This approach means that latency can only be captured from hosts that respond to TTL-expired packets (other techniques such as http SYN/ACK or icmp probes can avoid this limitation, but have other limitations in their use). Latency captures the round-trip packet travel time. Latency is important to applications that are sensitive to delays such as voice over IP. It is also worth noting that passively collected data such as an http SYN/ACK latency retrieved from a web page with a 1 pixel gif would mean that the Source processing block would effectively collect the test data, which would then push the Test data directly to the Action processing block, bypassing the Tools block. http://www.ddj.com
Dr. Dobb’s Journal, June 2004
27
of your network. New tools, sources, and actions can be added and processing blocks can be redistributed across your network. A flexible monitoring tool is a must have
for performance critical networks. PerfScout can fulfill this role by providing a customizable framework that eases integration into a network environment. Once inte-
Listing One
DDJ
private: string m_taskName;
template class Task : public Task_Base { public: /* * Constructor and Destructor */ Task (const string &taskName, int iMaxQueueSize = 1000); virtual ~Task (void); /* * Initialize object */ virtual int init(); /* * orderly shutdown of task */ virtual int shutdown(); /* * putter/getter for message queue */ virtual int put(T *pT); virtual T* get(); /* * size of message queue */ virtual int size();
// Simple Barrier to make sure all service threads have // entered their loop before accepting any messages. ACE_Barrier m_barrier;
};
// This Lock_Adapter is used to synchronize operations // on the ACE_Message_Block objects static ACE_Lock_Adapter lock_adapter_; //count of queue int m_iQueueCt; int m_iMaxQueueSize;
Listing Two class Test { public: typedef enum {kBegin = 0, kBandwidth = 1, kLatency = 2, kEnd = 3} TestType;
/* * thread entry point */ virtual int svc(); /* * derived class must implement. main processing entry point. */ virtual void run() = 0; /* * name of task */ string getName() {return m_taskName;} /* * Returns a pointer to the lock adapter. */ static ACE_Lock_Adapter *lock_adapter (void); protected: int initTask();
28
grated PerfScout can monitor, record, and act on events within your network.
public: typedef map ResultColl; typedef map::iterator ResultIter; public: /* * Constructor and Destructor */ Test() : m_ulTarget(0), m_curTest(kBegin) {addTest(kBegin);} ~Test(); /* * Copy Constructor */ Test(Test &test); //copy constructor /* * Assignment operator */ Test& operator=(Test &test); //assignment operator /* * Returns target host */ unsigned long getTarget() {return m_ulTarget;} /* * Sets target host
Dr. Dobb’s Journal, June 2004
http://www.ddj.com
};
*/ void setTarget(unsigned long ulTarget) {m_ulTarget = ulTarget;} /* * Copies test object */ void copyTest(TestType type, ResultBase *pResult); /* * Adds new test type */ void addTest(TestType type); /* * Returns curren test */ TestType getActiveTest() {return m_curTest;} /* * Finds next test */ TestType nextTest(); /* * Have all tests been iterated? */ bool isLastTest(); /* * Sets Latency */ void setLatency(double dVal); /* * Sets Bandwidth */ void setBandwidth(long lVal); /* * Gets Latency */ double getLatency(); /* * Gets Bandwidth */ long getBandwidth(); /* * Returns result collection */ ResultColl getResults() {return m_results;} /* * Converts value to test type */ static TestType toTestType(unsigned char ucType); private: unsigned long m_ulTarget; //target address ResultColl m_results; //result collection TestType m_curTest; //current test
Listing Three /** * ToolManager::run() * Main entry point for tests. Tests are cycled through tools. * **/ void ToolManager::run() { ToolBase *pToolBase; cout getActiveTest()); if (pToolBase != NULL) { pToolBase->put(pTest); } else { delete pTest; } } else { cout put(pTest); } } } }
DDJ
http://www.ddj.com
Dr. Dobb’s Journal, June 2004
29
HTTP-Based Anonymous Communication Channels Keeping servers from knowing your true identity Marc Waldman and Stefan Köpsell
relatively easy for third parties to track and catalog an individual’s online activity, including web-browsing history, online purchases, news postings, and the like. Anonymizing tools provide individuals with some form of protection against these third parties. For more information about anonymizing tools and the need for them, see “Privacy-enhancing Technologies for
T
he need for Internet-based anonymous communication has been well documented in the media. Unfortunately, the Internet currently lacks a deployed, general-purpose infrastructure that supports anonymous communication between distributed applications. With a little work, however, you can use existing web-based anonymity tools to support anonymous communication. In this article, we describe how to create a generalpurpose request-reply anonymous communication channel by utilizing currently deployed web-based “anonymizing” tools. Channels such as these let clients and servers exchange messages in such a way that servers don’t know the true identity of clients. Admittedly, anonymous Internet-based communication is a controversial topic. As with a lot of software, there is always concern that individuals could abuse anonymity for malicious purposes. However, today’s Internet-based applications make it Marc is an Assistant Professor in the Computer Information Systems Department at Manhattan College. He can be contacted at
[email protected]. Stefan is a Ph.D. student at the Institute for System Architecture, Dresden University of Technology, Germany. He can be contacted at
[email protected]. 30
the Internet,” by Ian Goldberg, David Wagner, and Eric Brewer (http://www.cs .berkeley.edu/~daw/papers/privacycompcon97-www/privacy-html.html). In this article, we examine the anonymity provided by various web-based anonymizing tools, and show how you can interface with an existing web-based anonymizing tool to create an anonymous communication channel between distributed applications. While all source code is implemented in Java, the techniques we present can be used with any programming language that supports socket-based communication. Web-Based Anonymity Over the past few years, a number of tools have been developed that let you anonyDr. Dobb’s Journal, June 2004
mously retrieve web pages. By anonymously we mean that the web server does not learn the true IP address of the webbrowsing client. Roughly speaking, these tools can be classified as being either mixnet-based or proxy-based. Proxy-based designs are simpler than mix-net-based designs, but usually offer a weaker form of anonymity. Figure 1 shows the typical architecture of proxy-based web anonymity tools such as Anonymizer (http://www.anonymizer.com/) and Rewebber (http://www.rewebber.de/). The web-browsing client sends its web-page request to the proxy server. The proxy server makes the web-page request on behalf of the client. The web page that results from this request is then sent back to the requesting client. The server only learns the proxy’s IP address. Clearly, this is not a strong form of anonymity, as the proxy itself knows the client and the requested web page. This information could be logged or shared with third parties. Mix-net-based anonymity tools address this problem. Figure 2 shows the typical architecture of mix-net-based web-anonymity tools. A mix-net consists of a collection of computers that store-and-forward encrypted messages. Each computer in the mix-net is called a “mix.” The web-browsing client uses a layered encryption technique to construct a web-page request message, suitable for sending over the mix-net. This message is then sent to the first mix in the mix-net. When a mix receives an encrypted message, it decrypts the message. This decryption reveals only the name of the next mix that the decrypted message should be forwarded to. All other information contained in the message is unreadable due to the layered encryption. When the last mix in the mix-net receives a message, it also decrypts it. However, this decryption reveals the URL that the client wishes to retrieve. The last mix then contacts the http://www.ddj.com
appropriate web server to retrieve the document specified by the URL. The retrieved document is sent back to the client over the same mix-net path that the client request took— the last mix sends the document to the next-to-last mix, and so on. The anonymity that can be provided by mix-net-based tools is much stronger than that of proxy-based tools because of the encryption/decryption performed by the client and the mixes. The client encrypts the initial request in such a way that each mix can fully decrypt and interpret only part of the request — the part of the request that specifies the next mix that the request should be forwarded to. Only the first mix in the mix-net learns the IP address of the client and only the last mix in the mix-net learns the URL that the client has requested. As the document makes its way back to the requesting client, each mix in the mix-net encrypts the document, hiding the content from the other mixes. Interfacing with Web Anonymity Tools Although the architecture of the proxy-based and mix-net-based anonymity tools are quite different, they do share an important design characteristic — they both carry HTTP-based messages. HTTP is the protocol that governs how web browsers and web servers talk to each other. All messages sent between browsers and servers must be formatted according to the HTTP specification. All of the web-based anonymity tools we just described speak the HTTP protocol — the tools expect HTTP-formatted messages. Therefore, if you want distributed applications to utilize these anonymity tools, your applications must be capable of sending/receiving HTTP-based messages. Figure 3 shows the HTTP message a web browser might send to Amazon.com to retrieve the file “index.html.” The first line specifies the type of action the browser wishes the server to perform. The word GET is called the HTTP “method.” In the context of HTTP, a method is essentially the name of a command. This HTTP method tells the web server to send the file “index.html.” The second and third line of Figure 3 specify the HTTP headers, which are essentially just a collection of name-value pairs that provide additional information to the recipient of the message. The HTTP header name appears to the left of the colon and the value appears to the right. The server responds with an HTTP-based message similar to that in Figure 4. Every HTTP response begins with a line, called the “status line,” which specifies the result of the corresponding request. Requests can succeed or fail. A numeric code, called the “response code,” specifies the result of the request. In Figure 4, the response code is http://www.ddj.com
200. A set of numeric codes have been defined by the HTTP standard. For example, the response code 200 means that the request succeeded. Immediately following the response code is a text string, called the “reason phrase,” that specifies (in natural language) what the response code means. In Figure 4, the reason phrase is “OK,” meaning that the request succeeded. Immediately following the status line are the HTTP headers, which serve the same purpose as those in the HTTP request message — namely to provide information to the recipient of the message. Following the HTTP response headers is a blank line — more specifically it is a carriage return character followed by a linefeed character (CR/LF). This line marks the end of the HTTP headers and the beginning of the response body. The response body usually contains the item that the browser requested. In Figure 4, the response body contains the HTML code that is stored in the server’s index.html file (the file requested in Figure 3). The size (in bytes) of the response body must be specified in the “Content-Length” HTTP header (Figure 4). The “Content-Type” HTTP header identifies the type of data being stored in the response body. This header essentially tells the browser how to interpret the data that is stored in the response body. The POST method is similar to the GET method in that it lets browsers request content from servers. However, it is typically used when browsers need to send data, usually from an HTML form, to servers. The data to be sent to the server follows the request HTTP headers. This data is referred to as the “request body” and is separated from the HTTP headers by a blank line (more specifically, a CR/LF). The content of the request body is not limited to ASCII text — any sequence of bytes can be sent in the request body. The size (in bytes) of the request body must be specified in the “Content-Length” HTTP header. The HTTP specification defines a number of standard HTTP headers. However, the spec does not exclude the inclusion of additional headers that might be application specific. For example, you can develop a web-based application that requires new HTTP header values. Web servers, browsers, or other programs that read HTTP-based messages are supposed to ignore headers they don’t understand. Ignored headers are simply meant to be passed to the next application that processes the HTTP-based message. As the HTTP message is sent over the Internet from one computer to another, it may gain or lose HTTP headers — the exact action taken depends on the headers and the applications that process them. A header that is unknown to all computers in this forDr. Dobb’s Journal, June 2004
warding chain is simply left untouched as it moves to its destination. You can use this aspect of HTTP to transport applicationspecific messages over web-based anonymity tools. To send messages over web-based anonymity tools, your applications need to send/receive HTTP-based messages. However, we are only interested in a small subset of HTTP— namely, the POST method and a few headers and response codes. Assume you are developing a system that lets students anonymously send/receive reviews of college courses. Clearly, students would be more forthcoming in their reviews if they believed the reviews could be sent anonymously. The system consists of two components: the server that stores the reviews, and the client that can send reviews to the server and retrieve reviews from the server. You want the client to contact the server using a webbased anonymizing tool. Therefore, both
Proxy
1
2
4
3
Client
Server
Figure 1: Retrieving a web page using a proxy. (1) The client sends the requested URL to the proxy. (2) The proxy makes the request on behalf of the client. (3) The server returns the document to the proxy. (4) The proxy returns the document to the client.
7 8
Client
1
6 Mix 2
Mix 1 2
Mix 3 3
4
5 Server
Figure 2: Retrieving a web page using a mix-net. (1) The client sends the layered encrypted URL to mix 1. (2) Mix 1 removes one layer of encryption from the URL and sends it to mix 2. (3) Mix 2 removes the next layer of encryption from the URL it received from mix 1 and sends it to mix 3. (4) Mix 3 removes the next layer of encryption from the URL it received from mix 2. This reveals the client requested URL. Mix 3 contacts the server, specified in the URL, and requests the specified document. (5) The server sends the requested document back to mix 3. (6) Mix 3 sends the document to mix 2. (7) Mix 2 sends the document to mix 1. (8) Mix 1 sends the document to the client. 31
the server and client must send/receive HTTP-based messages. Figure 5 shows the HTTP that the client generates to store a review on the server. The client simply creates an HTTP-based POST message that contains the server’s name (www.mycourseserver.edu) and listening port (4321) in the URL. A colon must separate the server name from the port. The client knows that the server is listening for connections on port 4321. We are not utilizing port 80— the port traditionally utilized by web servers. By spec-
ifying the port in the URL, we can accommodate any number of applications running on the same server, each awaiting connections on a different port. (You may need to use port 80 if the other server ports are blocked by a firewall.) The client generates three HTTP headers. The first two are standard HTTP headers that specify the size and type of the request body. The “Anon-Message-ID” header was created specifically for our application and is not a standard HTTP header. Its purpose is to identify the type of message we
GET http://www.amazon.com/index.html HTTP/1.0 Accept: text/html Referer: http://www.cnn.com
Figure 3: HTTP GET request.
are sending. This would be unnecessary if the client could only send one type of message to the server, but that is rather limiting. The request body consists of a serialized version of the object that stored the course comments. How the comments are stored in the request body is immaterial — they could just as easily be sent as ASCII text. The “Content-Length” header specifies the size, in bytes, of the request body. Therefore, the size of the request body must be computed before the HTTP headers can be generated. The client sends this HTTP-based message to the appropriate web-based anonymizing tool. This is typically done by opening a socket connection to the tool and sending the message. Once the client’s message has been received by the server, via the anonymizing tool, the server parses the HTTP message and prepares a response. Figure 6 shows the server’s HTTP-based response. The first line of the response indicates that the client’s request was successfully processed (response code 200). The HTTP headers indicate the size and type of the response body. We once again utilize the “AnonMessage-ID” header to identify the type of message the server is sending back to the client. The response body holds a serialized version of an object that will be sent back to the client. For example, this object could hold a username and password that clients could later use to update their course review. The response body is optional — it is presented simply to show the HTTP necessary to return an object to clients. The anonymizing tool sends the entire response back to clients. Implementation The Java code we present interfaces with JAP, a freely available mix-based web anonymizing tool developed by Technische Universität Dresden (http://anon .inf.tu- dresden.de/index_en.html). Although we are using Java in our example implementation, you can use any programming language that supports sockets. Since our program will be sending/receiving HTTP-based messages, it should be relatively easy to alter the code so that it can be used with other web-based anonymizing tools. JAP is a free mix-based anonymizing tool that lets you browse the Web anonymously by configuring your browser to use JAP as an HTTP web proxy. Setting a web proxy is a standard option on almost HTTP/1.0 200 OK Content-Length: 20 Content-Type: text/html Hello World
Figure 4: HTTP GET response. 32
Dr. Dobb’s Journal, June 2004
http://www.ddj.com
all web browsers. However, our programs will not be using JAP in this way. Instead, our Java program generates HTTP-based messages and sends these requests to JAP. JAP delivers these messages to recipients, using a mix network, and returns the recipient’s reply. Listing One is used to connect to JAP. This source code assumes that JAP is running on the client (localhost) and that JAP is listening for client connections on port 4000. The port number that JAP listens on can be set using JAP’s user interface. The source code simply creates a socket connection to JAP. Listing Two shows the HTTPMessageHeader class. This class contains two static methods that generate HTTP messages. The prepareRequestMessageHeader method generates the POST method and associated HTTP headers are used when a client is sending a message to a server. The terms “client” and “server” could easily be replaced by “sender” and “receiver,” respectively; the methods described here are meant for peer-to-peer communication. The prepareReplyMessageHeader method is used to generate the status line and HTTP headers needed for a response message. As required by the HTTP specification, CR/LF characters are placed at the end of every line. The size (in bytes) of the request and reply body must be known at the time the headers are created. This is because the “Content-Length” HTTP header appears before the request and reply body.
POST http://www.mycourseserver.edu:4321 HTTP/1.0 Content-Length: 16384 Content-Type: octet/stream Anon-Message-ID: 1 < Serialized ClassReview Object >
Figure 5: Client storage request. Listing Three is the sendMessage method, which sends an HTTP-based message to the recipient via JAP. The code sends a serialized copy of an object of type SendObject to the recipient. First, a socket connection is opened to JAP. Next, the object to be sent to the recipient is serialized — converted to a series of bytes that can be used by the recipient to reconstruct the object. Once the object has been serialized, its size, in bytes, can be determined. The intended recipient of the message is “courseserver.edu.” The port number must be appended to the server’s name — the port number follows the colon. This host and port format is specified by the HTTP standard. Finally, the HTTP-based message is sent to JAP. JAP will read the message and format it for transport over the mix-net. The last mix in the mix-net sends the HTTP-based message to the intended recipient (courseserver.edu, port 1765). The intended recipient of the message (courseserver.edu) must be listening for mix-net connections on port 1765 (based on our example code). Once the mix-net
Listing One
< Serialized Response Object >
Figure 6: Server reply. delivers the message to courseserver.edu, the HTTP-based message must be parsed to determine the message ID and the size of the enclosed object. These items can be found using Java’s regular expression matching library. Given the size of the enclosed object, the recipient can read the serialized bytes and reconstruct the object. The recipient is then free to send a reply to the sender using the HTTPMessageHeader.prepareReplyMessageHeader method (in Listing Two). An example Java program that utilizes these methods is available electronically from DDJ (see “Resource Center, page 3) and at http://home.manhattan.edu/ ~marc.waldman/DDJ_ JavaAnon.zip. DDJ
sb.append(CONTENT_LENGTH); sb.append(contentLength); sb.append(CRLF+CRLF); try{ return ByteBuffer.wrap((sb.toString()).getBytes("US-ASCII")); } catch(UnsupportedEncodingException e){ return null; }
SocketChannel japConnection; try{ japConnection=SocketChannel.open(); japConnection.connect(new InetSocketAddress("localhost",4000)); } catch(Exception e){ System.err.println("Unable to connect to JAP - exiting"); System.exit(1); }
} }
Listing Three
Listing Two public class HTTPMessageHeader{ private static final String CONTENT_LENGTH="Content-Length: "; private static final String MESSAGE_ID="Anon-Message-ID: "; private static final String CRLF="\r\n"; private static final String OCTET_CONTENT="Content-Type: octet/stream"; private static final String HTTP_OK_RESPONSE="HTTP/1.0 200 OK"; public static ByteBuffer prepareRequestMessageHeader(String recipient, int messageID, int contentLength){ StringBuffer sb=new StringBuffer("POST http://"); sb.append(recipient); // recipient string should include a colon and port number sb.append(" HTTP/1.0"+CRLF); sb.append(CONTENT_LENGTH+contentLength+CRLF); sb.append(OCTET_CONTENT+CRLF); sb.append(MESSAGE_ID+messageID); sb.append(CRLF+CRLF); try{ return ByteBuffer.wrap((sb.toString()).getBytes("US-ASCII")); } catch(Exception e){ return null; } } public static ByteBuffer prepareReplyMessageHeader(int messageID, int contentLength){ StringBuffer sb=new StringBuffer(HTTP_OK_RESPONSE+CRLF); sb.append(OCTET_CONTENT+CRLF); sb.append(MESSAGE_ID); sb.append(messageID+CRLF);
http://www.ddj.com
HTTP/1.1 200 OK Content-Length: 6144 Content-Type: octet/stream Anon-Message-ID: 2
public void sendMessage(Object sendObject) throws Exception{ String recipient="courseserver.edu:1765"; SocketChannel japConnection=null; int messageID=1; // connect to JAP try{ japConnection=SocketChannel.open(); japConnection.connect(new InetSocketAddress("localhost",4000)); } catch(Exception e){ System.err.println("Unable to connect to JAP - exiting"); System.exit(1); } // serialize sendObject ByteArrayOutputStream baos=new ByteArrayOutputStream(); ObjectOutputStream oos=new ObjectOutputStream(baos); oos.writeObject(sendObject); oos.close(); ByteBuffer bb=HTTPMessageHeader.prepareRequestMessageHeader(recipient, messageID, baos.size()); if (bb==null){ System.err.println("Unable to create HTTP Header"); System.exit(1); } ByteBuffer objectBB=ByteBuffer.wrap(baos.toByteArray()); ByteBuffer bbArray[]=new ByteBuffer[2]; bbArray[0]=bb; bbArray[1]=objectBB; // send the header and object to JAP japConnection.write(bbArray); }
Dr. Dobb’s Journal, June 2004
DDJ
33
The OpenCable Application Platform Transitioning to open, standards-based architectures Linden deCarmo
F
or years, North American cable operators have been enslaved by closed networks and proprietary APIs. Fortunately, U.S. cable vendors believe that the OpenCable Application Platform (OCAP) will emancipate them from their dependence on closed technology. OCAP not only leverages Java-based APIs and uses open networking standards, but also promises to foster competition and reduce costs. In this article, I examine why OCAP is a massive improvement over proprietary solutions and delve into the OCAP architecture. In a follow-up article, I’ll show how you can use Java to exploit the capabilities of OCAP-based set-top boxes that soon will be connected to High-Definition TVs.
from either General Instruments (later bought by Motorola) or Scientific Atlanta because these products were the quickest means of completing this transition. Once deployed, these networks became so entrenched that it is now very costly for cable providers to switch to open architectures. There are two core aspects to any closed cable solution — Conditional Access (CA) and software APIs. CA is the technology that encrypts and secures content. DigiCipher II is the CA for Motorola platforms
Java Application MHP Middleware Set-Top Box Hardware
Figure 1: MHP middleware that lets Java applications be written for cable and satellite markets.
A Sad State of Affairs In this age of open-source software, you’d probably be shocked to hear that the U.S. cable market is mired in a proprietary quagmire. For a variety of reasons, the closed nature of U.S. cable platforms arose by necessity, not necessarily by the design. To explain: Cable vendors were eager to transition from analog to digital networks. As a result, they selected proprietary solutions
and PowerKEY is used in Scientific Atlanta environments. Given that such technology is constantly under siege by pirates, it is doubtful that CA technology will ever be based on an open standard. Besides CA, closed solutions utilize proprietary APIs that are tailored for the embedded cable market and tightly coupled to the CA. While this may result in efficient software, the development environment is breathtakingly expensive. Consequently, there aren’t hordes of software vendors competing in this arena.
Linden is Consultant Engineer at Pace Micro Technology Americas and the author of Core Java Media Framework (Prentice-Hall, 1999). He can be contacted at
[email protected].
Europe to the Rescue? While the American cable market was entangled in a web of proprietary solutions, Europeans developed ambitious plans to incorporate Internet and Java-based stan-
34
Dr. Dobb’s Journal, June 2004
dards into their digital television products for cable, satellite, and terrestrial (off-air) markets. This effort, called the “Multimedia Home Platform” (MHP), was spearheaded by the Digital Video Broadcasting Project and eventually adopted as a standard by the European Telecommunications Standards Institute. The core of MHP is a Java Virtual Machine and Java-based subsystems such as AWT, JavaTV, and Java Media Framework. These Java APIs leverage MPEG-2 for digital video and Digital Storage Media for Command and Control for file I/O, plus they offer custom interfaces for the interactive TV market. Since the Java APIs in MHP let applications access low-level multimedia drivers in a device-independent manner, it is categorized as “middleware” (that is, software that acts as a conduit to digital video features and functionality); see Figure 1. Although MHP middleware was oozing with potential, skeptics noted that the processing requirements of the platform would make the machines too expensive to deploy. However, while MHP machines may not have been cost-effective in 2001, these concerns have evaporated due to the stunning advances in embedded hardware. Today’s 32-bit embedded processors not only offer more MIPS, but are capable of running advanced operating systems (such as Linux). Mayflower Redux During this same timeframe, CableLabs (a consortium sponsored by leading American cable companies; see http://www .cablelabs.com/) was given the charter to create OpenCable. This charter is threefold: • Define the feature set of next-generation hardware. • Foster competition based on open standards. http://www.ddj.com
• Enable boxes to be sold at retail locations. To achieve these goals, CableLabs published a series of specifications. While most of these specifications are U.S. centric, CableLabs realized that most MHPs could be reused in OCAP’s mission to provide a middleware API that is operating system, hardware, and network neutral. Like MHP, OCAP revolves around a JVM and Java APIs. However, many Java classes have been enhanced with U.S.-specific functionality. The most noticeable architectural difference between MHP and OCAP is the Conditional Access infrastructure. In OpenCable, CA is done via the CableCARD. Traditional CA technologies like PowerKey and Digicipher II are closely integrated with cable set-top boxes for security purposes. As a result, it’s impossible to deploy a Scientific Atlanta set-top box on a Motorola network. By contrast, each CableCARD is a self-contained Conditional Access subsystem in a PCMCIA form factor — you simply insert it into the cable card slot in the set-top box and the CableCARD handles CA processing. Since CableCARD-compatible set-tops aren’t bound to a proprietary CA technology, these set-top boxes can be sold anywhere in the U.S.— one of the major goals of OCAP/OpenCable. Hallway Monitor OCAP also differs from MHP in its use of a monitor application. In MHP, applications are launched and terminated based on the services (or network video features) that they are manipulating. If multiple applications try to simultaneously access a resource like a video tuner, MHP uses a priority-based arbitration scheme to resolve the conflict (the resource is granted to the higher priority application). Due to the complexities of the American cable environment, OCAP has enhanced MHP’s priority-based approach with a monitor. This monitor architecture consists of two layers (see Figure 2). The lower layer contains privileged APIs to handle resource conflicts, respond to emergency messages, and manipulate the CableCARD. The upper portion of the architecture is the monitor application and other applications that have been authorized to access monitor functionality. This application is specific to the Multiple Service Operator (MSO) and tailored to its particular requirements (MSO roughly equates to a cable company). For instance, one cable vendor’s monitor application may grant video-on-demand applications higher priority when resource conflicts occur because video-on-demand apps generate more revenue for that MSO. By contrast, another MSO may grant preferential treathttp://www.ddj.com
ment to interactive games because their clientele demands it. If the MSO declines to write a custom monitor application, the lower-level privileged APIs default to a priority-based resource management scheme. However, this default scheme may not be adequate for the intricate environment that a particular MSO must serve. For instance, FCC Rule 47 requires that captions and emergency alerts always be visible and not overlap each other. A simple MHP-like priority-based resource sharing algorithm may not comply with such rigorous requirements. Another reason to implement a monitor is to prevent rogue applications from running on the MSO network. A robust monitor rejects attempts by intruders to load unauthorized applications onto the set-top box. While this has the side effect of preventing you from trying your favorite Java apps on your cable set-top box, it stops
intruders from creating havoc on specific boxes or even bringing down the network. Ties that Bind A third fundamental difference between OCAP and MHP involves application capabilities. Java applications that conform to MHP requirements are referred to as “MHP-J,” whereas OCAP-compliant applications are labeled “OCAP-J.” Both middleware platforms have rigorous certification requirements and are finicky about which applications are considered compliant (for instance, while your PCbased Java Media Framework application may theoretically run on a particular OCAP middleware platform, it probably uses classes and interfaces not available on all OCAP systems and, therefore, can’t earn the OCAP-J moniker). In MHP, applications are bound to a particular channel (or service). A service describes content available for a particular
Monitor Application
Monitor Class Application Priviledged APIs
CableCARD
EAS
Captions
Figure 2: OCAP monitor applications use privileged APIs to arbitrate resource conflicts, handle emergency alerts, and control other critical system resources. Field
Number of Bits
application_information_section() { table_id section_syntax_indicator reserved_future_use reserved section_length application_type
8 1 1 2 12 16
reserved version_number
2 5
current_next_indicator section_number last_section_number reserved_future_use common_descriptors_length
1 8 8 4 12
for(i=0;i
