No title

GLOSSARY © 2006 Nature Publishing Group http://www.nature.com/naturebiotechnology A note on nomenclature Laura H Reid ...

Author: Andrew Marshall

6 downloads 1532 Views 28MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

GLOSSARY

© 2006 Nature Publishing Group http://www.nature.com/naturebiotechnology

A note on nomenclature Laura H Reid & Janet A Warrington The following glossary defines key terms and concepts that are used throughout the Microarray Quality Control (MAQC) consortium manuscripts. Wherever possible, the definitions are based on the Clinical and Laboratory Standards Institute harmonized terminology database (http://www.clsi.org). Detection call. A qualitative value that suggests a level of confidence in the signal calculated for that probe. In the MAQC study, the detection calls were binary and reduced to either ‘0’ for ‘not detected’ or ‘1’ for ‘detected’. For some platforms, the detection call reflects the quality of the nucleic acid spot on the microarray, similar to ‘Flag/No Flag’ scores. On other platforms, the detection call reflects the abundance of the target transcript or the concordance of results between multiple probes in a probe set, similar to ‘Absent/Present’ calls. Although the final detection call is qualitative, it is usually based on quantitative assessments and complex statistics. External RNA control. An RNA species added to a biological sample during processing for the purpose of assessing technical performance of a gene expression assay. Different external RNA controls may be used to monitor different processes. In microarray research, external RNA controls are added either to a total RNA sample (to assess the enzymatic processes involved and the hybridization step) or to the labeled cRNA (to assess hybridization efficiencies only). Gene. An expanded definition of this term was adopted by the MAQC consortium to denote both a DNA segment and the collection of RNA transcripts derived from it. In the DNA usage, a gene is a locatable region of genomic sequence, corresponding to

Laura H. Reid, Expression Analysis, Inc., 2605 Meridian Parkway, Durham, North Carolina 27713, USA. Janet A. Warrington, Affymetrix, Inc., 3420 Central Expressway, Santa Clara, CA 95051, USA

ii

a unit of inheritance, which is associated with regulatory regions, transcribed regions and/or other functional sequence regions. In the RNA usage, a gene often refers to the targets measured in a gene expression assay. Probe. A discrete piece of nucleic acid used to identify specific DNA or RNA molecules bearing the complementary sequence. Some microarray platforms rely on a single oligonucleotide probe to assay an RNA target; others combine data from multiple probes, arranged in a probe set, when calculating expression values for a target. Bead-based assays attach oligonucleotide probes to a microscopic bead surface. PCR-based assays use a pair of oligonucleotide primers (also referred to here as probes) to identify and amplify their intended RNA target, and in some cases, an oligonucleotide detection probe is hybridized to the amplified target. Repeatability. The ability to provide closely similar results from replicate samples processed in parallel at the same test site using the same gene expression assay. Reproducibility. The ability to provide closely similar results from replicate samples processed with different microarray platforms or at different test sites using the same gene expression assay. Signal. The quantitative expression value for each probe derived from a hybridization image after preprocessing steps, such as background subtraction and summarizing of data from multiple probes, as well as normalization procedures that remove systematic artifacts. Signals are not the raw fluorescence or chemiluminescence intensities captured in a pixelated microarray image. Target. Nucleic acid whose identity and/or abundance is revealed during the assay. The gene expression assays in the MAQC study have RNA targets. Multiple RNA targets can be transcribed from a single gene and individual transcripts can be alternatively spliced into multiple targets with different functions and expression patterns. Thus, a gene expression assay designed for one target may actually detect multiple RNA transcripts.

VOLUME 24 NUMBER 9 SEPTEMBER 2006 NATURE BIOTECHNOLOGY


IN THIS ISSUE

MicroArray Quality Control project Since 2004, when the US Food and Drug Administration (FDA; Rockville, MD) started accepting voluntary genomic data submissions, the number and scope of DNA microarray–based expression data analyses filed as accompanying (but non-binding) information with new drug applications has been steadily increasing. And although the potential value of the information contained in these submissions is undisputed, no clear guidelines and standards have as yet been established for their use as part of a regulatory decision-making process [Foreword, p. 1103; Commentary, p. 1105]. But human healthcare is not the only area where microarrays represent a promising technology; environmental monitoring of pollutants through toxicogenomics, for instance, could also greatly benefit from their adoption. In a similar manner to their use in drug monitoring, microarrays could also be applied to detect early, subchronic exposure to pollutants using model systems or, at the very least, to characterize some of the underlying molecular mechanisms in toxicity [Commentary, p. 1108]. The practical challenges in implementing microarrays for the above applications will not be trivial, however. To translate the outcome of microarray analyses into the clinical and regulatory realms, many questions regarding sensitivity, reproducibility and ultimately biological significance remain to be answered [Commentary, p. 1112]. It is in this context that the MicroArray Quality Control (MAQC) project was conceived by a group of regulatory, academic and industrial partners to comprehensively tackle some of the technical issues surrounding the robustness and comparability of some of the most widely used microarray platforms. Starting with two well-defined, commercially available RNA samples, this consortium has carried out a sideby-side evaluation of seven different platforms with the aim of establishing a series of metrics that would facilitate future standardization approaches [Article, p. 1151]. To validate that microarray data are comparable to data obtained from other, more traditional gene expression assays, the MAQC data set was also assessed against three quantitative molecular assays for measuring gene transcription; and it turns out that the overlap is encouragingly high [Analysis, p. 1115]. Another important question to be addressed by the MAQC consortium was the use of RNA aliquots, external to the actual samples, that can serve as internal, technical controls for evaluating the level of performance at different steps of the experimental protocol, from reverse transcription to labeling of the samples [Analysis, p. 1132]. If adopted widely by the community, these and similar external RNA controls could provide researchers with a qualitative assessment of their assay’s performance. In a separate experiment, the consortium also put the quantification capabilities of the different platforms to test. Using a series of titration samples, good concordance of predicted and actual measurements was reported across platforms [Analysis, p. 1123]. In the early days of microarrays, two-color detection protocols were often preferred to those using one-color labeling of RNA because they could compensate for some of the imperfections and inaccuracies in microarray probe spotting. However, with improvements in microarray manufacture, the performance of one-color versus two-color platforms is becoming a central question for high-volume data generation with microarrays, in that robust and reliable single-color protocols would greatly facilitate implementation, and reduce the cost, of analyses [Analysis, p. 1140]. In a final report, the MAQC group applies their approach to real-world toxicogenomic analysis of rats exposed to three plant-derived carcinogenic compounds, aristolochic acid, riddelliine and comfrey. Again, the results across platforms showed high accuracy, reproducibility and biological relevance [Article, p. 1162]. AM & GTO

Next month in • Activated sludge metagenomics • Genome of a bioplastic producer • Knock-ins for knockout anti-inflammatory mAbs • Arrested protein chip fabrication • High-definition microarray for DNA binding site searches

In This Issue written by Michael Francisco, Peter Hare, Sabine Louët, Andrew Marshall, Gaspar Taroncher-Oldenburg & Jan-Willem Theunissen.

NATURE BIOTECHNOLOGY VOLUME 24 NUMBER 9 SEPTEMBER 2006

Patent roundup • Timothy Caulfield and colleagues report that policy makers may respond more to media controversies than systematic data on gene patenting. [Patent Article, p. 1091] MF • A US federal appeals court ruled on August 3 that Cambridgebased Transkaryotic Therapies (TKT), acquired last year by Shire, has infringed two patents held by Amgen for the production of erythropoietin. [News in Brief, p.1048] SL • Recent patent applications in tissue engineering. [New Patents, p. 1095]

MF

vii


EDITORIAL

Making the most of microarrays A major, multicenter study of microarray performance is a first step in translating the technology from bench to bedside.

N

o technology embodies the rise of ‘omic’ science more than the DNA microarray. First reduced to practice in the early 1990s, it has since undergone numerous iterations, adaptations and refinements to achieve its present status as the platform of choice for massively parallel gene expression profiling. Today, several thousand papers describing data from microarrays are published each year. Sales of arrayers, array scanners and microarray kits to the academic and industrial R&D community represent a multi-billion-dollar business. The microarray has even made its first forays into the clinic, with the US Food and Drug Administration’s approval of the ‘AmpliChip’ to help physicians tailor patient dosages of drugs that are metabolized differentially by cytochrome P450 enzyme variants. And yet doubts linger about the reproducibility of microarray experiments at different sites, the comparability of results on different platforms and even the variability of microarray results in the same laboratory. After 15 years of research and development, broad consensus is still lacking concerning best practice not only for experimental design and sample preparation, but also for data acquisition, statistical analysis and interpretation. Though problematic for bench research, lack of resolution of these issues continues to even more seriously hamper translation of microarray technology into the regulatory and clinical settings. Indeed, several regulatory authorities have been wrestling with the problem of how and when (and indeed whether) to implement microarray expression profiling data as part of their decision-making processes. The move in the past two years to accept voluntary genomic data submissions by regulatory agencies overseeing human and environmental safety was the first in a long series of steps that will be needed. One of the next steps can be found in this issue, which presents the first formal results of the MicroArray Quality Control (MAQC) Consortium—an unprecedented, community-wide effort, spearheaded by FDA scientists, that seeks to experimentally address the key issues surrounding the reliability of DNA microarray data. MAQC brings together more than a hundred researchers at 51 academic, government and commercial institutions to assess the performance of seven microarray platforms in profiling the expression of two commercially available RNA sample types. Results are compared not only at different locations and between different microarray formats but also in relation to three more traditional quantitative gene expression assays. Although the direct comparison of microarray platforms and the establishment of common controls for microarray experiments is nothing new—several cross-format studies have already been published, and other groups, such as the External RNA Controls Consortium’s (ERCC), are developing standardized RNA controls—it is the size and comprehensiveness of the data set generated by the MAQC effort that is unique. In the main study, ~60 hybridizations were carried out on each of the seven platforms; >1,300 microarrays were used during the entire project.


MAQC’s main conclusions confirm that, with careful experimental design and appropriate data transformation and analysis, microarray data can indeed be reproducible and comparable among different formats and laboratories, irrespective of sample labeling format. The data also demonstrate that fold change results from microarray experiments correlate closely with results from assays like quantitative reverse transcription PCR. The levels of variation observed between microarray runs by MAQC were relatively low and largely attributable to cross-platform differences in probe binding to alternatively spliced transcripts or to transcripts that show a high degree of cross-hybridization to probes other than their own. Thus, although factors as diverse as day-to-day fluctuations in atmospheric ozone levels (which effect cyanidine 5 fluorescence), nuclease levels in sample tissues and the quality of microarray production between batches have all been cited as influencing array performance, on the basis of the data presented here, experimental variability appears manageable. Another clear finding is that the days of the simple two-sample t-test as a means of ranking differentially expressed genes are surely numbered. A key take-home message is that statistical analysis in regulatory submissions and clinical diagnostics is likely to be different from that used in basic research and discovery. In the case of the MAQC study—where the goal was to optimize intra- and inter-platform reproducibility—the approach was to limit the number of transcripts identified and to sort differentially expressed genes using fold-change ranking with a nonstringent P-value cutoff. But for experiments that seek to identify differentially expressed transcripts at or near the lower limits of detection, this tradeoff between reproducibility on the one hand and precision and sensitivity on the other is likely to shift, and a different type of statistical analysis will be required. There is no one-size-fits-all statistical solution. Overall, the MAQC study represents a landmark in DNA microarray research because it provides the community with a thoroughly characterized reference data set against which new refinements in platforms and probe sets can be compared. It complements other initiatives, such as the ERCC, in providing the community with two commercially available human reference RNA samples that can be used to calibrate arrays in ongoing quality control and performance validation efforts. It can be used as the foundation for combining other microarray studies, thereby realizing the true cumulative potential of microarray data, which will undoubtedly lead to new insights. And from a clinical perspective, it validates the DNA microarray as a tool that is sufficiently robust and reliable to be embraced for use on hard-to-obtain human tissue samples. Clearly, microarrays have a long way to go before they can be used to support regulatory decision-making or accurate and consistent prediction of patient outcomes in the clinic. But the MAQC study has given us a solid foundation from which to build.

1039


EDITORIAL

Can Europe accelerate out of trouble? Europe should seriously consider the ‘accelerator’ concept to foster the sustainability of its biotech companies.

M

any of Europe’s biotech firms appear permanently stuck in a state of arrested development. Indeed, compared with their counterparts over the ocean, European startup companies continue to find it hard to achieve the size and stature requisite for commercial success. A recent report reveals that compared with the United States, Europe has an awful lot of small companies that, on average, grow much slower than their US counterparts. But help may be at hand in the form of a new incubator concept pioneered on the West Coast of the United States. According to the latest report from EuropaBio and consultants Critical I, Biotech in Europe: 2006 Comparative Study, two-thirds of European companies have under 20 employees, whereas two-thirds of US companies employ more than 20 people. One would expect that companies established for, say, 2 years or less would be small, and the report confirms this, both for Europe and for the United States. In the United States, however, the initial phase of company growth is rapid: by the time US companies are 5 years old, 75% of them have more than 20 employees; in Europe, by contrast, companies employing ‘less than 20 employees’ are the largest group, right up until the firms are beyond 15 years old. One reason that US biotech fares better is that US entrepreneurs and investors continue to look for ways of growing companies more efficiently. One of the models that is growing in popularity is the accelerator. Like incubators, accelerators provide customizable laboratory and business space for young companies. Unlike incubators, which bring small chunks of fluffy capital, cramped facilities and low-grade access to a centralized team of distracted and generically qualified management mentors, accelerators provide a combination of concentrated capital overlaid with specific and committed technical, clinical or market expertise. The availability of greater amounts of seed and startup cash (on the order of ~$4 million per company) certainly reduces one of the major risks that young companies face, and by favoring companies that are past the point of discovery, accelerators certainly cut out a large chunk of technology risk. However, accelerators endeavor to take risk reduction even further. Consider, for instance, the eponymous Seattle, Washington–based Accelerator, started in 2003. Leroy Hood of the Institute for Systems Biology is Accelerator’s president (p. 1055), and Amgen’s venture fund is a founding partner. That gives companies backed by Accelerator (five, so far) instant access to world-class understanding of technology and market issue. Through its founders and management, Accelerator has close ties to several of the Pacific Northwest’s (and America’s) leading venture capital firms, such as MPM, Versant and ARCH. Although Accelerator backs companies addressing various slow steps in the healthcare product development process, other accelerators focus on particular areas of clinical practice. One of the most highly focused is the Hackensack, New Jersey, firm Advanced Technologies,

1040

which has started or re-started six companies that are each developing medical devices for interventional cardiology products. The team running Advanced Technologies includes seasoned investors, cardiologists and clinicians, all of whom have clear roles to play in speeding up the development, clinical adoption and commercialization of cardiovascular devices and hence in providing expedited investment and business exits. More accelerators are on the way. A consortium of large pharmaceutical firms is said to be considering creating one in the Cambridge, Massachusetts biotech cluster. And another may be built in the San Diego biotech cluster. Oddly, just as accelerators are finding new ways to make the milieu for new US firms more encouraging and less risky, the opposite may be true in Europe. In the United States and the more advanced parts of Europe, the rate of formation of new companies has slowed in recent years. Consequently, a large proportion of the new European foundlings are arising in nations or regions that are themselves new to biotech. Often, there are precious few biotech-relevant resources in these locations, beyond a bit of seed money: there are no substantial finance streams, no management skills, no biotech-experienced support infrastructure of lawyers, accountants and consultants. Such environments are precisely the opposite of accelerators, and are likely to have precisely the opposite effect. Global competition and technology supercession means that biotech firms need to have a ‘Red Queen’ mentality. But trying to ‘run as fast as you can just to stay still’ is difficult if you are wading through mud. The lesson for companies in nations with new, fledgling biotech sectors is that they need to reach out beyond national borders to management and financiers in other, more established biotech clusters. It’s important to work with these experienced executives and investors because they are familiar with the idiosyncrasies and protracted timelines of life science ventures and they have the requisite historical and international perspective to place new biotech platforms or products in their proper global competitive context. In this respect, the Accelerator model looks particularly interesting. Given the difficulty of pooling investors and management expertise and the relative scarcity of truly globally competitive ventures emerging at the national level, perhaps a pan-European accelerator could be an effective approach. Certainly, if European centers of scientific excellence don’t want much of their first-class intellectual property to be hamstrung by underfunding, naive management and unsupportive surroundings, they should seriously consider the concept. Europe doesn’t need more biotech ventures; it needs more successful ones. And starting biotech accelerators would be one means of bringing together the sort of expertise and funding that could increase the chances that that would happen.


NEWS

Companies eye slice of age-related macular degeneration market Genentech’s new antibody therapy Lucentis (ranimizumab), approved in June, has the potential to dominate the market for a common eye disease. In a few years, however, it may face direct competition from Avastin (bevacizumab), a sister drug made by the same company. Meanwhile, OSI Pharmaceuticals and QLT, which already have drugs approved for the same eye disease, are trying to consolidate their market positions. As yet, several other potential competitors with drugs in phase 2 development for the same indication have not demonstrated any advantages over Lucentis. On June 30 the US Food and Drug Administration (FDA) announced it had approved Lucentis, a treatment for wet agerelated macular degeneration (AMD). The drug is administered as an injection into the eye and is a humanized antibody FabV2 fragment that targets vascular endothelial growth factor (VEGF), a protein associated with growth and leakage of blood vessels that causes vision to decline. Lucentis, which is made by S. San Franciscobased Genentech, is the first approved treatment to restore sight in a significant percentage of patients afflicted with the disease. AMD is a major cause of blindness in people over 50 years old. Until now, drugs approved for AMD could only slow the progression of the disease, rather

Scripps Howard Photo Service/John Rottet/ Raleigh News & Observer/NewsCom


ALSO IN THIS SECTION New clinical trials policy at FDA p1043 Amgen’s TPO mimic faces stiff competition p1044 BioXell: an Italian biotech success story? p1045 News in brief p1048 Profile: Abe Abuchowsky p1050

An eye surgeon performs microsurgery on a patient with age-related macular degeneration. New drugs, such as Lucentis, could reduce the need for such procedures.

than reverse it. But in phase 3 clinical studies of Lucentis, vision improved in more than one third of the individuals who took it. Experts say Lucentis will likely steal a significant market share from the two AMD treatments on the market: Pfizer/OSI Pharmaceuticals’ aptamer Macugen (pegaptanib) and Novartis/ QLT’s small molecule Visudyne (verteporfin). Lucentis may also discourage companies from

developing new therapies. “The success of Lucentis has raised the bar so much that it makes it difficult to come up with a drug that’s better,” says Julia Haller, a professor of ophthalmology at Johns Hopkins University in Baltimore. But physicians are discovering on their own that Genentech’s approved cancer drug, Avastin, an anti-angiogenic antibody that binds VEGF, may work for AMD just as well and just as safely

Table 1 Drugs currently in development for the treatment of wet AMD Product

Company

Mechanism of action

Phase

Evizon (squalamine)

Genaera (Plymouth Meeting, Pennsylvania)

Anti-angiogenic; inhibits VEGF, PDGFβ, thrombin & bFGF intracellular pathways

3

PTK787 (vatalanib)

Novartis (Basel,)/Schering (Berlin)

Small-molecule VEGFR kinase inhibitor

3 (cancer) 2 (AMD)

Retaane (anecortave acetate)

Alcon (Fort Worth, Texas)

Small molecule angiostatic cortisene; inhibits angiogenesis induced by basic fibroblast growth factor, VEGF and other known stimulators

3 (application withdrawn from EMEA after asked for more data; awaiting FDA approval)

AG-13958

Pfizer (New York)

Inhibits tyrosine kinases, including VEGF

2

CAND5

Acuity Pharmaceuticals (Philadelphia, Pennsylvania)

Gene silencing siRNA therapy that reduces production of VEGF

2

Combretastatin A4 Prodrug (combretastatin)

OXiGENE (Watertown, Massachusetts)

Tubulin inhibitor; disrupts the structure of endothelial cells lining the tumor vasculature to stop flow of blood and nutrients to tumor

2

VEGF Trap

Regeneron Pharmaceuticals (Tarrytown, New York)

Recombinant decoy receptor fusion protein that binds to all forms of VEGF-A and placental growth factor

2

Source: Evaluate Pharma (http://www.evaluatepharma.com/) and company websites and information. PDGFb, platelet-derived growth factor b; VEGFR, VEGF receptor. EW


1041


NEWS as Lucentis —and for a fraction of the price. Industry insiders say that if independent investigators complete enough safety and efficacy studies, off-label Avastin prescribed for AMD may become Lucentis’ greatest competitor (Box 1). However, analysts such as Joshua Schimmer, at SG Cowen in New York, believe Lucentis will likely capture 65% of the $1-billion US market. Indeed, Lucentis’ arrival on the market will be a major blow to Macugen, which was developed by OSI Pharmaceuticals and is marketed by Pfizer. Researchers say the difference between the drugs may be the amount of VEGF each drug inhibits. Although Lucentis can bind and inhibit all the active molecular forms of VEGF, Macugen binds to only one form of VEGF, called VEGF-165.“Macugen is effective but not as effective as Lucentis,” says Tony Adamis, chief scientific officer of OSI. “Lucentis’ data is so impressive…even I have to admit that.” In August, OSI announced that due to competition, it would suspend or curtail all R&D for eye diseases, except on Macugen. OSI’s Adamis says he believes Macugen could find a spot on the market as a maintenance drug—something individuals can take after they’ve reaped the benefits of Lucentis. Based on their interpretation of Genentech’s data, OSI theorizes that patients could take Lucentis for their first three doses, and then switch to Macugen, which costs about half as much, and, on average, is injected less frequently. The company has set up a clinical trial to test this proposal. Many experts, however, say OSI’s theory is based on little or no data, and is a last attempt to salvage their product. “They are trying to use a marketing ploy to feed off ophthalmologists who are not in the know,” says Peter Campochiaro, a professor of ophthalmology at the Wilmer Eye Institute at Johns Hopkins. “They’ve been pretty shameless.” Lucentis’ other approved predecessor, Visudyne is a treatment called photodynamic therapy, in which the drug is injected into the bloodstream and activated in the eye by a light beam. Like Macugen, it can slow the progression of AMD, but usually cannot reverse it. “It’s hard to see it playing a meaningful role going forward,” says Schimmer. Visudyne may be useful for individuals who cannot endure an injection in the eye, he says. It may also be used as a combination treatment with Lucentis, although data so far have not supported this treatment regimen, he says. As Lucentis edges out drugs already on the market, it may also douse enthusiasm and funding for some early-stage AMD candidates. Just two months after Genentech announced in July 2005 its phase 3 results for Lucentis, Alnylam Pharmaceuticals, a biotech

1042

Box 1 Avastin could become Lucentis’ greatest competitor With little immediate competition from candidates in the development pipeline, Lucentis’ greatest competitor may be its sister drug, Avastin. The drug stems from the same murine monoclonal antibody as Lucentis. Avastin, however, is a full-length antibody, whereas Lucentis is an antibody fragment. Avastin is also designed as an intravenous drug and has a longer half-life than Lucentis. Genentech scientists say these components make Lucentis better tailored for the eye, with less chance of inflammation and better binding with VEGF. When a standard vile of Avastin is split into eye-sized doses, the drug costs less than $50 per injection compared to the nearly $2,000 per dose estimated for Lucentis. In anecdotal reports and small independent studies of Avastin used off-label for AMD, the drug appears safe and effective, and word has spread in the ophthalmology community. “Reports have been small and anecdotal,” says Jeffrey Heier, a vitreoretinal specialist at Ophthalmic Consultants of Boston. “But all of us have used it on enough patients to know that the results are real.” Genentech, however, decided in the late 1990s to stop pursuing Avastin as an AMD drug. In response, some clinicians are trying to organize their own large-scale study so the doctors will have more concrete data on which to base their prescription choices. EW

Year 1989

Discovery/event • Napoleone Ferrara at Genentech discovers and clones VEGF

1993

• Ferrara and colleagues published pre clinical data showing that an anti-VEGF antibody can suppress tumor growth and angiogenesis—the formation of new blood vessels (Nature 362, 841–844, 1993)

1994

• Studies suggest VEGF may have a role in ocular diseases (NEJM 331, 1480–1487, 1994; Am. J. of Opthalmol. 118, 445–450, 1994)

1996

• Adamis and other Researchers at Massachusetts Eye and Ear Infirmary in Boston discover that a mouse monoclonal antibody against VEGF could be injected into monkey eyes to prevent blood vessels from growing (Arch. Ophthalmol. 114, 66–71, 1996). The cross-species experiment didn’t cause inflammation, suggesting that a humanized version may not cause inflammation if injected into human eyes.

1996

• Genentech humanizes the anti-VEGF antibody

1997 1997

• Phase 1 trials begin for Avastin, a full-length monoclonal antibody targeting VEGF • Genentech compares full-length anti-VEGF antibodies with antibody fragments (Fab) and finds that the fragments better penetrate the retina (Toxicol. Pathol. 27, 536–544, 1999). Their findings compel the company to steer Avastin down a cancer pipeline, and develop a new therapy—Lucentis—for the eye. Researchers later suggest that the study was flawed. “While the Fab appeared to penetrate better than the full-length antibody, the study was flawed due to the fact that the two molecules recognized different antigens: the Fab was directed against VEGF, and the full-length antibody was directed against an antigen expressed within the inner retina known as HER2,” writes Philip Rosenfeld, an ophthalmologist at the University of Miami’s Bascom Palmer Eye Institute, in a 2006 issue of Ophthalmology.

1999

• Phase 1a trial begins for Lucentis, an antibody fragment targeting VEGF made from the same murine monoclonal antibody as Avastin

2004

• FDA approves Avastin for metastatic cancer of the colon or rectum

2005

• Stephan Michels and his colleagues suggested in a small study that Avastin is safe and can improve macular anatomy and vision in people with wet AMD (Ophthalmol. 112,1035–1047, 2005) • Small studies and anecdotal reports conducted by clinicians support Michels’ findings

2005 2006

• Ziad Bashshur and colleagues at the American University of Beirut Medical Center in Lebanon publish the first prospective study of Avastin for AMD. (Am. J. Ophthalmol. 142, 1–9, 2006). Conducted in Lebanon on 17 human subjects, the study found marked improvement in nearly every eye studied, with no side effects.

2006

• FDA approves Lucentis for wet AMD

2006

• Clinicians vow to conduct a large-scale US clinical study of Avastin

EW

Lucentis timeline: the evolution of two anti-VEGF drugs under one roof.

company in Boston, announced that it would halt development of its AMD drug because of competition. But others persevere. Among the most promising candidates in the development pipeline, some experts say, is the VEGF Trap by Regeneron Pharmaceuticals. Scientists believe the drug works by binding more effectively with VEGF, thereby blocking VEGF receptors. A phase 1 study showed that a single injection lasted at least six weeks. Some companies are exploring drug candidates that can be delivered systemically, or into

the bloodstream (Table 1). Although Lucentis is one step ahead of competitors, it has some drawbacks, and industry insiders say there is still some room on the market for new products. Eye injections are rough on patients and carry risk. Lucentis must be injected into the eye every month for the first four months, and then at varying frequency afterward. A drug that lasts longer or can be administered less invasively and less frequently than Lucentis has potential. Emily Waltz, New York


NEWS

In a bid to speed drug development, the US Food and Drug Administration (FDA) is encouraging drug companies to design clinical trials with flexible enrollment, dosing and other parameters. Called ‘adaptive design,’ the approach promises quicker results with smaller trials, but also carries risks of manipulation, according to observers. At a July meeting in Washington, DC, FDA deputy commissioner for medical and scientific affairs Scott Gottlieb, laid out the agency’s plan to develop five guidance papers over the next several years. Although not binding, the documents will help drug companies design and implement adaptive trials that the FDA considers up to snuff. “We have a dilemma. [Trial] costs are spiraling upwards, trials are getting bigger, patient resources are shrinking, there are a lot of drugs in the pipeline, and it’s getting harder to measure endpoints. The old paradigm just isn’t working,” says Brian Schwartz, senior vice president for research at Ziopharm Oncology, of New York. In a typical clinical trial, parameters such as drug dosages and the number of patients in each arm of the trial are predetermined and immutable. Adaptive trials, in contrast, allow tweaking of dosages, patient pool sizes, and so on in response to incoming data. Proponents describe adaptive trials as iterative, with each new round of parameters informed by lessons learned on the fly. “It’s more of a seamless approach,” says Gottlieb. Gottlieb also says that adaptive trials will more quickly rule out unsafe or ineffective drug candidates. “The ability to fail faster is an important advance,” but “adaptive procedures are more complicated to design and analyze, and in some settings more difficult to implement.” In addition to these challenges, the FDA could have trouble getting buy-in on the concept, says Mark Senak, a consultant at Fleishman-Hillard who runs the ‘Eye on FDA’ blog. “The agency and industry will have a tough time selling the concept to policy makers and to a public that is already skeptical of clinical trial design and safety,” he says. Already, though, industry is embracing the concept. Wyeth recently hired a new vice president for adaptive trials, and Robert O’Neill, director of the office of biostatistics at the FDA’s Center for Drug Evaluation and Research, says that each of the FDA’s drug evaluation branches has received adaptive trial proposals. “The FDA is very interested in the concept,” says Mark Chang, a biostatistician at Millennium Pharmaceuticals, Cambridge, Massachusetts. “They’ve begun working closely with industry on adaptive trial designs, and they’re encouraging companies to

AP Photo/Hermann J. Knippertz


New clinical trials policy at FDA

Bayer Healthcare was one of the first companies to follow adaptive clinical trials protocol to determine for which cancer its drug Nevaxar would be more potent.

approach them early in the process.” Although the concept is widely embraced— few would argue against speeding up phase 2 and 3 clinical trials, which can drag on for five or more years—the mechanics of adaptive trials present thorny statistical challenges, says Chang. And Schwartz says that companies interested in adaptive trials tend to underestimate the difficulty of collecting real-time data. “By the time they look at the first 300 patients, there might be 900 patients in the trial,” he says. Companies need to develop simulations to test adaptive scenarios, says Chang. In his models, the two most common variations involve ongoing assessment of sample size and enrichment of the treatment arm with patients most likely to benefit. For instance, Chang will model a range of patients’ responses to a drug, a key factor in sizing trials—smaller variations require smaller sample sizes. Enrichment scenarios, by comparison, often call for first discovering biomarkers in the best responders and then adding more of those patients to the protocol. Chang and Gottlieb also envision ‘pivotal’ trials that combine phase 2 dosing and phase 3 effectiveness studies. “You can run a lot more doses, maybe five instead of two,” says Chang. Bayer Healthcare, based in Leverkusen, Germany, adopted an adaptive approach for its phase 2 trial of a new cancer drug. Without knowing which types of cancer Nexavar (sorafenib tosylate) would fight best, the company enrolled patients suffering a range of advanced cancers. “We knew pretty quickly, within ten or so patients, that kidney cancer was the best responder,” says Schwartz, who helped


run the trial before joining Ziopharm earlier this summer. Bayer then designed a traditional phase 3 trial for renal cell cancer. This approach illustrates another impetus for adaptive trials: many new cancer drugs stop a tumor from spreading but don’t necessarily shrink it. “The traditional endpoint of tumor shrinkage just doesn’t make sense anymore,” says Schwartz. He cautions, however, that committees for evaluating and modifying trials on the fly need to be “completely independent” from sponsors. “That’s the only way to maintain integrity. Industry can’t be within an arm’s length” of the evaluation committee, meaning “companies will have to give up some control,” says Schwartz. He urges the FDA to “very explicitly” spell out the role of the new committees. Most large trials already deploy a data safety and monitoring board empowered to end trials if wide benefits or severe adverse events appear early. However, these traditional committees simply collate data against pre-determined stopping points; the new committees will have much more power. Gottlieb says the FDA will issue two guidance papers in January 2007, with three more to follow. The first will provide guidelines for evaluating multiple trial endpoints; the second will outline how to enrich trials with patients most likely to benefit. “This is a wonderful opportunity,” comments Schwartz. “We want to get drugs to patients quickly and it’s frustrating to look back at some of our trials and see if we had changed this or that we could’ve had the drug to patients six months earlier.” Brian Vastag, Washington, DC

1043

NEWS


Amgen’s TPO mimic faces stiff competition With seemingly novel science, unsatisfactory existing treatment options and a sizeable potential patient population, it’s hard to see how Amgen’s phase 3 platelet growth factor candidate to treat blood disorders could go wrong. But, with several small molecules in the pipeline, it could face stiff competition in platelet growth factor market. Amgen, headquartered in Thousand Oaks, California, has a first-in-class treatment for a deficient platelet count slated to wrap up phase 3 trials by the end of this year and hit the market in 2007. If approved for several related blood disorders, the drug—known as AMG 531— could be a treatment option for more than a half-million people in the US and Europe. And for many indications marked by reduced platelet counts, such as immune thrombocytopenic purpura, an illness causing abnormal bleeding, and chemotherapy-induced low platelet count, the treatment options are currently meager and include steroids as well as the infusion of platelets. AMG 531 is the result of more than a decade of effort to create protein-based thrombopoietin-stimulating agents to increase platelet counts. First discovered in 1994, recombinant versions of human thrombopoietin (rTPO) had the catastrophic side effect of inducing the production of antibodies that cross-reacted with the subject’s own TPO and the development of low platelet counts in normal subjects. Several companies, including Amgen, Genentech, Pfizer, Johnson & Johnson and Schering Plough, all subsequently abandoned their rTPO efforts once it was clear that it had the unintended effect of lowering rather than boosting platelet count. Around this time, research was published by peptide company Affymax of Palo Alto, California, describing a peptide that functioned as an erythropoietin (EPO) mimetic, by binding to and stimulating the EPO receptor. This never bore fruit, because it was less effective than EPO, so Affymax “then published on a similar strategy to identify a TPO mimetic peptide, which was close to TPO in specific activity,” remembers Kenneth Kaushansky, chair of the department of medicine at the University of California, in San Diego. “Hence was born a peptide approach to stimulating the TPO receptor,” he adds, “Others thought that screening large libraries of small organic molecules could also net mimetics, and that is where several other small-molecule mimics have come from.” In the wake of the debacle of rTPO, research has advanced along these two strategic paths— peptide and small-molecule development—to

1044

A new generation of platelet growth factors could succeed where recombinant thrombopoietin has failed.

create TPO mimetics. By the late 1990s researchers were quite successful at identifying a number of small molecules and peptides that bound with the TPO receptor. That’s when the peptide part of AMG 531 was identified in Amgen’s laboratory; it increases platelet production by binding to the TPO receptor and stimulating megakaryocytes, large cells in the bone marrow from which pieces break off to form platelets. Once Amgen researchers identified an effective peptide that did not seem likely to trigger an antibody response, they needed to improve the life span of the peptide in the bloodstream. To create the AMG 531 peptibody, Amgen combined its preselected peptide with a carrier molecule that extends the life of the drug in the patient’s circulatory system, according to Roy Baynes, Amgen’s vice president of oncology and supportive care. If it is approved by the US Food and Drug Administration, AMG 531 will be the first drug known as a peptibody to make it to market. Still, there are at least a half-dozen smallmolecule and small-protein platelet-stimulating agent projects currently at various stages of clinical development to treat diseases marked by platelet-deficiency, according to life sciences clinical trial research firm La Merie, located in Barcelona, Spain. AMG 531 is among the most clinically advanced treatments; but although it does not cross-react like rTPO, AMG 531 is still a relatively inconvenient treatment requiring weekly intravenous doses. By contrast, eltrombopag, developed by GlaxoSmith Kline (GSK) in London, which is also in phase 3 trials, is a small-molecule treatment for patients with low platelet count. Eltrombopag may be among the first small mole-

cules to modulate protein-to-protein interactions, a particularly hard target for this platform, according to market research firm Decision Resources, based in Waltham, Massachusetts. This scientific advance translates into a market advantage: administration orally via a tablet. Mark Schoenebaum, a research analyst at investment bank Bear Stearns in New York City, who follows Amgen thinks this could be a big obstacle for AMG 531. He expects the candidate, if approved, to peak out at a mere $300 million in sales. “It’s not thought to be a big drug,” asserts Schoenebaum. “It’s going to face serious competition from a pill from GSK if they are both approved. Since that’s an oral pill, it is cheaper to manufacture and more convenient.” The initial indication targeted by both Amgen and GSK is immune thrombocytopenic purpura, a condition common in HIV-infected people, in which the body produces antibodies against platelets in the blood. But the next indications targeted for approval are likely to include a whole range of conditions characterized by low platelet counts, including chemotherapy-induced thrombocytopenia. In the chemotherapy market, where Amgen has several major products including anemia treatment Epogen (erythropoietin) and which requires regular intravenous infusions, AMG 531 may still have an edge. But neither Amgen nor GSK are likely to have the last word. “There are 20 or 30 companies that are quietly working on small molecules with much better pharmacologic properties than the GSK molecule,” concludes Kuter. Stacy Lawrence, San Francisco


NEWS

Although BioXell’s successful initial public offering (IPO) on the Swiss Exchange SWX on June 22, which grossed CHF 57.8 ($46.9) million, can be considered a relatively unremarkable event for a company at its stage of development, it could have a wider significance for the Italian biotech sector. Indeed, Italian biotech has so far struggled to convert the country’s strength in life sciences research into a thriving commercial industry. Other, less heralded developments, including a series of regional initiatives and the entry of new investors into the sector also provide some grounds for optimism on the part of the industry’s supporters, but it still faces considerable financial and cultural constraints that could choke further development. BioXell’s decision to seek a listing in Zürich rather than its hometown of Milan underlines the lack of development of a fully fledged investment infrastructure for Italian biotech. In a similar vein, Villa-Guardia-based Gentium, a 2001 spinout of Crinos Industria Farmacobiologica, raised cash in recent, successive offerings on the American Stock Exchange and on Nasdaq in New York City, whereas the Italian founders of NiCox opted to establish that company as a French entity, located in Sophia Antipolis and quoted on the Euronext exchange in Paris. A Milan IPO “wasn’t really considered” says BioXell CEO Francesco Sinigaglia, whereas the Zürich exchange is home to several biotech successes and has the support of investors who understand the sector. Even so, the share offering, which was launched shortly after the general decline in global stock markets in early June, was priced at the bottom of the indicative price range of CHF 44–CHF 48 ($35.5–$38.8)

that the company published and investors took up the minimum number of shares on offer. However, the share price has held up since the IPO, hovering close to the initial offering price for the first six weeks of trading. The BioXell success remains a largely isolated one in the Italian landscape. Despite its prominence in fields such as oncology, immunology and neuroscience, Italy has been Europe’s most egregious underperformer in biotech during the past decade. Italy was bottom of a league table of 14 western European states that measured each country’s gross domestic product against its total number of biotech companies, according to the 2006 Ernst & Young biotech report “Beyond Borders.” An absence of risk capital, deficits in areas such as patenting and technology transfer, a historic inattention to the sector on the part of government and a general lack of interest in commercial biotech on the part of academic scientists have all contributed to this state of underdevelopment. “There is still very modest entrepreneurship in the biotech sector and not many structured and savvy intermediaries. Deal flow is not significant compared to other EU countries of similar size,” says Joël Besse, senior partner with Atlas Venture in London, who participated in investments in two Italian biotechs: Milan-based Novuspharma and Bressoheadquartered Newron Pharmaceuticals. These two firms, along with BioSearch Italia and Milan-based BioXell were all established as either spin-outs from or management buyouts of international pharma R&D centers that had been located in the country. Only BioXell and Newron Pharmaceuticals remain independent. BioSearch Italia merged with Versicor, to form Vicuron Pharmaceuticals, an

BioXell


BioXell: an Italian biotech success story?

Francesco Sinigaglia, BioXell’s CEO, is at the helm of one of Italian biotech’s success stories.


1045


NEWS anti-infectives specialist located in King of Prussia, Pennsylvania. Pfizer then acquired Vicuron for ~$1.9 billion in cash in September 2005. Novuspharma was acquired in January 2004 by Cell Therapeutics, of Seattle, in a stockbased deal initially valued at $236 million. It is difficult to predict whether other biotechs would follow the example of the likes of BioXell. The gap between these companies, all of which were established with relatively broad clinical development pipelines, seasoned management and access to international venture capital finance, and the rest of Italy’s fragmented and, for the most part, undercapitalized biotech industry has been considerable. The great challenge for the sector has been how to close that gap. Regional authorities, notably in Lombardy and Piedmont, where the bulk of Italy’s 160 biopharmaceutical firms are based, are actively involved in promoting biotech, through funding technology transfer agencies, incubators and seed funds (Box 1). Newer initiatives have sprung up in Tuscany and in Sardinia too.

“Clearly the lack of Italian specialist venture capital funds is a problem,” says Sinigaglia. However, individual companies are pursuing alternative funding models. Some, most notably MolMed, have managed to raise cash directly from financial institutions and private investors. MolMed, located in the San Raffaele Science Park, adjacent to the San Raffaele University Hospital and the eponymous Scientific Institute, the country’s largest private clinical research center, has so far secured some €60 ($77) million by this route and may undertake an IPO during the first half of 2007. “We have a broad pipeline and we think we would be ready in the near future,” says Marina Del Bue, general manager at Milan-based MolMed, which is developing cell-based therapies and biotech drugs for cancer. Elsewhere, investors in Genextra, a holding company with a controlling interest in four companies, agreed to double their commitment, to €60 ($77) million, this summer, following their participation in a $41-million investment round in Intercept

Box 1 Italian biotech park taps into traditional industries Italians are often praised for making up for the deficiencies of their country—burdened by bureaucracy and lack of flexibility—with individual creativity. The Canavese Bioindustry Park may be a proof that there is some truth in this cliché. The park is located near the northern city of Turin and its creation was supported in the 1990s by the Piedmont region with the aim of reinforcing the high-tech dimension of the local economy after a major crisis. As a result the park shareholders are 70% public and 30% private. Since 2004, seed capital for startups has been available thanks to a model of financing based on the business angel concept devised to bypass the lack of interest from venture capital investors for early stage projects. “We collect money from wealthy people with no experience in biotech, such as local small entrepreneurs in the textile or mechanical sector, lawyers or accountants. Before meeting us they never thought of becoming business angels,” says Silvano Fumero, who conceived the park when he was still head of R&D at Serono. Thirty people gave a total of €3 ($3.8) million, each contributing a small sum and becoming founding members of a seed capital society called Eporgen Venture. “We are hopeful in a couple of years the most promising newborn companies may attract investments from [the] biggest players, maybe one [of] the international venture capitalists we have involved in the selection of projects,” explains the park project manager Fabrizio Conicella. Birth rate is unusually high by Italian standards with five new startups born last year and the intention of starting another five companies by mid 2007. The initial success of the project is already a blow against the cultural and political foot-dragging of the country, but is it replicable? “We are actually examining the way to implement a similar model in the Rhône-Alps region [of France] but it’s not so easy,” says Valérie Ayache, managing director of the biotech association near Grenoble, Adebag. She points out that the motivations of people investing in Eporgen are very much related to the history of the territory, the charisma and experience of the project fathers, and the very integrated model they have created between the park and Eporgen. Other Italian regions are trying to learn a lesson from the Canavese experience, too: until now biotech has played a minor role in the national business angels network (Iban) but its secretary general Tomaso Marzotto Caotorta thinks it’s time to create a club of senior managers scouting Italian life sciences institutes for innovative ideas. Anna Meldolesi, Rome

Pharmaceuticals, a company headquartered in New York City but based on research into the bile acid–activated nuclear receptor farnesoid X performed at the University of Perugia. “Although it is supplying mentoring and administrative support, Genextra is neither an incubator nor an investor. We are not an investment fund. We are a biotechnology group,” says Paolo Fundaro, Genextra chief financial officer. Milan-based Genextra has high visibility in Italy because its backers include leading entrepreneurs and industrialists, such as its founder, telecoms entrepreneur Francesco Micheli, Marco Tronchetti Provera, chairman of Pirelli & Telecom Italia, FIAT chairman Luca Cordero di Montezemolo and Diego Della Valle, CEO and chairman of the luxury shoemaker Tod’s. The model is borrowed directly from that of another Micheliled enterprise, the internet and telecoms group eBiscom, now FastWeb, which raised $1.5 billion at the beginning of the decade. Its progress, along with that of BioXell—now the country’s flagship biotech firm—could help to shape investor sentiment toward the sector. Assobiotec, in Milan, which represents the industry, thinks the country’s new national government can help as well. One measure, says Assobiotec president Roberto Gradnik, would be to create a national agency for innovation that would support technology transfer and partnering. “At the moment, if anybody, such as a private investor, is interested in investing in biotechnology, they don’t know where to go,” he says. Risk-averse Italian investment funds might engage with the sector if a ‘guarantee fund’ were put in place—a sort of voluntary insurance scheme that would allow venture capital funds to offset their investment losses against profits on more successful ventures. Assobiotec is also trying to persuade the government to adapt the ‘Young Innovative Company’ concept, originally developed in France to provide tax breaks and other fiscal supports to research-intensive startup companies, to the Italian tax code. Italy had a change of government in May. In its new cabinet, led by Prime Minister Romano Prodi, responsibility for innovation policy was transferred from the research ministry to the industry ministry, headed by Pier Luigi Bersani. Gradnik interprets this as a positive move. But, says Sinigaglia, a real shift away from manufacturing and towards a knowledge-based economy still needs to happen. “We need to see government to commit to that switch.” Cormac Sheridan, Dublin

1046


NEWS IN BRIEF

University of Queensland (UQ), Australia, molecular geneticist Robert Birch got more than he bargained for when he introduced a bacterial gene into sugarcane to convert sucrose into its high-value isomer, isomaltulose. In fact, the gene encoding sucrose isomerase, cloned from Pantoea dispersa, a harmless colonist of the crop’s leaves, delivered twice as much sugar as Birch expected. Some of the transgenic plants were producing isomaltulose at up to 110% of the normal concentration of sucrose. Others produced little or no isomaltulose, but yielded up to 100% more sucrose. In the past 50 years, breeders had been unable to improve sugarcane’s yield by even 1%. Last August, CSR, Australia’s biggest sugar refiner, and its commercial partner, UQ’s commercial arm, UniQuest, received an AUD$5 ($3.8) million federal research grant, under AusIndustry’s Renewable Energy Development Initiative, to develop Birch’s high-yield sugarcane, dubbed ‘SugarBoost’, as a source of the ‘green’ fuel ethanol. The partners recently planted the first, small-scale, contained field trial of the transgenic sugarcane, with approval from the office of the gene technology regulator. Queensland University of Technology molecular geneticist James Dale, also founder and CEO of Brisbanebased ‘biopharming’ company Farmacule, describes the development as “huge,” in terms of its significance to Australia’s nascent ethanol industry. Indeed, Australia has been battling to keep its sugar industry alive in the face of cheap sugar from Brazil. Dale adds that it might eventually be possible to engineer similar yield increases in other ethanol feedstock crops like sugar beet and maize. GON

Senate compromise on SBIR reform A bill has been approved by the US Senate committee on small business and entrepreneurship that would allow companies that are primarily owned by venture capitalists (VCs) to obtain small business innovation research (SBIR) grants. Since 2003, companies whose majority investors are VCs have been ineligible for SBIR funds. Still, companies with some VC investment have been able to access the grants, according to a General Accounting Office

Bioengineered scents available soon from New Zealand The Horticultural and Food Research Institute of New Zealand—known as HortResearch—has filed patent applications for the use of the genes that produce the scent of green apples and red roses. Auckland-based HortResearch examined its databases of fruit genes and compounds to find the genes that encode enzymes that make alpha faresene synthase (green apple scent) and germacrene D synthase (rose scent). HortResearch used its flavor compounds databases to build maps of hypothetical pathways of how the resulting compounds are synthesized in the fruit or flower. These hypothetical pathways then allowed the scientists to postulate what types of enzyme might catalyze each step. The scientists then looked in their gene databases for genes that encode enzymes that can perform these steps. Likely genes were tested in Escherichia coli to see if they did produce those compounds and then in model plants. To manufacture the enzymes, HortResearch uses biofermentation. “What we are suggesting is that you could actually use real enzymes from the plant,” says HortResearch scientist Richard Newcomb, “and it’s even more ‘nature identical’.” Steve Meller, head of Global Biosciences at Procter and Gamble located in Cincinnati, Ohio, believes that a technological process that could produce cost effectively the flavors and perfumes manufacturers need would be a benefit. He adds: “The really desirable odorants out there are those that are much more complex, so I think that’s really where the hurdle is going to be.” HortResearch’s work of producing flavors and fragrances is the flipside of the work done by Californian company Senomyx which focuses on the receptors that enable humans to perceive taste (Nat. Biotechnol. 22, 1203–1205 2004). KG HortResearch


200% ethanol boost from Oz sugar

(GAO) report released in the first half of this year. In 2004, about 22% of National Institutes of Health SBIR grants, or $127 million, went to companies that were held in the minority by VCs, according to a report released in the first half of this year by the GAO. The Small Business Administration Reauthorization bill includes an amendment that would commit one-quarter of SBIR funds to companies that are majority-backed by VCs. “We’re supportive of the compromise and we look forward to working with the Senate,” asserts Alan Eisenberg, the executive vice president for capital formation and business development for the Washington, DC-based Biotechnology Industry Organization. The bill is now on its way to be considered by the full Senate. StL

Europe backs ES celsl News in Brief written by Alla Katsnelson, Kim Griggs, Stacy Lawrence, Linda Nordling, Graeme O’Neill, Peter Vermij & Emily Waltz

1048

After a heated debate, the EU voted in late July to continue funding embryonic stem (ES) cell research, but with narrower crite-

ria. The EU council agreed to continue to support research on ES cells, but not their procurement, which often requires destruction of the embryo. Several countries, notably Germany, with strict laws on stem cell research attempted to block the decision. The funding is part of the EU’s €72.7 ($93) billion research budget for 2007–2013. The vote came just days after US President George W. Bush blocked the passage of a bill that would have allowed federal money to fund similar stem cell research in the US. The bill would have supported use of embryos destined for disposal at in vitro fertilization clinics. The contrasting decisions of the EU and US may give European biotechs an edge in recruiting scientists, experts say. “This is a missed opportunity for the US to assert leadership in the field,” says Michael Werner, president of the Werner Group, a Washington, DC-based biotech research consulting firm, and former chief of policy at Biotechnology Industry Organization, also in Washington, DC. “The EU is taking advantage of that.” EW


NEWS IN BRIEF


New products table Product

Details

Lucentis (ranibizumab injection) Genentech (S. San Francisco, California)

On June 30, the US Food and Drug Administration approved Lucentis, a recombinant humanized antibody fragment, for the treatment of neovascular wet age-related macular degeneration (AMD). An estimated 1.7 million people in the US suffer from severe AMD, the leading cause of blindness in the elderly. The ‘wet’ form of the condition is caused by growth of abnormal blood vessels that leak fluid and blood, leading to retinal scarring. Lucentis inhibits the activity of the angiogenesis protein human vascular endothelial growth factor A (VEGF-A). In trials, Lucentis maintained vision or restored partial vision loss in most wet AMD patients. Recommended treatment is intravitreal injection administered once a month.

Elaprase (idursulfase) Shire Pharmaceutical Group (Basingstoke, UK)

On July 24, the FDA approved Elaprase, an enzyme replacement therapy for the treatment of Hunter Syndrome. Also called Mucopolysaccharidosis II, Hunter Syndrome is a life-threatening X-linked recessive condition resulting from absence or insufficiency of duronate-2-sulfatase, causing the accumulation of cellular waste products in tissues and organs. The condition affects about 1 in 65,000 to 132,000 births. Elaprase, the first-ever treatment for Hunter Syndrome, is administered in weekly infusions. It has also under review by the EMEA in Europe. AK

UK panels urges tightening of phase 1 rules British pharmaceutical industry organizations say they are generally pleased with draft recommendations by a scientific expert panel for tighter rules governing phase 1 trials of “novel and potentially higher risk drugs…such as monoclonal antibodies.” The UK government convened the panel earlier this year when six volunteers experienced very serious adverse effects from TGN1412, a T cell–targeting ‘super monoclonal antibody’ with an agonist activity developed by TeGenero of Würzburg, Germany (Nat. Biotechnol. 24, 475–476 2006). In an interim report released on July 25, 2006, the panel said that in higher risk studies “the first dose in man should be given to one person only, leaving sufficient time for any adverse reaction to develop before further administration or administration to additional people.” The experts urged drug developers to inform regulators earlier about elevated risks and suggested enrolling people with the targeted disease rather than healthy volunteers into phase 1 trials of higher risk drug candidates “particularly if the drug is expected to affect the immune system.” The recommendations generally echoed those published a day earlier by a joint task force of the Association of the British Pharmaceutical Industry and the UK BioIndustry Association, both based in London, including the proposal to set starting

doses in first-in-man trials of biologicals below a point at which no biological effect is expected. The industry task force, however, limited some of its advice to “novel agents stimulating the immune system,” excluding from extra scrutiny agents with inhibitory effects. Such agents, the task force writes, “are widely used” and “rarely have acute adverse effects.” The expert panel is due to issue its final report in November. PV

GM sorghum stalled in SA In July, the South African government rejected an application to conduct field trials of genetically modified (GM) sorghum on its soil— research that received $16.9 million from the Bill and Melinda Gates Foundation. The decision, which received a lot of media interest throughout Africa, was based on a judgment that the containment level proposed was too low for a native African plant. Similar concerns about contamination to native plants have been raised in Mexico in the past, as the country tried to develop GM corn (Nat. Biotechnol. 23, 6, 2005). Gatsha Mazithulela, executive director of the biosciences arm of the Council for Scientific and Industrial Research (CSIR), located in Pretoria, says the rejection, far from destroying the public image of biotech, actually could inspire confidence. “It’s giving a clear message that the South African GM [organisms] legislation is working and if you don’t


submit the right application you won’t go through,” he says. “The issue here is that sorghum’s center of origin is Africa and that’s why there’s a cautious approach,” explains Jocelyn Webster, executive director of AfricaBio, a non-governmental organization supporting research, development and application of biotech in Africa, adding, “ There’s more information required by the applicants, and I suspect that there will be the usual process followed by the regulators.” Meanwhile, researchers working on the sorghum project are hopeful that a second application, which proposes higher containment levels, will be accepted before the end of this month. LN

TKT infringes Amgen EPO patents A US federal appeals court ruled on August 3 that Cambridge-based company Transkaryotic Therapies (TKT), acquired last year by Shire, has infringed two patents held by Amgen for the production of erythropoietin. The ruling effectively bars US sales of TKT/Shire’s EPO product, Dynepo, in the US until the patents expire in 2015. However, the court also ruled one of Amgen’s patents invalid and sent another claim back for review. Although both sides won two battles, notes Kevin Noonan, partner at the law firm McDonnell Boehnen Hulbert & Berghoff in Chicago, Illinois, Amgen won the war. “In the grand scheme, the patentee only has to win one” to demand an injunction against the competitor. But such rulings give “some certainty to how these claims can be interpreted,” he added, and can be a “spur for other [companies] to figure out how to get around them.” An additional suit is pending against Swiss company Roche, which has plans to sell its new product CERA, a long-acting EPO, in the US. The ruling “could widen the window of opportunity for Roche to craft an infringement defense for CERA that capitalizes on these new interpretations,” writes David Witzke, biotech analyst at Banc of America in New York, in a research note. Amgen’s patents expired in Europe in 2004, and European sales of Dynepo are slated to launch this year. CERA was submitted for review for approval by the European Medicines Agency in April. AK

1049


Abe Abuchowski As CEO of one of the first companies to make protein delivery into a profitable business, Abe Abuchowski knows what it takes to bring a new technology to market. Although his technology— PEGylation—is now considered an industry gold standard, its three-decade development history illustrates the often rocky path to commercial success for platforms.

Drug delivery is a dicey business, and Abe Abuchowski was one of the first to make it work. At the dawn of the biotech industry, proteins’ promise as therapeutics was undisputed. But taken from animal or recombinant bacterial sources, their therapeutic potential was often undone by high immunogenicity. Short circulating life posed an additional problem—frequent doses were required to maintain therapeutic levels, again increasing the likelihood of an immune response. As luck would have it, when Abuchowski began his doctoral research in biochemistry in 1971, his thesis advisor at Rutgers University, Frank Davis, put him to work on this very problem. A few years back, Davis had happened upon a paper suggesting that poly(ethylene glycol) (PEG), a polymer widely used in foods and cosmetics, could provide a solution. Initial studies indeed showed that “hanging a bit of PEG” onto a protein reduced immunogenicity and improved circulating life, recalls Davis, and along with two colleagues he patented a technique for PEG-protein delivery. Within a few years Abuchowski and his colleagues hit the jackpot when looking for a general method for attaching PEG to a protein: a formulation of PEGylated bovine serum albumin. This was the first protein molecule created that was neither immunogenic nor antigenic. “It was a real Eureka moment,” says Abuchowski. “Even after we did it we couldn’t believe it, quite honestly.” More importantly, the researchers went on to show in mice that a PEGylated protein could cure a previously untreatable enzyme deficiency. At a time when researchers were just beginning to venture into the commercial side of discovery, Abuchowski was happy to take the leap. “I think Abe very quickly saw the business applications,” says Davis, who is now retired. In 1982, the duo formed Enzon Corporation in New Jersey to bring PEG-based treatments to the clinic. In 1990 the company’s first product, PEGylated adenosine deaminase enzyme (ADA), known as Adagen, gained US Food and Drug Administration (FDA) approval—making Enzon the fifth company to have a biotech drug approved. Inherited absence of ADA had recently been found to cause one type of severe combined immunodeficiency disorder. Without PEG, ADA has no therapeutic effect. Four years later, the company received approval for Oncaspar, PEGylated Lasparaginase for acute lymphoblastic leukemia. “A company doesn’t exist to do research, but to get products on the market,” says Abukowski. The decision to go after two products with almost no market was a deliberate one. “I think Enzon was pretty smart,” notes Roger Harrison, an associate at Plexus Ventures, a global pharma consultancy based in Maple Glen, Pennsylvania, and an independent consultant specializing in drug delivery. “There’s an established belief that anything you do [to a protein] will create a problem with the FDA,” he says. But both Adagen and Oncaspar minimized this added uncertainty because both were made possible by the technology, and both approval processes could be expedited by orphan drug status. Even with Enzon’s irrefutable clinical data on Adagen, Abuchowsky notes, “up until the day of [FDA] approval, I probably had half of Enzon management betting against me.” Ultimately, getting the two products out in quick succession essentially proved the technology.

1050

Meanwhile, big pharma was beginning to appreciate PEG’s potential. Enzon signed a deal with Schering-Plough to develop a PEGylated version of alpha-interferon (PegIntron) for treating hepatitis C. But as Enzon’s management waited to see whether the project would succeed, resources dwindled, stock price fell and disagreement began to brew. A messy restructuring ensued, its outcome being a much-diminished R&D program and Abuchowski’s departure—not just from Enzon but, for a time, from biotech. PegIntron’s approval in 2001 pushed Enzon into profitability, and also marked the first time that a second-generation protein superior to the first generation due to a biotech improvement. Within a year, it had captured about 65% of the market share of a protein that had already been on the market for over a decade. “Had Enzon had the ability to prepare their own proteins and chosen more of a therapeutics model than a drug delivery model, they could have done alpha-interferon on their own,” notes Robert Shorr, who served as vice president of research and development from 1991 to 1997. Shorr is now CEO of Cornerstone Pharmaceuticals in New York. He also serves a scientific advisor to Abukowski’s new company, Prolong Pharmaceuticals in Monmouth Junction, New Jersey. By all indications, Abukowski is not about to make the same mistake twice. “Enzon was a company that developed the technology and

“A company doesn’t exist to do research, but to get products on the market,” says Abukowski. introduced it,” he says. “Prolong is a product company.” Part of the plan is to realize some of the projects that languished in Enzon’s deep-freezer. But with several biotech drugs coming off patent and manufacturing costs falling, Prolong is also looking to Asia. For Abukowski, one of the lessons of Enzon was that the sooner you get to revenue, the more freedom you have to decide where to go next. “There’s an alignment in philosophy between Abe and many players in Asia,” notes Gurinder Shahi, director of the Global Biobusiness Initiative at the University of Southern California in Los Angeles. Unlike in the United States, “in Asia there is no risk capital, so companies are forced to use a quick-to-revenue strategies and use that revenue to make a product.” Because the technology is now well established, it creates a proprietary dimension to generics. In the five years since the technology’s acceptance became official with the approval of PegIntron, about ten other PEG products have come to market, with several more in trials. Yet other technologies are emerging. Even for old products, notes Walter Blatter, CEO of ImmunoGen in Cambridge, Massachusetts, protein modifications other than PEG, such as the two additional N-glycosylation sites in Amgen’s long-acting erythropoietin Aranesp, can increase the circulation life of a molecule. Whether PEG has the capacity to surpass its primary role as a second-generation modification remains to be seen, says Samuel Zalipsky, associate director of protein and linker chemistry at ALZA in Mountain View, California. He concludes: “It’s still got some life in it, though it’s not the only game in town as it was in the past.” Alla Katsnelson, New York


D ATA PA G E

Biotech R&D goes further afield

Top 20 pharma as portion of R&D deals

The life sciences continued to account for 54% of federal R&D funding for the third year in a row.

The last three years have seen a rapid decline in the share of biotech R&D deals conducted by big pharma.

35.0

60%

30.0

50%

25.0

40%

20.0

30%

15.0

20%

10.0

10%

0.0

0%

쐽 Life science funding as percentage of total federal funding

19 8 19 3 8 19 4 8 19 5 86 19 8 19 7 8 19 8 89 19 9 19 0 9 19 1 9 19 2 93 19 9 19 4 9 19 5 96 19 97 19 9 19 8 9 20 9 00 20 0 20 1 0 20 2 0 20 3 04

5.0

쐽 Life sciences funding

Year

Top 20 pharma-biotech deals as a percentage of total

US life sciences federal research funding

(% total federal funding)

contributing $18 billion. Biotech firms increasingly dominate dealmaking compared with big pharma, which now contributes just under half of the funding for all deals; the value and number of these biotech partnerships remained buoyant.

($ billions)

The governments in New Zealand, Korea and Canada are placing big bets on their biotech sectors. But the United States continues to dwarf other countries in terms of total investment; last year, the US public sector spent $30 billion on the life sciences, with the private sector

90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Deal value

77%

74%

72% 59%

Deal number

48% 41%

39%

34%

2001

31%

2002

2003

27%

2004

2005

Year

Source: National Science Foundation

Source: Windhover, Burrill & Company, Nature Biotechnology

Biopharmaceutical research and product alliances

Number of partnership deals by stage

Windhover’s data reports $12 billion in R&D deals; Burrill pegs the figure at a more outstanding $17 billion.

Very late stage and discovery deals actually declined last year, whereas all other categories held steady or increased. 76 66 75

Marketed 514 9.1

12.0

($ billions)

606 10.3

621 12.3

555 10.4

617 12.2

700 600

10.0

500

8.0

400

6.0

300

4.0

200

2.0 0.0

100 2001

2002

2003

2004

2005

9

Total deal value

(Number of deals)

14.0

Number of deals

12 14 8 20 15 23 16 29 25 27

Approved Filed for approval Phase 3

26

Phase 1

0

81

2005 2004 2003

41 37

Phase 2

2002

50 57 62

38 45 44

119 119127 134

Pre-clinical

Year

235

169

Discovery

Windhover data include any deal involving a biotech firm and use only the first indication figures (if provided), whereas Burrill looks at all potential products and focuses on the research money going to biotech firms. Source: Windhover, Burrill & Company.

193

137

0

50

100

150

200

250

Number of deals

Source: Windhover, Burrill & Company

New Zealand, Korea, and Canada have devoted the largest share of their public research funding to biotech.

Country Biotech R&D spending by companies (millions)

Percentage of total business R&D spending

Based on earliest year available, 2002, 2003 or 2004. Source: Organisation for Economic Co-operation and Development

1052

20%

400 200

15%

10%

212 2%

300

6% 90

7% 149

131

10% 29 1%

105

100

5 0%

0

5%

d

en

an el Ic

w ay or

N

la Fi n

Sw ed

k

nd

d

ar m

al an

en

Ze

D

n

m do N

Ki ng

ai te d

ew

a

0%

ad

0%

25%

453 0%

500

Sp

67

2 2% 9 5 1%

8 4% 4

8 2% 8 95

2 4% 01

10%

U U nite ni d te S d ta Ki te ng s G do er m m a Fr ny a C nce an D ad en a m a Sw Ko rk itz rea er la n Is d C hi ra na e ,S I l ha taly ng Au ha st i ra l N ew S ia p Ze ain al a So Fin nd ut lan h Af d r Ic ica el a N nd or w Po ay la nd

0

1 3% 99

5 20

9 46

6 3% 99

4 19

7

1,

72

9% 2 5% 51 23 3% 6

% 12

8

1 3% ,34 7 1 6% ,34 2

2,000

20%

00 2,

4,000

7%

6,000

600

ni

21

%

30%

30% 24% 550 12%

an

% 24

8,000

727 15%

700

U

40%

10,000

800

C

50%

12,000

($ millions)

60%

51

14,000

Ko re a

%

,2 14

16,000

Public biotech spending as a percentage of total

Iceland, Denmark and New Zealand are pouring anywhere from one-quarter to one-half of all their business R&D money into biotech. Biotech R&D as a percentage of all R&D spending

International biotech public sector R&D spending

32

R&D spend on biotech from business in OECD countries

($ millions)


Stacey Lawrence

Biotech public R&D spending

Country Public biotech R&D as a percentage of total

Includes government and higher education biotech R&D spending. Based on earliest year available, 2002, 2003 or 2004. Source: Organisation for Economic Cooperation and Development


N E W S F E AT U R E

Systems Biology, Incorporated?

In June, privately held VLST Corp. in Seattle announced that it had raised $55 million in a Series B venture financing round. At the time, that was reported to be the 16th largest venture capital deal of the year across all industries1— no small feat for an upstart biotech when even established players in the field were finding it tough to curry favor with investors. But the deal was particularly remarkable for another reason. VLST (which derives its name from Viral Logic Systems Technology) is the first company to graduate from Seattle’s Accelerator Corp., a venture-backed life sciences incubator formed in 2003 by a group of venture investors—MPM Capital, Arch Ventures, Versant Ventures, Alexandria Real Estate Equities and later OVP Venture Partners and Amgen Ventures—in conjunction with the Institute for Systems Biology (ISB), of Seattle. As such, the recent financing seemingly marks a victory for the young ISB and a new milestone in the commercialization of systems biology—a group of marquee venture capitalists (VCs) putting major money into a very early-stage platform technology company at a time when most venture capitalists are avoiding biotech altogether or are trying to reduce risk with various accelerated commercialization strategies2,3. Is systems biology finally coming of age? Systems biology catches on Certainly, systems biology as a discipline has gained in popularity over the past several years. Since the ISB was founded in 2000 by Leroy Hood, close to a dozen independent systems biology institutes have been created around the world, and many more universities have created systems biology departments. But the commercial success of the discipline has been thus far ambiguous. Systems biology ideally seeks to understand complex biological systems in their entirety by integrating all levels of functional information into a cohesive model. That stands in contrast to the reductionist approaches that became standard in the twentieth century, with biologists teasing out functional information on organisms one gene or one protein at a time. Strategies for systems biology vary, but generally come down to some combination

of bottom-up data collection (for instance, amassing comprehensive information on an organism’s genome, proteome, “transcriptome,” “metabolome,” “interactome,” “transportome” and any other “-omic” approach, at all possible levels of complexity) and top-down computational modeling and simulation, in which known functions and behaviors of biological components are described mathematically and linked into complex models that allow for the dynamic interaction of large numbers of variables. Hood insists that both approaches are necessary, and that true systems biology requires, whenever possible, a global attitude toward data collection. Some companies are indeed using both topdown and bottom-up approaches to discover new knowledge, but many have focused on modeling and simulation, gathering their basic data from scientific literature or collaborative partners and tacitly accepting the greater rates of error or uncertainty that go with incomplete understanding of an organism’s constituent parts and how they interact. But there are more than technical challenges to pursuing a systems approach to biology. Because it requires understanding of biological function from phenotype down to the molecular or even atomic level, a systems approach

ISB, Seattle


As the first ‘systems biology’ companies achieve some measure of success, the question remains whether systems biology can provide a viable business model. Karl Thiel investigates.

requires biologists, chemists and physicists of many stripes. And because of the intense data collection, processing, modeling, and simulation required, systems biology also requires computer scientists, mathematicians, software engineers and other people not usually found in university biology departments. Hood left the University of Washington in Seattle to found ISB because he believed that he couldn’t effectively build the necessary cross-disciplinary teams in a university environment, nor find the financial backing necessary to create the required infrastructure. And he believed that systems biology would produce an enormous amount of valuable intellectual property that could be better managed outside a university setting. That would seem to make systems biology better suited for a private, commercial enterprise. But there are challenges here, too. Theoretically, a systems approach to understanding and treating human disease should identify the best means of therapeutic intervention. But others will still need to translate that information into an actual therapy. Therefore, systems biology sounds like just one more ‘tool’ strategy for drug discovery and development—a platform that will feed new targets, or new interventional strategies, to drug makers. And that’s exactly what VCs don’t want to hear right now. “That’s the noninvestable model,” says Carl Weissman, president of Accelerator and a venture partner at MPM Capital of Cambridge, Massachusetts. “That’s what VCs are not interested in—people who are trying to build up some sort of a stacked royalty and services business.” That would seem to put the young companies trying to commercialize systems biology into a tough position. How do they create a winning

The Institute for Systems Biology (ISB) brings together researchers from diverse backgrounds and provides a place for interdisciplinary work outside the usual academic setting.


1055



business model in a field that is still struggling to define itself? Enter Accelerator Unlike most incubators, the Accelerator was formed specifically to nurture startups that would benefit from an affiliation with ISB, and to provide management support and equitybased backing along with the more typical facilities and infrastructure addressed by many life sciences incubators. Indeed, says Weissman, Accelerator was initially conceived as a vehicle to specifically nurture ideas spun out of the ISB, as an earlystage testing ground where for a relatively small investment—usually in the $2 million to $5 million range—new concepts could either prove worthy of significant further investment or be shut down without major loss. That’s still the idea, but Accelerator’s reach has widened. VLST, for instance, did not come out of ISB, but rather was founded by Craig Smith and Steve Wiley, two scientists who most recently came from Immunex/Amgen of Thousand Oaks, California. Smith codiscovered the rheumatoid arthritis drug Enbrel (etanercept) while at Immunex. The company’s platform technology is based on using virulence factors found in various viral genomes as a guide to drug targets for autoimmune and chronic inflammatory diseases. According to VLST president and CEO Martin Simonetti, Smith hypothesized that many viruses rely on these secreted proteins to slow or evade the immune system and thus gain a foothold in their host. For instance, he says, Smith found that many viruses encode a protein that “looks a lot like” the p75 tumor necrosis factor (TNF) receptor—a recombinant form of which ultimately became Enbrel. At the same time, the p55 TNF receptor, which some researchers investigated as a potential drug, did not prove effective. After retrospective analysis, Simonetti says viral genomes may explain why: “We couldn’t find any viruses that coded for p55.” The idea that virulence factors could lead to what he calls “prevalidated” targets was retrospectively validated not only with Enbrel, but with other targets like interleukin (IL)-1 and CD30, he says. “If you knew what the virus was telling you, you would have saved yourself a lot of time and money in the clinic.” The company plans to use a bioinformatics approach to identify virulence factors in viruses, then use proteomics to identify the specific target, and then finally to create therapeutics to mimic the behavior of the virulence factors. The $55-million Series B round is a big step up from the approximately $4.5 million the company initially got from Accelerator. “When I first joined, we weren’t going to raise anywhere near

1056

that kind of money,” acknowledges Simonetti. “But when we sat back and thought about it, the real transforming event is the proof-of-concept phase 2 clinical trial.” Thus, the Series B round, divided into three tranches, is intended to take the company though a phase 2a trial, at which point a successful outcome should make further financing relatively easy. VLST is, in short, aiming to be a fully integrated drug company and to simply bypass the whole ‘tool’ conundrum. Ceci n’est pas Systems Biology The only problem is, despite its affiliation with ISB, VLST is not really a systems biology company by most measures. It is not seeking a systems-level integration of the human immune system to better understand targeted diseases, but rather it is using the adaptive evolution of viruses as a shortcut guide through the darkness to better targets. Hood takes it a step further and asserts that none of the companies at Accelerator—including one called Homestead spun out of his own lab—are really pursuing systems biology. “It’s still too soon,” he says. Homestead “is using systems thinking to identify biomarkers in the blood that may be useful in diagnostics. I think that kind of company has a chance of making a real contribution, but it’s not really a systems biology company—it just defines one aspect of a systems approach.” The same goes for two other companies— MacroGenics of Rockville, Maryland and NanoString of Seattle—that were spun out of ISB but are not part of Accelerator, and indeed, by Hood’s standards, for most other companies claiming to be working in the space. “Any company that claims to be in systems biology is doing it on a very marginal basis,” he says. “Because we’re just now developing the necessary tools.” But for those companies that are at least basing their businesses on modeling and simulation of complex systems, the problem remains—how do they successfully turn an essentially tooloriented platform into a growth opportunity? For some companies, the answer has been to go after as much capital as possible and try to build a fully integrated drug company—an approach that requires finding willing VCs with long time horizons, a steep challenge these days. And one company initially pursuing this path, BG Medicine of Waltham, Massachusetts, actually switched from a drug discovery model to a service model4 in 2005. For others, slow growth and modest capital budgets have been the key. One of the earliest simulation and modeling companies to begin operations was Foster City, California’s Entelos. Founded in 1996, it has created a series of ‘PhysioLabs,’ dynamic models of various disease states that integrate

information down to individual protein interactions based on information derived from published literature, with behaviors represented as differential equations and linked into a simulated patient. Different ‘virtual patients’ can then be created to represent either known variations in—or uncertainty about—the underlying parameters. Groups of diverse ‘patients’ are then used to simulate various interventional outcomes. Entelos has certainly had some tangible success both in terms of partnerships and in the fact that after raising about $50 million in venture capital, it went public on London’s Alternative Investments Market (AIM) in April, raising $20 million in its initial public offering (IPO). But the company’s path has not always been smooth. Its approximately $78-million market capitalization upon IPO was only slightly more than the roughly $70 million it has raised in private and public rounds, suggesting that the market does not yet see a great deal of surplus value in what the company has created with its capital. Part of that could come down to the revenue model, which has thus far been mostly based on some form of fee-for-service compensation. But the company is now expanding its deal structures to take a greater stake in some of its projects. In February 2005, Entelos announced it had expanded a collaboration on rheumatoid arthritis therapies with Organon of Oss, The Netherlands, into a codevelopment, comarketing deal that gives it a bigger piece of the possible upside. “Royalty deals are great, and we all want to get them,” says Entelos CEO James Karis, “but getting single digits on something that’s ten years out—I’m not sure that’s got a whole lot of value. But when you get to collaborate with someone and have the opportunity to codevelop and potentially comarket a drug, that’s a different level of value added.” “We also in some cases own other aspects of the biology that come out of our relationships,” says chief technical officer Alex Bangs, noting that the company has filed for patents on potential drug targets it has identified through its simulations, which it could choose to later out-license or even develop. The urge to move from service to products motivates more than just Entelos. San Diego’s Genomatica, a company that has used a systems approach in modeling microbes and mammalian cells primarily to help clients improve the production efficiency of chemicals and recombinant proteins, has like many systems biology companies raised very little venture capital (Table 1). After an initial $3.5million round from Iceland Genomic Ventures in 2000, Genomatica has relied on organic




Table 1 Selected systems biology companies Company/founded

Focus

Equity financing background

Entelos/1996 (LSE: ENTL) (Foster City, California)

Complex human disease models, PhysioLabs.

About $50 million in Series A–D rounds; $20-million IPO on London’s AIM market in 2006.

Genstruct/2001 (Cambridge, Massachusetts)

Drug discovery based on iterative wet lab data collection and in silico modeling.

Raised $6.5-million Series A round in 2003; none since.

Genomatica/2000 (San Diego)

SimPheny client-server application builds predictive models of organisms based on cellular metabolism.

Initial $3.5 million venture round from Iceland Genomic Ventures; none since.

GeneGo/2000 (St. Joseph, Michigan)

MetaCore platform integrates and visualizes cellular function data into complex models.

$1.4 million from Michigan Life Sciences Corridor; no institutional backing.

Ariadne Genomics/2002 (Rockville, Maryland)

“Natural language processing” and statistical algorithms. PathAssist software for visualization and analysis of regulatory pathways.

Various grants and government funding; no reported venture backing.

Gene Network Sciences/2000 (Ithaca, New York)

VisualCell data integration tool. Licensees include ISB. Builds predictive models of cells.

Some angel investor backing.

Ingenuity Systems/1998 (Mountain View, California)

Pathway analysis software.

Venture backers include Affymetrix as well as institutional VC firms.

BG Medicine/2000 (Waltham, Massachusetts) (Founded as Beyond Genomics)

Focus on systems pharmacology, including biomarkers for liver toxicity. Wet lab as well as in silico work. Switched from drug discovery to service focus in 2005.

Over $26 million in institutional and strategic funding.

BioSeek/2002 (Burlingame, California)

Human disease models, used for partners and internal discovery.

$8.4-million Series A (2002); $19 million in total private equity.

Target Discovery/2002 (Palo Alto, California)

Diagnostics based on protein isoforms.

$7-million Series A (2002–2003); none since.

growth and government funding to move its business forward. But now, the company is migrating from what was essentially fee-for-service consulting work to product ownership. “I think the business plan has to be built around chemical and biological products,” says Christophe Schilling, Genomatica’s president and chief scientific officer. That means negotiating royalties from the sales of drugs and biologics that Genomatica helps clients produce, but could also mean outright ownership of some future projects. Schilling acknowledges that when the company was starting out a few years ago, the technology was at an early stage and needed further development and validation before it would “present the kind of business case where we would want to raise tens of millions of dollars.” So the company instead opted for a business plan that didn’t require much capital and relied mostly on slow, organic growth. But now Schilling believes that Genomatica’s technology has proven its value and that he can offer investors a compelling growth opportunity. “We’re at a point today where that scenario has definitely changed,” he says. Although he believes the company could be successful on a smaller scale with a low-capital, slow-growth approach, Schilling now sees reason to accelerate the process. Still, not all companies want to create products. Gene Network Systems of Ithaca, New York, is using a modeling approach to reverse-engineer experimental data into integrated models of complex biological networks, explains CEO Colin Hill. “We are definitely a tool company, a

platform company. We are not trying to make drugs ourselves,” he says. But he, too, feels the pressure to move towards a product focus. “We’ve had various people discuss that with us—’Why don’t you become a drug maker?’ And I honestly think that’s a horrible idea, at least right now,” he says. “Many young companies even now get forced into doing that,” he says, when they have no particular competitive advantage—indeed, many considerable disadvantages of scale—in drug development. “Until our technology really demonstrates a huge, huge improvement in drug development success rates, I don’t see why it makes sense for a platform company to switch—unless they have no choice because the investors are pushing them to do that,” he asserts. Gene Network, even though it has raised almost $12 million from government grants and angel backers, does not have any institutional backing. The companies that have combined a computer-based simulation and modeling approach to systems biology with internal drug discovery programs have, not surprisingly, raised more venture capital than their in silico-only counterparts, but it remains to be seen whether the advantages that a systems approach to biology bring are enough to overcome the challenges of establishing a new drug development organization. At the same time, there’s something to be said for keeping companies small. Gerry Langeler, a general partner at Seattle’s OVP Venture Partners—one of the backers of VLST, Accelerator and ISB-spinout NanoString— echoes MPM’s Weissman when he says his firm is “relatively uninterested” in platform


companies with variations on fee-for-service models, which he sees as being unlikely to reach a scale that will make institutional backing a worthwhile investment. But that doesn’t mean companies in the space can’t succeed with that model. “I think there is sometimes a mistaken belief that unless you scale to a very large size, you haven’t been successful,” says Langeler. “But if you can build a $20-million-a-year company that’s throwing off $3 million to $4 million a year in profit, hats off to you. Don’t take my money, keep it for yourself! There’s a lot to be said for the modestsized company that may never be the big home run but can be a very successful enterprise for the entrepreneurs and maybe a few small angel backers.” That approach may also help young companies still seeking to prove the value of their platforms mature into something that can sustain large capital investment. “In this market, where the venture community is largely trying to de-risk their opportunities and look at triedand-true products, it’s hard to imagine how you get many systems biology companies funded,” acknowledges Weissman. Tomorrow’s systems biology successes may have to work outside the system. Karl Thiel, Portland, Oregon 1. Cook, J. Biotech startup VLST gets $55 million. Seattle Post-Intelligencer June 16, (2006). 2. Lawrence, S. Bioentrepreneur, published online 22 December 2005 (doi:10.1038/bioent897). 3. Thiel, K. Nat. Biotechnol. 22, 1087–1092 (2004). 4. Hendrickson, D. Mass High Tech, published online Sept. 9, 2005 http://masshightech.bizjournals.com/ masshightech/stories/2005/09/12/story5.html

1057

N E W S F E AT U R E based in Geneva—which states that all clinical studies should be registered, including phase 1 trials—would stifle innovation4. Furthermore, as most phase 1 trials are small and use healthy volunteers, sick patients wouldn’t need to know about them. “It is unclear how disclosing active Clinical trial databases are sprouting like weeds, but do they provide phase 1 trials would benefit patients,” says the information the public needs? Aaron Bouchie investigates. Goldhammer. Following PhRMA’s lead, the Biotechnology Industry Organization (BIO) in Washington, DC, encourages all of its members to regOn August 3, US Senators Enzi (R-WY) and include all trials that test for efficacy (excluding ister all “hypothesis-testing” trials (that is, Kennedy (D-MA) introduced legislation, which, only those early-stage trials that test for safety), late-stage, some phase 2 and all phase 3) on if enacted, could help bolster the public’s confi- not just for those that are experimental and for ClinicalTrials.gov. Most drug developers interdence in the drug industry and the government life-threatening diseases3. All 11 ICMJE member viewed for this article agree with this strategy, agency that regulates it. The bill, called The journals now require a trial to be registered at or although some still submit only those that are Enhancing Drug Safety and Innovation Act of before the onset of patient enrollment in order required by law. One notable exception is 2006 (S. 3807), calls for the establishment GlaxoSmithKline (GSK; Brentford, of a mandatory clinical trials registry UK), which was sued in June 2004 by and results database. In requiring that New York Attorney General Eliot Spitzer outcomes be included, such a registry for suppressing negative results from differs significantly from the existing clinical trials with the anti-depressant government database, ClinicalTrials.gov drug Paxil (paroxetine hydrochloride) which mainly lists ongoing clinical triin adolescents. GSK makes public all of als1. The Enzi-Kennedy bill is a response its active clinical trials including phase 1 to increasing public distrust of the drug on ClinicalTrials.gov. “We have decided industry and its oversight by the US Food to include all phase 1 trials in the puband Drug Administration (FDA) that is a lic registry to support the movement of result of recent high-profile drug safety transparency, which was led by WHO and debacles (Box 1). ICMJE,” explains Rick Koenig, GSK’s vice But the bill has sparked controversy. president of R&D communications. Public advocacy groups say it does not go GSK’s policy, though admirable, is far enough, whereas critics from indusnot required by law, nor will it be if the try say that releasing clinical trial data is Senators Mike Enzi (left) and Edward Kennedy introduced legislation in early August that would mandate the public Enzi-Kennedy bill passes. Although the unnecessary and may actually stifle inno- disclosure of late-stage clinical trial data. new bill would require the registration of vation. As outlined in the Enzi-Kennedy all late-stage trials, not just those that are bill, however, greater transparency of clinical trial data appears to offer little threat to to be considered for publication. This is no idle experimental and for life-threatening diseases, drug developers, as the most sensitive business threat considering heavyweights such as The it does not go so far as to require companies to information—that on early-stage, exploratory Lancet, The New England Journal of Medicine and report early-stage trials. The senators clearly have trials—will remain in companies’ hands. the Journal of the American Medical Association listened to industry’s interests by not including phase 1 trials in the bill’s registry requirements. are members. The current approach Most pharmaceutical and biotech companies The FDA Modernization Act of 1997 (FDAMA) have complied with the ICMJE’s policy, not just The call for greater transparency required the US Department of Health and so they can get published in reputable journals, A few months after Spitzer sued GSK, Merck Human Services to set up a registry of clinical but also because in excluding early-stage trials, of Whitehouse Station, New Jersey, voluntarily trials “of experimental treatments for serious sensitive business information is not revealed. withdrew its anti-inflammatory drug Vioxx or life-threatening diseases or conditions2.” To In explaining this exclusion, Alan Goldhammer, (rofecoxib) from the shelves because it increased achieve this, the National Library of Medicine vice president of regulatory affairs at the the risk of heart trouble. Although it created bad (NLM) launched ClinicalTrials.gov in 2002, the industry group Pharmaceutical Researchers publicity for the pharmaceutical industry, these primary purpose of which is to help patients and and Manufacturers of America (PhRMA) in two events also highlighted apparent deficienphysicians find information on nearby clinical Washington, DC, points out that phase 1 trials cies in the FDA’s post-marketing surveillance trials. According to the NLM website, the regis- are exploratory, or “hypothesis-generating,” and processes. The industry responded by launchtry currently contains ~31,700 clinical studies in the drugs being tested in this early stage are still ing a number of databases of clinical data on far from regulatory approval. “If you’re breaking their marketed products in hopes of improving over 130 countries. Many believe that, although ClinicalTrials.gov ground in a new therapeutic area, then listing its public image (Table 1 and Supplementary is a good start, it could (and should) do more to phase 1 trials would be telling the competition Table online). Clinical data can also be submitted to benefit the medical community. In September what you’re doing,” says Goldhammer (Box 2). For this reason, many in the drug industry ClinicalTrials.gov, for example, by linking to a 2004, the International Committee of Medical Journal Editors (ICMJE, a small working group believe the recommendations published in May, published journal article, although such discloof journal editors) called for the registry to 2006, by the World Health Organization (WHO) sure is not required by law. The Enzi-Kennedy Newscom/UPI Photo/Kevin Dietsch


Clinical trial data: to disclose or not to disclose?

1058




Box 1 Enzi-Kennedy bill basics In addition to the clinical trials registry and results database, the Enzi-Kennedy has three other elements: It outlines a plan to improve post-market monitoring of drugs by the FDA and companies. Before a drug can be approved, a company will be required to submit a risk evaluation and management strategy (REMS) that will help the FDA respond to risks identified after a product reaches market. Noncompliance results in fines of up to $250,000 per violation. According to FasterCures’ Simon, the FDA should receive additional appropriations to take on this added authority, rather than relying on user fees. It creates the Reagan-Udall Institute for Applied Biomedical Research, a public-private partnership that would foster the creation of a new generation of predictive tools to speed product development and increase safety. This institute would identify and coordinate research priorities and distribute grants. “The FDA analyzes drugs using technologies that are 20 years old,” according to Peter Pitts, president of the nonprofit Center for Medicine in the Public Interest. Thus, the institute would “help the FDA move to the edge of 21st century medicine.” It increases transparency and predictability in the FDA’s process for screening advisory committee members for potential financial conflicts of interest. Last month, the FDA announced it was looking into how to improve the process. C-Path’s Woosley says that it would be difficult to get balanced opinions without bringing in people with industry experience. “If you want an expert opinion, then you want that expert opinion, no matter where the person works,” he says. AB

bill, if passed, would mandate disclosure of results from some late phase 2 trials and all phase 3 and 4 trials. Under the legislation, failure to comply has dire consequences—it could hold up drug approval or the release of funds to trials funded by federal agencies. Even so, some patient advocates believe this bill does not go far enough, and that complete data transparency as soon as possible after the

trial is completed is necessary to benefit patients. For example, a patient looking to enroll in a trial should be able to base the decision on existing clinical data for all products that are in trials, argues Sidney Wolfe director of Public Citizen’s Health Research Group, a nonprofit watchdog based in Washington, DC. Wolfe also objects to a provision that allows companies to delay making trial results public for up to two years if it

Box 2 What is competitive business information in clinical trials? The typical clinical trial scenario—phase 1 for safety, phase 2 for toxicology and for determining dosage determination and phase 3 for efficacy—has evolved over the years. Now, press releases come out daily that describe a drug in a phase 1/2 trial, or phase 2b, or some other name that further breaks down the stage of clinical development. When determining which trials harbor competitive business information, it may be more useful to think in terms of two categories: hypothesis generating (also called exploratory) and hypothesis testing (also called confirmatory or pivotal). When a company is testing a drug, it performs lots of clinical trials to try out different delivery methods, indications and patient subpopulations. According to Hoover Institute’s Miller, “At last count, on average the results of more than 70 clinical trials are submitted by a corporate sponsor to support a submission to the FDA for approval to market a new drug, but generally only two or three of these are ‘pivotal’ trials that provide the required definitive evidence of safety and efficacy.” In other words, the pivotal trials, which would most likely be phase 3 or late-stage phase 2, would be hypothesis testing. The other 67 trials, mainly phase 1 and early phase 2, would be hypothesis generating, and data from these trials would not have to be made public under the Enzi-Kennedy bill. PhRMA’s Goldhammer says that companies aren’t as concerned about disclosing data from hypothesis-testing trials because they are “closer to the finish line.” At that point, the timeline to approval is not as long as when exploratory phase 1 trials are being done, so disclosure will have less impact on the company’s pipeline. Goldhammer also notes that companies have a fiduciary duty to investors and the Securities and Exchange Commission to disclose results of late-stage trials, because they can have a greater impact on the success of a company than phase 1 trials. AB


is applying for marketing approval or attempting to publish the data. “There needs to be the shortest amount of time possible between a trial ending and the data being made public,” says Wolfe. Art Caplan, director of the University of Pennsylvania Medical Center’s Center for Bioethics in Philadelphia says that such a database should be a requirement, not an option. He believes that companies have an obligation to make the data public. Patients entering clinical trials are promised that the results will be made known to help advance medicine, but companies often renege on that promise—especially if the results are negative. “This [database] fulfills companies’ promises to patients,” says Caplan. But not everyone agrees. Henry Miller, fellow at the Hoover Institution and the Competitive Enterprise Institute in Palo Alto, California, thinks that concerns about drug companies obscuring negative results are exaggerated. “Except for offering a bonanza to plaintiffs’ attorneys trolling for business, the benefit of a publicly available database of clinical trial results would be minimal,” says Miller. Nonetheless, some companies are already complying with the provisions in the bill through a voluntary database set up by PhRMA shortly after the Vioxx withdrawal. PhRMA recommends that its members make public “the results of all hypothesis-testing clinical trials…regardless of outcome.” Many big pharma companies, such as Lilly in Indianapolis, Indiana, Roche in Basel, Switzerland, GSK and AstraZeneca in London, publish such data on their websites as well (Table 1). Another provision of the Enzi-Kennedy bill is the requirement for summaries along with the raw data. Such summaries are important, according to Greg Simon, president of the biomedical think tank FasterCures of Washington, DC, because he is “more worried about burying the public in data sets and statistics.” Debra Aronson, director of BIO’s bioethics committee, believes that data is best presented to patients through peer-reviewed journal articles, but if not there, then the results and a summary should be verifiable before going into a public database. “I think there should be a peer-review process for such a database. I know some don’t like that answer, but that would be best,” says Aronson. Share and share alike Most agree that making phase 1 data public would not help patients. As around 80% of drugs fail at this stage, and for many drugs, safety data are obtained by giving the drug to healthy volunteers, such data would not benefit the public. Merrill Goozner of the Washington, DC’s Center for Science in the Public Interest says that in some cases, such as hormone

1059


Table 1 Selected clinical trial results databases Database/launch date

Organization

Description

Clinicaltrials.gov/2002

National Library of Medicine (NLM)

Mandatory registry of trials “of experimental treatments for serious or life-threatening diseases or conditions.” Companies can register other trials and submit results, but that is not required by law.

Clinical Study Results/2004 www.clinicalstudyresults.org

Pharmaceutical Researchers and Manufacturers of America (PhRMA, Washington, DC)

Voluntary results database of hypothesis-testing clinical trials regardless of outcome. Contains information on trials of over 200 drugs from about 50 companies.

SearchClinicalTrials.org/projected launch end of 2006 www.searchclinicaltrials.org

The Center for Information & Study on Provides access to multiple registries. CISCRP is a nonprofit with supClinical Research Participation (CISCRP, port from individuals, government and research institutions, foundaDedham, MA, USA) tions and corporations.

Eli Lilly and Company Clinical Trial Registry/2004 www.lillytrials.com

Eli Lilly and Company (Indianapolis, IN, USA)

Registers all their phase 2, 3 and 4 clinical trials at initiation and results of phase 1, 2 and 3 trials for all commercially marketed products when the drug is available for patient use. Posts any significant safety findings as soon as possible.

Clinical Trial Protocol Registry and Results Database/2005 www.roche-trials.com

Roche (Basel)

Registers all their phase 2, 3 and 4 clinical trials that are ongoing and data from phase 2, 3 and 4 ‘confirmatory’ trials.

AstraZeneca Clinical Trials/2005 www.astrazenecaclinicaltrials.com

AstraZeneca (London)

Registers all their ongoing hypothesis-testing trials and results from all hypothesis-testing trials for its marketed products.

GlaxoSmith Kline Clinical Trials Register/2004 http://ctr.gsk.co.uk/welcome.asp

GlaxoSmithKline (Brentford, UK)

Holds data and summaries from all their clinical trials, including phase 1, for all marketed products. Results are posted for nonmarketed products if GSK sees a safety problem related to mechanism of action.


http://clinicaltrials.gov/

Sources: Organization websites and organization spokespersons.

replacement therapies, efficacy might be seen in phase 1 trials. In these cases, Goozner believes that the data should be made public, but these instances are very rare. The WHO cited a more recent clinical trial disaster (Würzburg, Germany-based TeGenero’s phase 1 antibody trial, in which six healthy volunteers experienced severe immune reactions that has left some with lasting medical problems) as another reason why the public’s trust in the drug industry is waning and how full transparency is necessary to regain that trust. Goozner believes the issue with phase 1 safety is an issue of communication among companies, more than an issue of making data public. The biotech and pharma industries could benefit from sharing such data for all phase 1 failures, even when the effects are not as drastic as those seen in the TeGenero study, by eliminating the duplication of dead-end studies. By sharing some details of their failures, the knowledge base of medicine would grow much faster, and the drug industry as a whole would become more efficient. “Competition in business is understandable, but science doesn’t work that way. Failures advance the field,” says Goozner. Ray Woosley, head of the Clinical Path Institute (C-Path) in Tucson, Arizona, agrees that companies need to learn from each other’s mistakes. “C-Path was created to do just that,” he explains. Woosley points to the institute’s Predictive Safety Testing Consortium as an example of such collaboration. Although the consortium was just launched in March, already 14 companies are working on a ‘precompetitive’ way of developing better preclinical safety tests. And getting companies to collaborate in a

1060

similar way on phase 1 data would be the next step, says Woosley. “If companies find the current efforts good, then I will approach pharma about clinical data,” he says. Caplan believes mishaps could be avoided by the FDA if they were given a little more money to be more vigilant. “The FDA by law gets phase 1 safety stuff and they should be much more aggressive about sharing it with others even if corporate or researcher secrets are jeopardized,” says Caplan. “In terms of human subjects, they [the companies] should understand the point of the study is to generate safety information and how that will be shared with the FDA—that is the goal of phase 1 studies—not [general] knowledge,” says Caplan. BIO’s Aronson worries that such data sharing of phase 1 trials could harm biotech firms, however. “Biotechs rely on venture capital money, and venture capitalists are investing in intellectual property. It would be hard to get investors if all your development ideas were shared with your competitors,” she says. There is always a balance to be kept between the need to share information so that others can use it and learn from it and the need to keep some information protected so that the idea can be developed into an innovative therapeutic, she adds. But who determines which data are shared to help progress the field and which are kept protected? “Establishing that balance is sometimes difficult and often will depend on the timing of disclosures,” says Aronson. Post-marketing blues The decision to deemphasize disclosure of early-phase trial results in the Enzi-Kennedy

not only mollifies company and investor concerns about competitiveness, but also may result in efforts being focused on what many see as the more serious problem. “The weakest part of regulatory oversight is once products get on the market,” according to Caplan. He cites the safety problems of Merck’s Vioxx and of a cardiac pacemaker from Minneapolis-based device manufacturer Guidant as examples of the FDA’s lack of teeth. “Anyone who thinks the current system is working is dreaming,” says Caplan. In this respect, there already may be a solution in the wings. Woosley thinks the Agency for Healthcare and Research Quality (AHRQ), in Rockville, Maryland, would be ideal to fix the problem. AHRQ gets about $300 million a year to fund 11 Centers for Education and Research on Therapeutics (CERTs), which are congressionally mandated to perform post-marketing studies on drugs, such as head-to-head comparisons that companies tend to avoid. Postmarketing studies are not gathered and made public, according to Woosley, and CERTs could play a role in helping patients understand their therapies. “The FDA is a passive system, driven by what people bring to it,” he says. Right now only the FDA and companies are educating the public about drugs. Vioxx and Paxil have shown us that system isn’t nearly enough. “What is missing is a learned intermediary,” explains Woosley. Aaron Bouchie, New York City 1. Clinical Trials.gov. http://www.clinicaltrials.gov 2. FDA Modernization Act of 1997. http://www.fda.gov/ cber/fdama.htm 3. De Angelis, C. et al. Ann. Intern. Med. 141, 477–478 (2004). 4. Sim, I. et al. The Lancet 367, 1631–1633 (2006).


The promise of the East: India and China as R&D options Simon Goodall, Bart Janssens, Kim Wagner, John Wong, Wendy Woods & Michael Yeh The East provides increasing opportunities for biotech companies seeking to optimize product development and accelerate time to market. But any undertaking in China or India requires close scrutiny of the risks.

S

mall to medium-sized enterprises (SMEs) in the biotech sector face a long, arduous journey toward successful commercialization of early-stage products. To get their products to market more efficiently and to realize their true commercial potential, biotechs are looking for new resources to tap for a productivity boost, and for new markets for their products. If pursued wisely, one of the most promising and practicable solutions is the sourcing of selected tasks to Asia, particularly to India and China. Both countries have already attracted considerable investment and involvement from pharma multinational corporations and could provide smaller biotechs with comparable opportunities. Consider some of the potential advantages: a huge and inexpensive talent pool (each country produces annually more than three times as many chemistry graduates as the US does), including an increasing number of Western-trained returnees; a vast patient population available for clinical trials; strong government support for biotech, both through investment (as in science parks) and

Simon Goodall is at The Boston Consulting Group, 355 South Grand Avenue, Suite 3200, 32nd Floor, Los Angeles, California 90071, USA; Bart Janssens is at The Boston Consulting Group, 14th Floor, Nariman Bhavan, 227 Nariman Point, Mumbai 400.021, India; Kim Wagner is at The Boston Consulting Group, 430 Park Avenue, New York, New York 10022, USA; John Wong is at The Boston Consulting Group, 34th Floor, Shell Tower, Times Square, Causeway Bay, Hong Kong, China; and Wendy Woods and Michael Yeh are at The Boston Consulting Group, Exchange Place, 31st Floor, Boston, Massachusetts 02109, USA. e-mail: [email protected]

Research (Chemistry)

Research (Biology) Target identification Genetic research Key activities and technologies


BUILDING A BUSINESS

Target validation

Compound generation and assay development

Functional genomics

Analog prep

Proteomics Protein biochemistry

Bioinformatics

Disease models

Structural chemistry

Expression profiling

Genetically modified mice

Analytical chemistry

Basic molecular biology technologies

Screening

Compound synthesis

Assay execution

Pharmacology

Clinical management

SAR evaluation

PKDM

Data management

Toxicology

Regulatory

Synthesis

Chemoinformatics

Clinical Preclinical

Lead optimization

Phase 1–4

HTS/UHTS Drug design Assay development

Bioimaging

Medicinal chemistry Cell-based models for efficacy

Animal models for efficacy

Common service Less common offerings service offerings India China

Figure 1 Indian and Chinese partnering opportunities along the R&D value chain. HTS/UHTS, high throughput screening / ultra-high throughput screening; SAR, structure-activity relationship; PKDM, pharmacokinetics and drug metabolism

through policies (such as tax concessions); and increasing private-sector funding and involvement. By making shrewd use of these attributes and actively working to manage the risks, your biotech could conduct operations in a leaner, more cost effective and perhaps faster way. There are dangers, however, and a considered approach remains the watchword. First, any involvement in the region should be undertaken as part of a global R&D strategy, not as ad hoc and opportunistic forays. Then, you need to think of a regional strategy, not a countryspecific strategy: the opportunity is a matter of China and India, not China or India. You need to consider that the offshoring process, though designed to ease the challenges and expenses of R&D, labors under its own set of complexities and inefficiencies. Although some opportunities will likely suit your company, others, equally appealing, might not, so you need to make precise evaluations each time. And even the surest opportunity involves possible


risk—most notably the risk to intellectual property (IP) and the chance of delays through red tape. Biotech SMEs may have more to lose with offshoring than a large pharmaceutical concern does, as they may lack the scale to tolerate IP theft or the failure of an outsourcing venture. They can also ill afford the diversion of internal resources to find the right set of sourcing partners or opportunities. The potential benefits do look increasingly viable, but at this point they remain more potential than proven. If you are seriously considering outsourcing to India or China, you need to start moving toward an integrated and effective strategy. There are three key issues to consider when doing so: your motivation in investing, the location of investment and the risks inherent in the activity. Motivations for investment The four likeliest motives for offshoring work to China and India are saving on R&D costs,

1061


BUILDING A BUSINESS reducing capacity bottlenecks, accessing talent and increasing market access. When broaching a strategy, you first need to clarify what weight to give to each of these motives. And then you need to assess how the available Indian and Chinese opportunities measure up in each case. The advantages of cost cutting go without saying, but offsetting them are the dangers inherent in any form of outsourcing: the possible need for greater supervision, and the potential for slower and lower-quality output. Reducing capacity bottlenecks is particularly advantageous for resource-strapped firms; by offshoring lower-priority projects, they can concentrate on higher priorities. Similarly, accessing talent to fill gaps as needed should give biotechs the freedom to concentrate on their core strengths. As for increased market access, the advantages again go without saying. Although the market is currently modest, its potential is very sizeable. Locating the investment Although India and China both offer outsourcing opportunities across all phases of the innovation value chain, the capabilities are uneven, and some of the more complex activities remain out of reach (Fig. 1). But don’t make any assumptions: new skills and resources keep coming online. A year ago, you would scour both countries in vain for preclinical services of US Food and Drug Administration/good laboratory practice (GLP)-quality; today, Bridge Pharmaceuticals in Beijing, or CDRI in Lucknow, India, will be happy to oblige. And if you need target discovery or validation, you could try various providers in Zhangjiang Life Science Park near Shanghai, or Triesta Sciences in Bangalore. Though less advanced overall than vendors in the developed world, Asian vendors have a clear advantage when it comes to price, offering cost savings of at least 60% in many areas, such as basic chemistry or clinical trials. Just make sure each time that those cost savings aren’t going to be canceled out by extra administrative expenses on your side, or lower productivity on the provider’s. With the right provider, you should be able to ease some of your pipeline bottlenecks and capacity constraints at a stroke. Which country to choose for any particular activity or project? And which to give greater emphasis to when devising a strategy? As things stand today, India’s greatest value is in giving you quick access to specific drug-development resources, so it might prove the better bet if your priorities are shorter time frames, easy setup, rapid results and very high cost savings. China’s main attraction is in potentially strengthening your foothold in its huge and

1062

fast-growing biopharma market, so if you have a particularly commercial agenda—developing government contacts, for example, with an eye to increasing market access—you would probably opt for China. And if you have a longer time frame, you might also favor China, and pursue lengthier projects there through an alliance partner, perhaps one of the prestigious government-funded research institutes. But a fully rounded strategy will leverage the assets of both countries, rather than just one of them, taking full advantage of their differences. In capabilities, China is considerably ahead in biology, though still at a modest level compared to developed nations standards. Chinese scientists participated in the Human Genome Project, and have made some notable advances in gene therapy and stem cell work. In 2003, Shenzhen-based SiBiono GeneTech was granted the world’s first license for a gene therapy medication. In chemistry, on the other hand, India arguably has a solid lead, with some vertically integrated suppliers now able to offer end-to-end services. As for clinical trials, India once again is quicker off the mark, with contract research organizations typically able to secure approvals and get launched within 3 to 4 months, against a norm of 9 to 12 months in China. India also possesses superior strengths in information technology–dependent areas, most notably biostatistics and clinical trials data management. There are also some broader considerations. India has the unquantifiable benefit of very high proficiency in the English language. And arguably, its managerial and scientific/edu-

cational culture is more Westernized than China’s—more open to breaking with tradition and more innovation minded. That said, Chinese scientists with advanced training from Western institutions are returning at everincreasing rates, often to take management positions at Chinese biopharma companies. What’s more, China has the distinctive strategic benefit of increased commercial potential for biotech products themselves (see Box 1). Companies that invest in China stand to enhance their commercial prospects by impressing doctors, key opinion leaders and officialdom. By raising technology standards in the country, R&D investors will earn government goodwill that could raise their chances of expedited approvals and easier market access. The risks, singly and jointly On the downside, there are risk factors specific to each of the two countries. If operations are ever disrupted by workforce disputes or animal-rights activists, that would be in India; if by government interference, that would more likely be in China. The infrastructure is also far more reliable in China; India still suffers from interrupted power supplies, antiquated ports and inadequate highways in many regions. China’s GLP standards are still evolving, and lag behind those of India—with few labs in either country being internationally GLP-approved. And the bureaucratic hurdles differ: the Indian authorities grant approvals for clinical trials far faster than their Chinese counterparts. But at the preclinical stage, Indian regulations are particularly stringent, making it difficult for

Box 1 The region’s market for biotech products The markets for biotech products in China and India are quite different from those in the developed world—with a far lower proportion of consumers who can pay even a tiny fraction of Western prices. But given the high rate of growth of the region, especially within the middle class, the opportunity may eventually be a lucrative one, especially in China. China’s overall pharmaceutical market is already 2–3 times more valuable than India’s, and will remain so. It should rise from $12 billion in 2005 to a predicted $37 billion in 2015 (graduating to become the world’s fifth most valuable market en route), against India’s $5.3 billion and $16 billion, in 2005 and 2015, respectively. What’s more, the proportion of generics (currently over 70% by value in both markets) versus branded drugs is declining more steadily in China than it is in India. And the price realization, though lower than that of developed nations, is considerably higher in China than in India. In each country, the target market for high-priced biotech drugs is probably no more than 5% of the population—those with private health insurance. Still, that’s 5% of a billion-strong population in each country. It all adds up: sales of biotech products in China reached $2.5 billion in 2005. Drugs that qualify as blockbusters in the United States can reach annual sales of $50 to $100 million in China with rapid success. GlaxoSmithKline’s (Brentford, UK) Heptodin (lamivudine) reached $80 million in annual sales in China within five years of launch. That said, the commercial factor is less a current consideration than a future one. Biotechs about to launch new products, at least for the next few years, may best be advised to outlicense them to established pharma companies with proper scale in China or India. M.Y.



BUILDING A BUSINESS laboratories to source genetically modified animals and to import and export human tissue or blood samples. Viewed more broadly, the main risks apply to both countries: red tape and insecure IP. In each case, the two governments have taken corrective steps, easing the bureaucratic constraints and tightening the IP statutes. How these measures translate into reality isn’t yet clear. There are cultural and human factors at work, not just regulatory ones. Western ideas of urgency and privacy may take some time to permeate. Although laws that approach Western standards now exist, their enforcement in the realm of biopharma, especially in biologicals has not yet been established (see Box 2 for further details). Biotechs can reduce their IP risk in both India and China through proactive management. First, you should carefully weigh the critical value of the IP against the perceived benefits of entering India or China, and refrain from any project with an unfavorable balance. When selecting a partner or vendor, you should make all necessary due diligence evaluations of the candidates on your shortlist. In particular, check on their IP-protection measures—physical, electronic, and other. One biotech, for instance, disables its printer drivers and tracks all data downloads. Some local vendors literally erect ‘Chinese walls’—separate rooms and facilities for client activities—and even withhold the client’s name from the workforce. And when negotiating contractual arrangements, you should ensure that legal recourse, both local and abroad, is properly registered. Vendors such as Beijing-based Bridge Pharmaceuticals and Aurigene in Bangalore maintain US-based operations in part to give assurance that they comply with all US IP regulations—and to give customers the option of pursuing US-based litigation if they don’t. Even if not offshoring work to India or China, biotechs might still consider it prudent to protect their most valuable and vulnerable IP in these countries. By licensing IP to Chinese or Indian companies, they stand a better chance of preempting patent infringement, or of being represented by a party with a ‘home court’ advantage in case of litigation. Choosing a sourcing model Let’s assume that after weighing the risks and potential benefits scrupulously, you’ve decided to take the plunge, or at least to test the water. You now need to choose an optimal business model. There are three basic models—outsourcing, partnership and captive investment

Box 2 IP developments in India and China Among executives contemplating offshoring, IP protection remains a key concern, especially for discovery work. The main IP laws in both India and China are new and relatively untested, so caution is appropriate. After major changes in India’s IP laws in April 2005 that shifted from process to product protection, India now appears to have a reassuringly tough set of IP standards. Strong trade secret laws and the new Contract Act, based closely on IP statutes in the UK, protect a company against risks related to information leakage or employee switching. In addition, they allow companies to pursue litigation in Western courts against Indian companies for IP breaches. Another source of comfort is the presence of R.A. Mashelkar, director general of the Council for Scientific and Industrial Research. Mashelkar is a leading proponent of biotech partnerships and a global authority on IP protection in developing nations, serving as vice chair of the Commission on Intellectual Property Rights, Innovation and Public Health for the World Health Organization (Geneva). Although India’s new IP laws have appeared to work well in other industries, such as business process outsourcing, which handle sensitive company data, it remains to be seen if they will work as well for biotech and for biological products. After all, the Indian pharma industry as a whole does have a tradition of patent challenges and deep reverse-engineering skills. China too has a strong set of IP protection laws in place, though perhaps not quite as strong as India’s overall, and perhaps not quite as strong for biologicals as for chemical molecules. Enforcement has been an ongoing issue, and the judicial protection of IP still has to prove itself. But since its accession to the World Trade Organization in 2001, the country has been subject to the Agreement of Trade-Related Aspects of Intellectual Property Rights, so the government is under pressure to enforce international standards. Its previous efforts to change underlying attitudes toward IP protection were not unqualified successes: the patent process remains awkward, Chinese courts continue to struggle with IP cases and protection is not always applied equally across domestic and foreign parties. M.Y.

model—offering different degrees of flexibility and control. For biotech SMEs, the starting point would generally be the outsourcing model: hands-off and low-commitment, and therefore involving minimal supervision and easy entry and exit. Of course, it also involves minimal control over output and IP, and for those dual reasons the projects outsourced would tend to be low-complexity work of less strategic import. Once your company has gained confidence and has decided upon a longer-term commitment to the region or to a particular vendor, you may choose to advance to a partnership model, assigning projects of higher complexity or greater breadth to a Chinese or Indian provider, with more of your own participation in supervising, training and monitoring. This would afford you greater control over quality and should improve communication and trust. However, by moving your partner up the learning curve, you risk finding that they use the enhanced know-how of their workforce to serve potential competitors of yours. The most committed model, captive investment—where a company acquires and operates its own R&D base in China or India—is unlikely to be adopted by smaller or cash-constrained biotechs. It certainly affords increased


control and IP security, but at the cost of a heavy investment of time and resources. It also means a host of new responsibilities. There is no longer a streetwise local intermediary to deal with red tape or make good any unexpected infrastructure gaps. One biotech that set up a captive base in China admits ruefully that it has had to manufacture its own rodent cages. Finding the right partner To match corporate investors with the right vendor or collaborator, both India and China have quasi-official dating agencies. In China, you would approach the administration in any of the biotech parks, and they would recommend a suitable match from the list of firms based there. In India, you would approach the Ministry of Science and Technology’s Department of Biotechnology or the Council for Scientific and Industrial Research, and they would fix you up with a potentially ideal partner. But it’s worth ranging far wider than these sources. After all, finding the right partner will make a big difference to your offshoring experience, so don’t stint on the time and effort invested. In both countries develop ‘guan xi’—good relations with influential people—to get the best advice and also some help in sealing the deal. Investors and providers are heavily networked, and you should link in to these

1063


BUILDING A BUSINESS networks right up to the last minute, as the landscape changes quickly. At this early stage, biotechs can afford to be cautious and methodical in their approach, as limited vendor capacity is not currently an issue. Over time, vendor capacity should grow to keep pace with demand, with perhaps more of a focus toward smaller biotechs as the sourcing market develops. That said, the earlier you take the plunge, the sooner you can reap the cost savings and the better your chances of accessing proven and established vendors. Looking ahead The virtue of the sourcing option goes beyond cost and time efficiencies. Biotech talent and drive are increasingly abundant in China and India, and innovative ideas, which can’t be far off, will be equally amenable to tapping. After all, the governments of the two countries aren’t investing in biotech to create sourcing opportunities but to establish vigorous high-tech industries of their own. Specific areas of China and India represent rapidly growing clusters of biopharma

1064

expertise and may ultimately be as important to biotech as San Diego or the Bay Area are. You only need to look at all the innovation emerging from the Taiwanese computer industry to see the parallels with Indian and Chinese biotech, and the pattern of success that the countries are sure to emulate. Small Western biotechs with large ambitions and a taste for adventure can get in at the ground floor and harness Asian innovation, rather than simply offshoring their own. One other possibility that India and China are opening up is a new model of biotech product development (and perhaps of manufacturing, too). Call it the ‘modular model,’ a kind of decentralized R&D system where different aspects of R&D are distributed globally and conducted almost autonomously in different locations. But you don’t have to look that far ahead. The opportunities in China and India are rapidly developing, with key pieces falling into place. Weigh the options carefully, delve into the realities and risks of operating within the two countries and decide carefully if you

want to enter. If you do, devise a precise and methodical strategy, find the right partners and implement the strategy with full commitment. With the right strategy, you stand to give your biotech SME a productivity boost and a handsome competitive advantage. ACKNOWLEDGMENTS While most of the material in this article derives from client work, it is backed by a detailed survey conducted by the Boston Consulting Group in 2005 and 2006, collating the views and experiences of executives at over 90 vendors in China and India and of officers at several government research institutes in the two countries, and of senior executives at over ten biopharma MNCs operating there. A report summarizing the findings from this study (Looking Eastward: Tapping China and India To Reinvigorate the Global Biopharmaceutical Industry, August 2006) along with other publications on the opportunities for biopharma R&D in India and China can be found at http://www.bcg.com This story was reprinted with some modification from the Building a Business section of the Bioentrepreneur web portal (http://www.nature.com/bioent), 25 July 2006, doi:10.1038/bioent910.



CORRESPONDENCE

How to stay out of a BIND To the editor: Your very sympathetic editorial in the February issue (Nat. Biotechnol. 23, 215, 2006) regarding the demise of the Biomolecular Interaction Network Database (BIND) assigns the blame for this resource’s passing to “...bureaucratic delays [and] government fiscal nitpicking....” and calls on science funding agencies to provide more long-term funding for databases. Worthy as your crusade to better direct my tax dollars may be, I don’t find BIND to be a particularly suitable poster child for the effort. According to your account, BIND, via the Blueprint Initiative, burned through $25 million in about two years. Even in Canadian dollars that burn rate is nothing short of shocking, especially given BIND’s relatively modest scope, and the ease with which its data were to be ‘scraped’ from a relatively small number of scientific publications (I have quite a bit of professional experience in this domain, so I say this with some insight.) Personally, I admire Genome Canada’s decision to stop the bleeding. I’m sure there were, and are, those who have found BIND useful. Whether or not it was another $20.8 million worth of ‘useful’ or a total of $46 million worth of useful, given all the other worthy scientific uses to which that sum could be put, was the question, and Genome Canada decided this in the negative, citing concerns regarding management, budget justification and financial plan— concerns your editorial brushed aside without comment. A happy consequence of Genome Canada’s decision is that BIND is now where many such efforts belong. . . in private hands (albeit under the same management), where the rigors of the marketplace can impose upon its owners some deep regard for efficiency and utility. If BIND is truly valuable, then

Christopher Hogue can charge users a modest access fee; perhaps research funding agencies will view their grantees’ carefully justified requests for these small sums with favor. He may then use such hardwon revenues prudently to sustain and improve the product. If, on the other hand, BIND isn’t a particularly important resource, then users won’t be willing to pay, and it will pass on. This is as it should be. Much the same may be said for the Alliance for Cellular Signaling’s Molecule Pages, which never really amounted to much (numerically, at least). Now under Nature Publishing Group’s cost- and profit-conscious guidance they will, no doubt, either flourish or fold. Rather than arguing for the importance

of long-term database funding by granting agencies, BIND’s saga in fact argues for greater caution and more demanding oversight when these agencies elect to fund a database’s initial development. Realistic plans for long-term sustainability must be demanded, as must some basic enterprise management ability on the grant recipient’s part. Such expectations are anything but fiscal nitpicking; they are a fiduciary responsibility. I have no bone to pick with researchers who bemoan the intermingling of capitalism and scientific research (if, in this Bayh-Dole era, there’s anyone left who can still do so with a straight face). But those who feel this way should be prepared to make every precious tax dollar go as far as it possibly can. Those who fail at this should be quicker to blame themselves, and slower to blame ‘bureaucrats’. William B Busa Busa Consulting, Renfrew, Pennsylvania, 201 Johns Schools Road, Renfrew, PA 16053, USA. e-mail: [email protected]

The dog as a cancer model To the editor: The dog has long been used as a model in drug discovery and development research because of its similarities to human anatomy and physiology, particularly with respect to the cardiovascular, urogenital, nervous and musculoskeletal systems. Compared with other animal models, it may also prove invaluable in research and development on cancer drugs, because dogs naturally develop cancers that share many characteristics with human malignancies. The completion of a high (7.5×) coverage canine genome1 now paves the way for the development of critical resources that will allow the integration of naturally occurring canine cancers within the mainstream of cancer research. To initiate and facilitate collaborative efforts and leverage the opportunities provided by the dog in


cancer research, scientific and clinical leaders from both human and veterinary oncology have come together to form a multidisciplinary consortium, the Canine Comparative Oncology and Genomics Consortium (CCOGC). Cancers in pet dogs are characterized by tumor growth over long periods of time in the setting of an intact immune system, interindividual and intra-tumoral heterogeneity, the development of recurrent or resistant disease, and metastasis to relevant distant sites. In these ways, dog cancers capture the ‘essence’ of the problem of human cancer in a manner not possible with other animal model systems. Compared with other large animals commonly used in biomedical research, such as pigs and nonhuman primates, an additional advantage offered by pet dogs is that they are cared for into the ages

1065


CORRESPONDENCE commonly associated with the highest risk for cancer. This risk, coupled with their large population size (>70 million in the United States), results in a cancer rate sufficient to power clinical trials, including assessment of new drugs. Using crude estimates of cancer incidence, in the United States alone, there are ~4 million new cancer diagnoses made each year in dogs2. Examples of these cancers include non-Hodgkin lymphoma, osteosarcoma, melanoma, prostate carcinoma, lung carcinoma, head and neck carcinoma, mammary carcinoma and softtissue sarcoma. For many of these cancers, strong similarities to human cancers are seen, including histological appearance, tumor genetics, biological behavior and response to conventional therapies. The compressed course of cancer progression seen in dogs allows timely assessment of new cancer therapies. With the recent release of the canine genome sequence, the dog is now also amenable to comparative genomic analysis. Indeed, preliminary assessment of the canine genome suggests that the dog and human lineages are more similar than the human and rodent lineage in terms of both nucleotide divergence and rearrangements. The CCOGC initially plans to take advantage of these opportunities through the following actions: • Develop a robust and well-annotated biospecimen repository of canine cancers and tissues—funding of a large, accessible biospecimen repository is difficult using existing resources. • Improve opportunities to link the efforts of veterinary and comparative oncologists with the work of basic cancer researchers and clinicians. • Initiate non-clinical trials using pet dogs with cancers that are integrated into the development path of new cancer drugs. Mechanisms for review of these non-clinical trials by regulatory bodies should be developed such that information from these studies, where appropriate, may help to focus the scope of early human clinical trials. To date, non-clinical studies in dogs with cancer have answered questions that would have been difficult or impossible to answer in either mice or humans. The lack of goldstandard veterinary treatments also provides the opportunity for the early and humane evaluation of new therapies for dogs with

1066

the development of better cancer drugs for humans and other pet dogs. An opportunity window now exists. With the realization of the need for more useful animal models in human cancer drug development, the organization of a number of consortia and collective groups, the completion of the canine genome sequence, the increasing availability of dogspecific biological reagents and investigative methodologies, (e.g. antibodies specific for dog proteins or dog-specific oligonucleotide arrays) and the interest of the animal health biotech and drug industry, the CCOGC hopes to further stimulate efforts to fully exploit the many advantages of the dog in cancer drug research.

The 2.4-billion-bp (7.5× coverage) sequence of a female boxer dog (pictured) published in December 2005 (ref. 1), together with that of a poodle sequence released in 2003, should facilitate the use of dogs in cancer studies.

cancer. Following institutional review of trials, pet owners would be given the option to enter their dogs into clinical trials and in so doing receive access to novel cutting-edge treatment options for cancer, many of which are less toxic than conventional treatment options currently available. Accordingly, studies in pet dogs offer opportunities in both human and animal healthcare. First, pet dog trials will help better define the safety and activity of new anticancer agents. They may also assist in the identification of relevant biomarkers associated with response or exposure to these drugs. Furthermore, these studies may allow rational development of combination strategies that will improve the success of these new drugs in human clinic trials. These data may be useful before the filing of an investigational new drug application (IND) at the US Food and Drug Administration (FDA; Rockville, MD) and as means to optimize the development of anticancer agents currently in early human trials. Second, data generated through such studies may inform the development of new cancer treatments for animals. Research and development of new anticancer treatments is increasingly recognized as an area of need in the field of animal health. In this way, pet dogs with cancer will be directly helped through access to new these new drugs; results may be translated and extended to

Chand Khanna1, Kerstin Lindblad-Toh2, David Vail3, Cheryl London4, Philip Bergman5, Lisa Barber6, Matthew Breen7, Barbara Kitchell8, Elizabeth McNeil9, Jaime F Modiano10, Steven Niemi11, Kenine E Comstock12, Elaine Ostrander13, Susan Westmoreland11 & Stephen Withrow3 1Comparative Oncology Program, Center for

Cancer Research, National Cancer Institute, 9610 Medical Center Drive, Room 315, Rockville, Maryland 20815, USA. 2Broad Institute of Harvard and Massachusetts Institute of Technology, 320 Charles Street, Cambridge, Massachusetts 02141, USA. 3Animal Cancer Center, Colorado State University, Fort Collins, Colorado 80523, USA. 4Department of Veterinary Biosciences, The Ohio State University, Columbus, Ohio 43210, USA. 5The Animal Medical Center, New York, New York 10021, USA. 6Department of Clinical Sciences, Tufts University School of Veterinary Medicine, North Grafton, Massachusetts 01536, USA. 7Department of Molecular Biomedical Sciences, College of Veterinary Medicine, North Carolina State University, Raleigh, North Carolina 27606, USA. 8Center for Comparative Oncology, Michigan State University, East Lansing, Michigan 44824, USA. 9Department of Veterinary Clinical Sciences, University of Minnesota, St. Paul, Minnesota 55108, USA. 10Integrated Department of Immunology and AMC Cancer Research Center, University of Colorado at Denver and Health Sciences Center, Denver, Colorado 80214, USA. 11Center for Comparative Medicine, Massachusetts General Hospital, Charlestown, Massachusetts 02129, USA. 12University of Michigan, 5111 Cancer Center, Ann Arbor, Michigan 48109, USA. 13National Human Genome Research Institute, National Institutes of Health, 50 South Drive, MSC 8000, Building 50 Bethseda, MD 208928000, USA. e-mail: [email protected] or [email protected] 1. Lindblad-Toh, K. et al. Nature 438, 803–819 (2005). 2. Vail, D.M. & MacEwen, E.G. Cancer Invest. 18, 781–792 (2000).


CORRESPONDENCE


GM sterile mosquitoes—a cautionary note To the editor: The article in your November issue by Andrea Crisanti and colleagues (Nat. Biotechnol. 23, 1414–1417, 2005) reported the development of a transgenic strain of Anopheles stephensi, an Asian malaria vector, that the authors suggested may be useful as a sexing strain in a sterile insect technique (SIT) program against this vector. The SIT relies on the release of massive numbers of sterilized male mosquitoes to reduce the reproductive capacity of wild populations that transmit malaria1–4. Sterile females can still transmit disease, hence the need for efficient sex separation systems. It is beyond doubt, therefore, that this new methodology addresses an important need of mosquito SIT programs currently under development. As can be seen from the data, the use of this sexing method under experimental smallscale conditions was successful, but we do wish to respond to the suggestion that this methodology, and even this strain, can immediately be transferred to a large-scale SIT program. On the basis of our experience with the development of comparable systems in other species, we expect that strain evaluation(s) will have to be extremely thorough and carried out under appropriate conditions before it will be possible to judge whether a strain or particular sexing procedure is suitable for use in mosquito control programs integrating the SIT. These strains will have to be reared at high levels of production and for an extended period of time before sufficiently reliable and realistic data on the overall fitness, the accuracy and efficiency of the sexing procedure and the stability of the sexing system become available. In addition, the field performance of these strains will need to be evaluated. All these data will be used by decision makers to weigh any potential negative characteristics of the strain(s) against the benefits they provide, and only then can a judgment on the suitability of a particular strain(s) for inclusion in an SIT program be made. Radiation-induced sterility provides

some level of risk mitigation when transgenic insects are released, and this approach has been proposed for a first evaluation of the use of this technology5. In operational programs, where insect competitiveness is a key factor for success, there is currently a trend to reduce the radiation dose to a level that maximizes the sterility induction in the wild population. In the case of transgenic strains, however, this level will depend on regulatory requirements and the type of strain that is being released; for example, what type of transgene is used, in combination with what operational strategy (that is, eradication versus suppression). It is conceivable that a lower dose chosen for a conventional strain would not be appropriate for a transgenic strain. More troubling is the perception, fuelled by comments made in the press, that any efficient transgenic sexing strain can be easily incorporated into a mosquito SIT program without much further consideration. Although release of sterile males for mosquito control has been practiced in the past, direct inclusion of modern biotechnological approaches such as transgenesis should not be taken for granted. This technology needs to be considered systemically-holistically and be integrated into a broader social context6—a notion that larger development agencies like the World Bank (Geneva, Switzerland) have recognized for many years, but which still appears to have eluded some scientists and funding bodies. Mosquito genetic control specialists have been discussing the merits and limitations of modern biotech for over five years from a molecular genetic7, ecological8 and transitional9 perspective. It is evident from these discussions that only when the benefits are judged to outweigh the publicly perceived risk of the technology10 will the release of genetically modified (GM) mosquitoes become a reality. Thus, it is imperative that important stakeholders, in particular endbeneficiaries, participate in the scientific and development process11. If not, millions will be poured into technologies that are


not acceptable or feasible, betraying those most in need. A participative-iterative-strategic approach to malaria control is necessary to cope with intrinsic uncertainties of the interventions and changes of the ‘environment’12. The inclusion of ethical, legal and social aspects in this debate has been rudimentary at best. Although it is argued that GM insects for disease control is still in its infancy, we contend that several negative developments in the field of GM organisms may seriously impede the future applicability of this approach. Given the intricacies of stakeholder management, even in developed parts of the world13, we propose a three-pronged strategy to anticipate potential antagonism. The first and most critical step will be to gain public support. The establishment of trust through openness and direct involvement of stakeholders, including public authorities and the press, in the decision making process will be critical. Failure in this regard could result in the polarization of viewpoints and scaremongering; indeed, in India in the 1970s claims of biological warfare in the press led to the abandonment of a World Health Organisation (Geneva, Switzerland)-funded mosquito genetic control program just two days before the start of releases and after several years of research of development14. To prevent history from repeating itself, the establishment of equitable partnerships with scientists in disease-endemic countries combined with the transfer of ‘problem ownership’ are necessary. Scientific funding agencies should appreciate the complexity of such issues and the resulting need to communicate through other means in addition to the peer-review process, as rationality and reductionism are embedded in the scientific method and culture15 and are not necessarily perspectives required to tackle complexity. This would lead to a research agenda also driven by developing nations. A second need relates to oversight. The search for potential field sites to release transgenic mosquitoes is currently proceeding, backed by hastily established and cosmetic partnerships with scientists and institutions in situ. It follows that in the absence of any governance over this process and research progression in years to come, serious problems may develop. Inasmuch as

1067


CORRESPONDENCE developing countries are actively developing policy to engage with GM crops, there is indeed very little going on in terms of GM insects, which, for the record, will ignore national boundaries. An international entity with broad, adaptive and adequate representation is therefore urgently called for. Given the right mandate, it can safeguard against uncontrolled expansion of activities while serving as a shield for antagonistic influences through active stakeholder engagement. Finally, following the foregoing multiple perspective debates on GM mosquitoes, we propose the rapid initiation of an international gathering to start addressing the complexity of ethical, legal and social aspects of GM mosquitoes for disease control, a process that should already have taken place16,17. We conclude that contrary to there being a ‘green light for mosquito control,’ as announced in your journal18, research on SIT using transgenic insects has, for now at least, stalled at a yellow light. Bart G J Knols1, Rebecca C Hood-Nowotny1, Hervé Bossin1, Gerald Franz1, Alan Robinson1, Wolfgang R Mukabana2 & Samuel K Kemboi2 1Entomology Unit, FAO/IAEA Agriculture and Biotechnology Laboratory, A-2444 Seibersdorf, Seibersdorf, Vienna, Austria. 2University of Nairobi, P.O. Box 29053, Nairobi, Kenya. e-mail: [email protected]

1. Dyck, A.V., Hendrichs, J. & Robinson, A.S. (eds.) The Sterile Insect Technique: Principles and Practice in Area-Wide Integrated Pest Management (Springer, Heidelberg, 2005). 2. Catteruccia, F. et al. Science 299, 1225–1227 (2003). 3. Andreasen, M. & Curtis, C.F. Med. Vet. Entomol. 19, 238–244 (2005). 4. Franz, G. Genetica 116, 73–84 (2002). 5. Benedict, M. & Robinson, A.S. Trends Parasitol. 19, 349–355 (2003). 6. Scott, T.A., Takken, W., Knols, B.G.J. & Boete, C. Science 298, 117–119. 7. Alphey, L. et al. Science 298, 119–121 (2002). 8. Takken, W. & Scott, T.A. (eds.) Ecological Aspects for Application of Genetically Modified Mosquitoes. (Kluwer Academic Publishers, Dordrecht, The Netherlands, 2005) 9. Knols, B.G.J. & Louis, C. (eds.) Bridging Laboratory and Field Research for Genetic Control of Disease Vectors (Springer, Berlin, 2005). 10. The Royal Society. Risk Analysis, Perception and Management. Report of the Royal Society Study Group (The Royal Society, London, 1992). 11. Wynn, B. Global Environ. Change June, 111–127 (1992). 12. Rondinelli, D. Development Projects as Policy Experiments. (Routledge, London & New York, 1993). 13. Lusk, J.L. & Rozan, A. Trends Biotechnol. 23, 386– 387 (2005). 14. World Health Organisation. WHO Chronicle 30, 131–139 (1976). 15. Ison, R.L. Rangeland J. 15, 154–166 (1993). 16. Macer, D. Ethical, Legal and Social Issues of

1068

Genetically Modified Disease Vectors in Public Health. TDR/STR/SEB/ST/03.1 (World Health Organisation, Geneva, Switzerland, 2003). 17. Touré, Y.T. & Knols, B.G.J. in Genetically Modified Mosquitoes for Malaria Control (Boëte, C., ed.) (Landes Bioscience, Georgetown, Texas, USA, in the press, 2006). 18. Atkinson, P. Nat. Biotechnol. 23, 1371–1372 (2005).

Peter Atkinson responds: Knols et al. draw attention to two important points: that any new genetic strain developed for use in the sterile insect technique must undergo rigorous testing to ensure that it meets the necessary quality control standards required for the successful application of this technique; and that there must be full consultation with the public, stakeholders and any other interested parties before transgenic

strains can be released. These self-evident facts are not in dispute; rather, the advance reported by Crisanti and colleagues in Nature Biotechnology illustrates that recombinant techniques are now generating genetic strains that may now be appropriate for assessment and, pending the outcome, deployment in insect genetic control programs. The application of these developments do need to be openly discussed in the type of forum outlined by Knols et al. and, toward this goal, preliminary workshops on this topic have already been convened1. 1. Takken, W. & Scott, T.W. (eds.) Ecological Aspects for Application of Genetically Modified Mosquitoes. Reports from a Workshop held at Wageningen University and Research Center, June 2002 (Kluwer Academic Publishers, Dordrecht, The Netherlands, 2003).

Sequencing errors or SNPs at splice-acceptor guanines in dbSNP? To the editor: Single-nucleotide polymorphisms (SNPs) are the most frequent type of human genetic variation. They are the major basis of our phenotypic individuality, particularly with respect to heritable differences in disease susceptibility. Large collections of mapped SNPs, public and private, are powerful tools for genetic studies1. The most comprehensive public SNP database, dbSNP (http://www. ncbi.nlm.nih.gov/projects/SNP), currently contains more than 12 million human SNPs (version 126). This wealth of data is extensively used by a broad community, including clinical, experimental and computational scientists, for both locus-specific and genome-wide studies. Therefore, the quality and completeness of dbSNP is of paramount importance and a recent meta-analysis of four confirmation studies estimated a false-positive rate of ~15–17%2. As we have an interest in alternative splicing in general3 and with respect to diseases in particular4, we searched dbSNP for human variations in a nine-nucleotide context (three exon and six intron positions) of all splicedonor/acceptor sites of mRNA RefSeqs. Contrary to our expectation for the highly conserved intron positions +1, +2 (donor) and –2, –1 (acceptor), the acceptor G at –1 showed a variability comparable to that of the random position –4 (Fig. 1a). As the disruption of the G at –1 normally results in the loss of the

acceptor site5, we questioned whether this surprising variability could be compensated by any of the known biological processes (for example, RNA editing) or is an indication for a yet unknown biological phenomenon. As we could not shape a plausible explanation for our observation, and before we considered undertaking a challenging, lengthy and potentially fruitless search for an unknown biological mechanism, we decided next to evaluate the possibility that false-positive entries in dbSNP are accountable for the inexplicable variability of position –1. To this end, we first used the dbSNP validation status description and classified the RefSNPs (dbSNP entries) in three categories: (C1) validated by frequency or genotype data from HapMap6 or any other submitter; (C2) validated by independent submissions, observation of the minor allele in at least two chromosomes or submitter confirmation; and (C3) single submission without confirmation. Conspicuously, position –1 showed the highest fraction in C3 (305 of 364, 84%; Fig. 1b). As experimental verification of RefSNPs depends on the availability of appropriate population samples and assays, it was not feasible for us to carry out such a study on a large scale. Therefore, we switched to a verification procedure making use of the electropherograms derived from automatic fluorescence-based DNA sequencing instruments (traces).


a

Figure 1 RefSNPs and sequence confidence. (a) Apparent hypervariability at splice-acceptor Gs. (b) Classification of RefSNPs at the splice acceptors according to their validation status. (c) Electropherograms (traces) illustrating the ‘G after A’ problem at splice-acceptor sites in the 5′-to-3′ sequencing direction. (d,e) Sequence confidence (Phred) values of trace data supported RefSNPs (d) classified according to their validation status and of G/H RefSNPs (e) classified according to the 5′ nucleotide; (d,e) numbers expressed as a percentage

Donor

500

Acceptor

No. RefSNPs

400 300 200 100 0

G

N

+1 +2

A

Intron

G

Exonic

–2 –1

Intronic Intron border

Currently, 76% of all RefSNPs are supplied with trace references and for nearly 60% these data are accessible via the US National Center for Biotechnology Information (NCBI) Trace Archive (http://www.ncbi.nlm.nih.gov/ Traces; Supplementary Notes). We manually examined the available traces for RefSNPs at acceptor positions –2, –1 and +1 and collected false-positive entries, which we classified as sequencing errors (wrong base calling due to low signal-to-noise ratio) and database errors (identity of genomic RefSeq and the trace supported RefSNP allele or ambiguous alignment in microsatellites). Sequencing errors were mainly detected among C3 RefSNPs that are solely based on single-pass trace data. Database errors occurred both in C2 and C3 RefSNPs independently of their trace coverage (single trace, multiple traces of the same strand, traces from both strands; Supplementary Notes online). The astonishing error rate of 93% among 181 RefSNPs with trace data at acceptor position –1 was exclusively caused by the wellknown suppression of G after A incorporation using thermostable, genetically engineered DNA polymerases in dye terminator sequencing reactions7 (Fig. 1c). Naturally, this problem occurs at acceptor sites only in forward (5′-to-3′) traces because the AG is CT in the reverse sequencing direction. Moreover, the ‘G after A’ problem is further enhanced by the polypyrimidine tract preceding the acceptor AG in the splice consensus8. Homopolymer stretches of T and C are known to cause problems with sequence accuracy as a result of polymerase slippage9, thus leading to elevated error rates not only at position –1 but also at –2 and +1. Altogether, we estimated false-positive rates at acceptor positions –2, –1 and +1 of 17%, 82% and 11%, respectively (Supplementary Tables 1–3 online). Excluding the estimated false-positive rates, no significant difference in the variability between acceptor positions –1 and –2 remains. Thus, we conclude that a systematic sequencing error (‘suppressed G after A’) and not a previously unknown biological phenomenon causes the high

b

c

500

rs6089914

400

No. RefSNPs


CORRESPONDENCE

300 200

rs12039312

100 0

A C1

d

C1 10 3

G C2

C3

C2 10 3

C3 Phred values

18

0–29 19

> 40

87

87

e

A(G/H)

G(G/H)

C(G/H)

9 38

30–39

63

41

T(G/H) 8

11

19

18

19

72

70

74

21

frequency of RefSNPs in splice-acceptor position –1. Sensitized by this analysis, we then asked to what extent dbSNP contains sequencing errors in general. First, a scan of all RefSNPs for the sequence confidence of the allele alternative to the genomic RefSeq confirmed our initial observation that false positives are very likely enriched among C3 entries (18% with Phred confidence value 10%. Altogether, we estimated that there were about 256,000 sequencing and 124,000 database errors, representing 3.2% and 1.5% of all RefSNPs. Among sequencing errors, the vast majority (85%) are caused by the ‘suppressed G after A’ problem. Most interestingly, some of the false RefSNPs were investigated in the HapMap project6 (Supplementary Tables 1–3 online) and, as expected, did not show any variation in all genotyped populations. The described error rates in dbSNP might both introduce serious biases in large-scale bioinformatic studies and misdirect experimental efforts, particularly if a special sequence context such as acceptor AG is considered. Therefore,

1069


CORRESPONDENCE we emphatically recommend all users of dbSNP to refer to the ‘validation status’ tag and use a simple SNP classification scheme, as described above, that aims at extracting RefSNPs with lower error rates. According to our classification, dbSNP (version 124) contains in C1, C2 and C3 2,077,680, 2,946,840 and 3,470,166 entries, respectively. To investigate the differences between those three classes, we extracted the available confidence information. C1 and C2 RefSNPs have higher average values (both 51.4) than SNPs in C3 (43.2, Supplementary Notes online). Furthermore, about 87% in C1 and C2 have confidence values of at least 40, in contrast to only 63% in C3 (Fig. 1d). As a low confidence value indicates a potential sequencing error, we recommend that bioinformatics and/or experimental efforts either use only C1 and C2 RefSNPs or find a way of excluding from C3 all dbSNP entries with Phred LOD

Sample C, A&B > LOD Percent difference

A StaRT-PCR

100

100

50

50

0

0

– 50

– 50

–100

–100 TAQ 850

GEX 192

QGN 223

TAQ 850

Sample C, A&B > 6K Percent difference

A TaqMan

100

50

50

0

0

– 50

– 50

–100

–100 GEX 75

QGN 183

QGN 223

Sample D, A&B > 6K

100

TAQ 95

GEX 192

TAQ 95

GEX 75

QGN 183

1117

A N A LY S I S 92 Common genes

42

48 5

44


97

StaRT-PCR

181 Common genes 15

15

10

10

10

5 0 –5 –10

5 0 –5

0

5

10

15

GEX Log2FC

0 –5

–15

–15 –15 –10 – 5

5

–10

–10

–15 995

53 Common genes

15

QGN Log2FC

134

TAQ Log2FC

625

QuantiGene

QGN Log2FC

TaqMan

–15 –10 – 5

0

5

10

TAQ Log2FC

15

–15 –10 – 5

0

5

10

15

GEX Log2FC

Figure 3 Correlation of fold change between alternative quantitative platforms. The sample B over sample A (B/A) fold changes (log2) for each gene common between two platforms were subjected to bivariate analysis. (a) TaqMan assays versus StaRT-PCR. (b) QuantiGene versus TaqMan assays. (c) QuantiGene versus StaRT-PCR. The dashed line on each graph represents the ideal slope of 1.0. The solid lines represent a linear regression fit. The overlapping gene list among the alternative quantitative platforms is represented in the Venn diagram. Linear fit: TaqMan assay versus StaRT-PCR, Y = –0.03647 + 0.9347X, R2 = 0.879; QuantiGene versus TaqMan assay, Y = 0.14 + 0.7825X, R2 = 0.8118; QuantiGene versus StaRT-PCR, Y = 0.4095 + 0.7707X, R2 = 0.8497.

for parameters i and ii because fewer genes were assayed for these platforms. Detailed site-by-site analysis of genes is provided for StaRT-PCR and QuantiGene in Supplementary Table 2 online and for TaqMan assays in Supplementary Figure 2 online. Detection sensitivity analysis was done for each alternative quantitative platform using the genes common to that platform and each of the microarray platforms. For this reason, assay ranges and expression characteristics of gene sets differed. There were 845, 157 and 197 genes determined to be present in sample A by TaqMan assays, StaRT-PCR and QuantiGene, respectively. At the lower ranges of gene expression, for each microarray, the fraction of genes detected decreased relative to each of the alternative quantitative platforms (Fig. 4a–c). In addition, detection sensitivities relative to each alternative quantitative platform varied among the microarray platforms. A fold-change comparison between each alternative quantitative platform and each microarray platform was also performed using LOWESS smoothing (Fig. 4d–f, ref. 22), which does not assume a linear relationship of fold-change values between platforms. We used a total of 392, 101 and 83 genes that were present in samples A and B at each site measured by each microarray platform and shared with TaqMan assays, StaRT-PCR and QuantiGene, respectively, for comparison. Although excellent fold-change correlations were observed, varying degrees of compression of signal-to-analyte response relative to the alternative quantitative platforms were also found. These data are consistent with the analysis presented elsewhere in this issue20. An additional analysis was done to show that compression effects are detectable for both low and high expressors (Supplementary Fig. 3 online). Traditionally, analysis of accuracy is carried out by analyzing the true positive rate (TPR) and false discovery rate (FDR). In this case, the actual rates were unknown. For this reason, we compared the microarray platforms to TaqMan, which became the reference platform. Using TaqMan assay calls as the reference, we constructed contingency tables against microarray platforms, in which the concordance was determined and both the P-value significance of the t-test and fold-change directionality (up- or downregulation) were taken into consideration. Specifically, true positives (TP) are genes differentially expressed (significant P value for the t-test) in both TaqMan and microarray platforms with fold change in the same direction; true negatives (TN) are genes not differentially expressed in either platform; false positives (FP), consist of two sets of genes: (i) genes not differentially expressed in TaqMan and differentially expressed in microarrays, or (ii) genes differentially expressed in both platforms with fold change in the opposite direction; false negatives (FN), genes differentially expressed for TaqMan and not for microarrays.

1118

For TPR analysis in TaqMan assays, microarrays were compared to genes considered differentially regulated at fold-change cut-offs of 0, 1.5 and 2.0 (Fig. 5a–c, Supplementary Table 3 online). For microarrays, differential expression was measured using a t-test and controlling for FDR at a 5% level23 for genes present in either sample A or B. For approximately half of the assay range assessed by TaqMan assays, there were consistent TPR values across array platforms. However, it is apparent that at low expression, detection percentages were directly proportional to TPR. As a result, there was also variation (up to 20%) in TPRs between array platforms (Fig. 5a, Supplementary Table 3 online). FDR analysis (Fig. 5d–f, Supplementary Table 3 online) using TaqMan assays as a reference also showed consistent FDRs for genes expressed at medium and high levels for the microarray platforms. As expected, alternative quantitative platforms showed ~5% discordance with arrays in agreement with the FDR cut-off used for defining differential expression in microarrays. However, genes expressed at low levels showed a variable and inverse relationship to FDR values (Fig. 5d, Supplementary Table 3 online). These results support the idea that differential expression measurement depends on the detection limit for each microarray platform. Discordant gene analysis Alternative quantitative platforms can also be used to resolve discordance among the microarray platforms because specific assays can be designed easily to identify the source of the discordance by probing different regions. Analysis of extremely discordant results among the 997 genes shared by microarray platforms and TaqMan assays resulted in 9 genes (~1%) that exhibit twofold or greater changes in opposite directions on different platforms with P < 0.0001 (Supplementary Table 4 online). Some of these genes such as POMC, LTA and EPHA7 (Supplementary Fig. 4 online) were considered low expressors by TaqMan assays (CT values > 32) and, as expected, were undetected in a majority of the microarray platforms. However, some genes appeared to exhibit true discordance, of which three (ELAVL1, IGFBP5, ABCD1) were selected for further analysis by the three alternative quantitative platforms. To investigate the nature of the discordance, we designed probes against different regions of the three genes. For IGFBP5 and ABCD1, alternative quantitative platform probes indicate consistently lower expression in sample A along the length of the transcripts (Fig. 6, Supplementary Table 5 online). These results suggest that discordance between the platforms in some cases is likely to be a result of cross-hybridization of microarray probes with other sequences. For ELAVL1, alternative quantitative platform probes were able to evaluate differential expression characteristics of the 5′ and 3′ ends of the gene. This result is consistent with a mapping



A N A LY S I S study showing that ELAVL1 has two alternative polyadenylation sites (unpublished observations). We also investigated some genes (DPYD, PTGS2, FURIN) that were discordant between the alternative quantitative platforms. DPYD discordant results were determined to be a result of probing different sequence locations in the gene. When probes from each alternative quantitative platform were designed to interrogate similar sequences, expression characteristics along the length of the gene were found to be in concordance. Although more 5′ probes appeared to have discrepancies in directionality of expression, these differences were found to be statistically insignificant (P > .01). Multiple probe locations for PTGS2 generated expression differences in the same direction of change across all three platforms. The only gene that remained discordant after using multiple probe designs for each of the three platforms was FURIN. For this gene both TaqMan assays and StaRT-PCR detected differential expression in probes specific to the 5′ end of the gene. Although all platforms interrogate this region of the gene, the smaller probes (TaqMan assays; base 25–95 and StaRT-PCR; base 22–182) may be detecting a splice variant not detected by probes interrogating a longer region of the gene (QuantiGene; base 1–501). Thus, by designing probes against different regions of a gene, alternative quantitative platforms can confirm location-specific expression characteristics of genes and aid in the resolution of discordant gene expression data.

indicating that although this metric reflects the ability of each platform to detect expression, it may also be subject to the stringency defined by the array manufacturer in applying detection calls. The consequences of these varying stringencies are that whereas a relaxed stringency in detection calls can lead to better detection and differential expression concordance, there will be a higher percentage of false positives. Supplementary Figure 2 online verifies that the discordance in differential expression is related to the intersite and interplatform variation in detection. Using StaRT-PCR or QuantiGene as references and more stringent criteria in which a fold-change cutoff of 2.0 was applied for genes that were considered present in at least three out of five replicates in both A and B samples did not eliminate intersite or interplatform variation in detection of differentially expressed genes (Supplementary Table 2 online). It is clear that this variation is nearly exclusively for genes expressed at low level. Even with these more stringent selection criteria, intersite variation in detection resulted in intersite and interplatform variation in lists of differentially expressed genes. Another source of discordance in differentially expressed genes in this study was interplatform variation in compression. Using alternative quantitative platforms as a reference, interplatform variation in signal-toanalyte response was observed (Fig. 4d–f) and it was particularly large among genes expressed in the high or low range (Supplementary Fig. 3 online). This platform-dependent compression was associated with discordance in differentially expressed genes (Supplementary Table 2 online). Whereas these results have identified specific causes of discordance in lists of detected, and/or differentially expressed genes, we found excellent fold-change correlation between each quantitative platform and each microarray platform for those genes that were detected by microarray platforms (Fig. 4d–f). Of the 845 genes detected in the microarray

DISCUSSION We have assessed three quantitative gene expression measurement technologies for their performance metrics, correlated the results obtained with them to DNA microarray data and then subsequently used them as a means to identify sources of discordance among microarray platforms. Our results show a good correlation between quantitative platform measurements and microarray data. This is true, regardless of whether RNA or cDNA levels were measured. A primary focus of this study was to identify possible sources of discordance. On the basis of a b c 100 100 100 data reported here, we have identified specific 80 80 80 reasons that partially explain why, as previously 60 60 60 reported22, groups of genes detected as differDetection 40 40 40 entially expressed on a particular microarray ABI ABI ABI AFX AFX AFX 20 20 20 platform are occasionally not reproducible AG1 AG1 AG1 GEH GEH GEH TaqMan QuantiGene StaRT-PCR ILM ILM ILM 0 0 0 across microarray platforms. 5 10 15 20 5 10 15 20 5 10 15 20 Whereas alternative quantitative platforms d f ABI e could detect over 85% of the genes shared ABI ABI 15 AFX AFX AFX 10 10 AG1 AG1 AG1 across alternative quantitative and array plat10 GEH GEH GEH ILM ILM ILM 5 5 5 FC forms in this study, microarray platforms were 0 0 0 correlation less sensitive in the detection of lower expressed –5 –5 –5 genes in this set (Fig. 4a–c, Supplementary –10 –10 –10 TaqMan QuantiGene StaRT-PCR Table 2 and Supplementary Fig. 2 online). In –15 –15 –10 – 5 0 5 10 15 –10 – 5 0 5 10 –10 – 5 0 5 10 addition, relative to the alternative quantitative platforms, detection levels varied by as Average signal much as 60% among microarray platforms for lower expressed genes in this set. Since sig- Figure 4 Performance of microarray platforms relative to alternative quantitative platforms. (a–c) nificant differential expression in microarrays Sensitivity of detection. Each microarray platform was compared to TaqMan (a), StaRT-PCR (b) or QuantiGene (c) for ability to detect genes expressed in sample A. Genes were analyzed based on present is largely dependent on the ability to reliably call criteria of being present in 3/5 replicates at one of the three microarray sites and in the majority of detect expression, intersite and interplatform replicates for each alternative quantitative platform (at least 3/4 for TaqMan, 2/3 for StaRT-PCR and variation can lead to discordant results in the QuantiGene). Genes detected by each alternative quantitative platform were sorted according to their signals (scaling as described in Fig. 1), and the percent of genes detected by both microarray and alternative gene lists. Using TaqMan assays as a reference, TPR quantitative platforms from bins of 30 consecutive genes (y axis) were plotted against the average signal and FDR for the various microarray platforms of those genes measured by the alternative quantitative platform (x axis). (d–f) Correlation of fold change measured by each microarray platform compared to TaqMan (d), StaRT-PCR (e) or QuantiGene (f). Pairdiffered across the assay range (Fig. 5a,d, wise Sample A to Sample B fold-change comparison, measured by each alternative quantitative platform Supplementary Table 3 online). TPR was (x axis) compared to each microarray platform (y axis). For each microarray platform, only genes present directly correlated to percent of detectable in both samples at each site were called present. Each line represents the Lowess smoothing fitting curve. genes whereas FDR was inversely correlated, The number of genes involved in each analysis varies with the platforms compared.


1119

A N A LY S I S Figure 5 Assessment of true positive rates and false discovery rates using TaqMan assays. (a–c) True positive rate (TPR) assessment using 80 80 80 TaqMan assays. All common genes between 60 60 60 TaqMan assays and microarray platforms were TPR 40 40 40 used for the TPR analysis. TPR was defined as the ABI ABI ABI AFX AFX 20 AFX percentage of differentially expressed genes in 20 20 AG1 AG1 AG1 GEH GEH GEH ILM sample A compared to sample B detected by each ILM 0 ILM 0 0 5 10 15 20 5 10 15 20 microarray platform out of the ones detected by 5 10 15 20 TaqMan assays data as truth [TPR = TP/(TP+FN)], No FC cut off FC cut off = 1.5 FC cut off = 2.0 ABI 100 ABI ABI 100 where TP is true positive and FN is false negative 100 AFX AFX AFX AG1 AG1 AG1 in microarray. Differential expression was detected 80 GEH GEH GEH 80 80 ILM ILM ILM by t-test, where false discovery rate (FDR) was 60 60 60 FDR controlled at the 5% level with fold-change 40 40 40 filters of 0 (d), 1.5 (e) and 2.0 (f). For TaqMan 20 20 20 assays, genes were ordered according to the 0 0 0 average signals of A and B and for bins of 50 5 10 15 20 5 10 15 20 5 10 15 20 consecutive genes, we compared the significant Average signal difference calls between each microarray platform and TaqMan assays. Concordance of differential expression was assessed for each platform. (d–f) False discovery rate (FDR) assessment using TaqMan assays. All common genes between TaqMan assays and microarray platforms were used for the FDR analysis. FDR was defined as FP/(TP + FP), where FP is false positive in microarrays. The FDR represents the percentage of differentially expressed genes detected only by microarray platforms out of all genes differentially expressed in microarray platforms. Notice that the FDR (relative to TaqMan assays) is slightly larger than 5%, which is expected from Benjamini Hochberg (BH) adjustment for multiple testing. Differential expression was detected by t-test (FDR at 5%), with fold-change level filters of 0 (d), 1.5 (e) and 2.0 (f).


a

No FC cut off

b

FC cut off = 1.5

c

100

100

100

d

e

f

platforms and commonly mapped to one or more of the alternative quantitative platforms, only 9 (1%) were ‘extremely’ discordant. A major factor contributing to these infrequent discordant results is differences in probe location. Assays designed to different locations of the discordant genes in this study demonstrated a utility of the alternative quantitative platforms (Fig. 6) to independently validate gene expression measurements from array platforms. This analysis was also useful in the study of discordance observed between alternative quantitative platforms. For example, discordant expression results for FURIN observed in alternative quantitative platforms is consistent with a probe location difference. The limited common gene list precluded a detailed analysis of the discordance caused by low expression genes among alternative quantitative platforms. In addition, another source of potential discordance may come from the difference of measuring mRNA directly versus measuring cDNA, which were not analyzed here. In summary, analysis of the MAQC samples by three alternative quantitative platforms revealed excellent fold-change correlation with microarray platform data while enabling identification of possible sources of intersite and interplatform discordance in lists of genes measured as differentially expressed. Advantages of the alternative quantitative platforms were partially due to assay specificity, lower detection threshold and expanded assay range. Another advantage was the ease with which they interrogated specific gene locations due to their flexible assay design. Further, analysis by these alternative quantitative technologies contributed to characterization of the MAQC samples and confirmed their value in guiding optimization of gene expression methods. METHODS Sample definition. Sample A was Universal Human Reference RNA (Stratagene) and sample B was human brain total RNA (Ambion). Concentrations of A and B were normalized based on total RNA as measured by OD260. C was a 3:1 volumetric mixture of A and B, and D was a 1:3 volumetric mixture of A and B. Selection of genes for validation by alternative quantitative platforms. A list of 1,297 RefSeqs was selected by the MAQC consortium. Over 90% of these genes were selected from a subset of 9,442 RefSeq common to the four platforms (Affymetrix, Agilent, GE Healthcare and Illumina) used in the MAQC Pilot-I Study (RNA Sample Pilot), based on annotation information provided by

1120

FC cut off = 2.0

manufacturers in August 2005. This selection ensured that the genes would cover the entire intensity and fold-change ranges and include any bias due to RefSeq itself. To aid in the titration study, we included a subset of (~100) genes based on tissue-specificity (A versus B). To address cross-platform data inconsistency, we also included another subset, which showed the largest variability in log2 fold change across platforms in the Pilot-I Study. Platform vendors were queried about their ‘favorite’ genes (e.g., CYP family, PPARA, HDAC family and a small number of these were included). Consideration was also given to the inclusion of genes that were available from QuantiGene and StaRT-PCR platforms. The final list was therefore not completely unbiased. Gene list for the MAQC study by alternative quantitative platforms. TaqMan assays: 1,000 TaqMan gene expression assays used in the study that matches with the MAQC gene list. These 1,000 assays were selected from > 200,000 available human TaqMan assays (>20,000 NCBI genes) and covered 997 genes (3 genes had more than one assay). StaRT-PCR: 103 genes were selected from the nearly 800 genes for which StaRT-PCR reagents are already available that match with the MAQC gene list. All genes that overlap with those measured by TaqMan assays and QuantiGene were included as well as an additional 102 genes for a total of 205. QuantiGene: we selected 245 QuantiGene assays (covered 244 genes) that matched with the MAQC gene list from nearly 2,600 genes for which QuantiGene probe sets are already available. All genes that overlap with those measured by TaqMan assays and StaRT-PCR were included. 55 genes were in common to all three alternative quantitative platforms. TaqMan assays. RNA Samples: total RNA samples A (universal human reference RNA (UHRR), Stratagene), B (brain, Ambion), C (3 UHRR:1 brain) and D (1 UHRR:3 brain) as described earlier were used for all TaqMan assays. There was no additional treatment to these samples before cDNA preparation. cDNA Preparation: cDNA was prepared from total RNA Sample A, B, C and D using Applied Biosystems cDNA Archive Kit and random primers. Multiple reactions containing 10 µg total RNA per 100 µl reaction volume were run for each sample following manufacturer’s recommendations. Individual reactions were pooled by sample and used for TaqMan assays analysis. TaqMan assays: each TaqMan Gene Expression Assay consists of two sequence-specific PCR primers and a TaqMan assay–FAM labeled MGB (minor groove binder) probe. Primer and probe design is described in Supplementary Methods. Each TaqMan assay was run in four replicates for each RNA sample. 10 ng total cDNA (as total input RNA) in a 10 µl final volume was used for each replicate assay. Assays were run with 2× Universal Master Mix without uracil-N-glycosylase on Applied Biosystems 7900 Fast Real-Time PCR System using universal cycling conditions (10 min at


QuantiGene. Assay procedure: the QuantiGene assays were performed according to the procedure of QuantiGene Reagent System (Panomics), which was previously described in detail24,25. Briefly, 10 µl of starting total RNA (500 ng)


ABCD1

DPYD

ELAVL1

1

2

0

0

0

–2

–1

–4

–2

FURIN

1

0

0

–1

–1

6,000

4,000

5,000

4,000

6,000

4,000

2,000

0

4,000

3,000

3,000

–2

–2

2,000

–5

2

1

0

0

3,000

2,000

PTGS2

1,000

5

2,000

0

IGFBP5 2

1,000

1,000

4,000

3,000

2,000

0

–2

1,000

3,000

2

2,000

2

1,000

4

0

StaRT-PCR. StaRT-PCR assays were performed according to the procedures previously described in detail4,12. Reverse transcription: for each of the four MAQC samples, two 20 µg aliquots of RNA were reverse transcribed. Each reverse-transcription reaction took place in a 90 µl volume containing Moloney Murine Leukemia Virus (MMLV) reverse transcriptase (1,500 units), MMLV RT 5× first strand buffer (final concentrations 50 mM Tris-HCl, pH 8.3, 75 mM KCl, 3 mM MgCl2) (both from Invitrogen), oligo dT primers (1.5 µg), RNasin (70 units), and deoxynucleotide triphosphates (dNTPs) (10 mM) (all from Promega). Calibration of cDNA: After reverse transcription, the two 90 µl cDNA products for each sample were combined into a single 180 µl volume. Each sample was then calibrated. A 2 µl aliquot of undiluted, tenfold diluted, or 100-fold diluted cDNA from each sample was PCR-amplified in presence of 2 µl of SMIS. In each µl of SMIS there are 600,000 JW molecules of ACTB internal standard (IS). It was determined that for each MAQC cDNA sample, a 50-fold dilution would result in approximate equivalence between ACTB NT and IS PCR products when equivalent volumes of each were included in the PCR reaction. After 50-fold dilution, there were 4,500 µl of each cDNA sample. It was then confirmed for each sample that the amount of ACTB cDNA in 1 µl was approximately in balance with the 600,000 ACTB internal standard molecules in 1 µl of SMIS. The amount of RNA that contributed to each µl of each 50-fold diluted working solution was 4 ng. StaRT-PCR reaction conditions: for each StaRT-PCR reaction, a 20 µl reaction volume was prepared containing 2 µl of the calibrated cDNA sample, 2 µl of SMIS, 0.5 units of Taq polymerase, 2.2 µl of buffer, 0.6 ml of MgCl2, 1 µl of each primer, 0.45 µl of dNTPs, and 10.65 µl of water. Range finding step: the expression level of each gene in each sample was initially unknown. Thus, to ensure that each measurement was in range of quantification (NT/IS > 1/10 and < 10/1), a range finding measurement was conducted for each gene in each sample with E SMIS. Each µl of E SMIS, contains 600 molecules of the target gene IS and 600,000 molecules of ACTB IS. After PCR amplification and electrophoretic separation of the PCR products, the SEM Center software then determined whether the NT/IS ratio of the PCR products was acceptable or, if not, predicted which SMIS should be used for quantification. This prediction was 95% accurate. Quantification: each 20 µl reaction volume contained 2 µl of the calibrated cDNA sample and 2 µl of the appropriate SMIS (that is, A–F), predicted to be correct in the range finding step. Triplicate measurements were made of each gene in each sample. The fold-change calculation for each gene was based on the ratio of the gene transcript in sample B over sample A.

from sample A, B, C or D was mixed with 40 µl of Lysis Mixture (Panomics), 40 µl of Capture Buffer (Panomics) and 10 µl of target gene-specific probe set (CE (capture extender), 1.65 fmol/µl; LE (label extender), 6.6 fmol/µl; BL (blocker), 3.3 fmol/µl). Each sample mixture was then dispensed into an individual well of a Capture Plate (Panomics). The Capture Plate was sealed with foil tape and incubated at 53 °C for 16–20 h. The hybridization mixture was removed and the wells were washed 3× with 250 µl of wash buffer (0.1× SSC, 0.03% lithium lauryl sulfate). Residual wash buffer was removed by centrifuging the inverted Capture Plate at 1,000g. Signals for the bound target mRNA were developed by sequential hybridization with branched DNA (bDNA) amplifier, and alkaline phosphatase-conjugated label probe, at 46 °C for 1 h each. Two washes with wash buffer were used to remove unbound material after each hybridization step. Substrate dioxetane was added to the wells and incubated at 46 °C for 30 min. Luminescence from each well was measured using a Lmax microtiter plate luminometer (Molecular Devices). Three replicate assays measuring RNA directly (independent sampling n = 3) were performed for all described experiments. Genomic DNA contamination in the RNA sample, if there is any, does not affect the QuantiGene assay, since it remains doubled-stranded throughout the entire procedure and thus cannot hybridize to the probe sets at the temperature used in the assay. Data analysis and filtering: the QuantiGene assays of 244 genes are performed for MAQC samples A, B, C, D. For all samples, background signals were determined in the absence of RNA samples and subtracted from signals obtained in the presence of RNA samples. Because the QuantiGene assay measures RNA directly, no data normalization against a reference gene is required in the data analysis. The presence and absence call is determined by limit of detection (LOD) of the assay, where LOD = background + 3 s.d. of background. If at least two samples out of A, B, C, D have signals below LOD in a gene, we call the gene absence. To determine gene expression fold change in sample A versus sample B,

0

95 °C; 15 s at 95 °C, 1 min 60 °C, 40 cycles). The assays and samples were analyzed across a total of 44–384 well plates. Robotic methods (Biomek FX) were used for plate setup and each sample and assay replicate was tracked on a per well, per plate basis. Data normalization: in QRT-PCR an endogenous control gene is used to normalize data and control for variability between samples as well as plate, instrument and pipetting differences. POLR2A was chosen as the reference gene because its CT value was within the range of most of the genes in the study and showed the least variation across the samples (Supplementary Fig. 5a,b online). Each replicate CT was normalized to the average CT of POLR2A on a per plate basis by subtracting the average CT of POLR2A from each replicate to give the ∆CT which is equivalent to the log2 difference between endogenous control and target gene. Data analysis and filtering: the ∆CT of each replicate for each of the 1,000 assays was presented in the final data set as the normalized data. When TaqMan gene expression assays are run on a 7900HT system in a 10 µl reaction volume, a raw CT value of 34 represents approximately ten transcript molecules (assuming 100% amplification efficiency). At a copy number less than five, stochastic effects dominate and data generated are less reliable. Thus, a raw CT of 35 was set as the limit of detection in this study: individual replicates which gave CT values >35 were considered not detected and flagged as not expressed (A, absent); replicates with CT < 35 were considered detectable and identified as expressed (P, present). A CT > 32 and 35. Fold-change calculation: the log2 fold change between two samples was calculated using ∆∆CT method21: the average ∆CT of sample A was subtracted from that of samples B.

Log2 fold change


A N A LY S I S

Gene coordinates − TaqMan assays

− StaRT-PCR assays

− QuantiGene assays

− Array platforms

Figure 6 Resolution of fold-change discrepancy results. Fold changes were calculated for Sample B vs. Sample A in all platforms. Each panel shows expression characteristics of a discordant gene across the transcript length. Y axis is log2 fold change. X axis represents transcript length starting from the 5′ end of the transcript. Gray bar graphically illustrates the transcript and the red vertical lines represent the exon-exon junctions. Colored bars represent expression value of each probe along the length of the transcript. The length of the colored bar represents the region interrogated by the probe for each platform. Two probes for FURIN (base 1–501, and base 217–2133) produced indistinguishable fold-change value in QuantiGene assay.

1121


A N A LY S I S we calculated the fold change (fold changes) using formula log2 fold changes = log2(SA/SB), where SA represents the assay range for a target gene in sample A and SB represents the assay range for the target gene in sample B. A gene is considered for fold-change analysis if the signal in both sample A and sample B passes the LOD. Relative accuracy calculation: relative accuracy measures the proximity of observed expression values for C and D to the predicted values based on measured expression values for A and B. Concentrations of samples A and B were each quantified and normalized on the basis of total RNA (OD260). They were then mixed on a volumetric basis to yield sample C (0.75A/0.25B) and sample D (0.25A/0.75B). If the assay range for the target mRNA is within the linear dynamic range of the assay, then the predicted assay signal for Sample C and Sample D can be calculated using the following formula: C′ = 0.75A + 0.25B and D′ = 0.25A + 0.75B. TaqMan assay and QuantiGene sample input was based on total RNA. For this reason the predicted values of C and D can be calculated from the volumetric proportions of A and B based on the formula C = 0.25A + 0.75B and D = 0.75A + 0.25B. With StaRT-PCR, as with the microarrays, each measurement was normalized to mRNA instead of the starting total RNA. As described in26 and27, if the fraction of mRNA is higher in sample A compared to sample B, the predicted C and D values will be different from the formula provided above. Based on analysis of optimal linearity among the MAQC samples for the StaRT-PCR data, the most likely formula was determined to be C = 0.88A + 0.12B and D = 0.45A + 0.55B. A data set recalibrated on the basis of these assumed formulas (Supplementary Methods) was used to assess relative accuracy for StaRT-PCR. Multi-platform data transformation for Figure 1. For StaRT-PCR, 6,000 transcript molecules were defined by a value of 6,000 or log2 (6,000) = 12.55. For TaqMan assays, first the CT values were transformed from a decreasing copy number scale to an increasing copy number scale. This was accomplished by taking the absolute value of the difference of every TaqMan assay CT value and the lowest value for TaqMan assays CT (40). This rescaling preserves the assay range measured by TaqMan assays in the log2 space. Given that a TaqMan assay CT value of 35 is estimated to correspond to 5 transcript molecules, the extrapolated CT equivalent for 6,000 transcript molecules is ~24.78. This value on the transformed scale corresponds to |24.78–40| or 15.22. To scale this to the StaRT-PCR value of 6,000 transcript molecules, a rescaling value of 2.66025 was applied to all values. This factor was calculated by taking the difference between the prescaling value in TaqMan assays that corresponds to 6,000 transcript molecules (15.22) and the value of StaRT-PCR that corresponds to 6,000 transcript molecules (12.55). The same transformation was applied to QuantiGene values resulting in a rescaling factor = 13.55. This factor was generated with the estimation of 6,000 transcript molecules defined by 0.5 RLU or –1.0 on a log2 scale. These transformations result in all platforms having a post-scaling value of 12.55 on a log2 scale for an approximate threshold of 6,000 transcript molecules. Note: Supplementary information is available on the Nature Biotechnology website. ACKNOWLEDGMENTS We would like to acknowledge the contribution to this manuscript from the following members of the MAQC team: Shawn B. Baker, Anne Bergstrom Lucas, Jim Collins, Eugene Chudin, Stephanie Fulmer-Smentek, Damir Herman, Richard Shippy, Chunlin Xiao and Necip Mehmet. DISCLAIMER This work includes contributions from, and was reviewed by, the FDA. The FDA has approved this work for publication, but it does not necessarily reflect official Agency policy. Certain commercial materials and equipment are identified in order to adequately specify experimental procedures. In no case does such identification imply recommendation or endorsement by the FDA, nor does it imply that the items identified are necessarily the best available for the purpose.

1122

COMPETING INTERESTS STATEMENT The authors declare competing financial interests (see the Nature Biotechnology website for details). Published online at http://www.nature.com/naturebiotechnology/ Reprints and permissions information is available online at http://npg.nature.com/ reprintsandpermissions/ 1. Vondracek, M. et al. Transcript profiling of enzymes involved in detoxification of xenobiotics and reactive oxygen in human normal and simian virus 40 T antigenimmortalized oral keratinocytes. Int. J. Cancer 99, 776–782 (2002). 2. Urdea, M. et al. Branched DNA amplification multimers for the sensitive, direct detection of human hepatitis virus. Nucleic Acids Symp. Ser. 24, 197–200 (1991). 3. Gleaves, C.A. et al. Multicenter evaluation of the Bayer VERSANT HIV-1 RNA 3.0 assay: analytical and clinical performance. J. Clin. Virol. 25, 205–216 (2002). 4. Bustin, S.A. (ed.). A-Z of Quantitative PCR. (International University Line Biotechnology Series, La Jolla, California, USA, 2004). 5. Wong, M.L. & Medrano, J.F. Real-time PCR for mRNA quantitation. Biotechniques 39, 75–85 (2005). 6. Lee, L.G., Connell, C.R. & Bloch, W. Allelic discrimination by nick-translation PCR with fluorogenic probes. Nucleic Acids Res. 21, 3761–3766 (1993). 7. Heid, C.A., Stevens, J., Livak, K.J. & Williams, P.M. Real time quantitative PCR. Genome Res. 6, 986–994 (1996). 8. Gibson, U.E., Heid, C.A. & Williams, P.M. A novel method for real time quantitative RT-PCR. Genome Res. 6, 995–1001 (1996). 9. Qin, L.X. et al. Evaluation of methods for oligonucleotide array data via quantitative real-time PCR. BMC Bioinformatics 7, 23 (2006). 10. Kuo, W.P. et al. A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies. Nat. Biotechnol. 24, 832–840 (2006). 11. Wang, Y. et al. Large scale real-time PCR validation on gene expression measurements from two commercial long-oligonucleotide microarrays. BMC Genomics 7, 59 (2006). 12. Willey, J.C. et al. Standardized RT-PCR and the standardized expression measurement center. Methods Mol. Biol. 258, 13–41 (2004). 13. Rots, M.G. et al. mRNA expression levels of methotrexate resistance-related proteins in childhood leukemia as determined by a standardized competitive template-based RT-PCR method. Leukemia 14, 2166–2175 (2000). 14. Mullins, D.N. et al. CEBPG transcription factor correlates with antioxidant and DNA repair genes in normal bronchial epithelial cells but not in individuals with bronchogenic carcinoma. BMC Cancer 5, 141 (2005). 15. Flagella, M. et al. A multiplex branched DNA assay for parallel quantitative gene expression profiling. Anal. Biochem. 352, 50–60 (2006). 16. Yao, J.D. et al. Multicenter Evaluation of the VERSANT Hepatitis B Virus DNA 3.0 Assay. J. Clin. Microbiol. 42, 800–806 (2004). 17. Elbeik, T. et al. Multicenter Evaluation of the Performance Characteristics of the Bayer VERSANT HCV RNA 3.0 Assay (bDNA). J. Clin. Microbiol. 42, 563–569 (2004). 18. Stenman, J. & Orpana, A. Accuracy in amplification. Nat. Biotechnol. 19, 1011–1012 (2001). 19. Cleveland, W. Robust locally weighted regression and smoothing scatter plots. J. Am. Stat. Assoc. 74, 829–836 (1979). 20. MAQC Consortium. The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nat. Biotechnol. 24, 1151–1161 (2006). 21. Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate - a practical and powerful approach to multiple testing. J. R. Stat. Soc. B. Met. 57, 289–300 (1995). 22. Shippy, R. et al. Performance evaluation of commercial short-oligonucleotide microarrays and the impact of noise in making cross-platform correlations. BMC Genomics 5, 61 (2004). 23. Livak, K.J. & Schmittgen, T.D. Analysis of relative gene expression data using real-time quantitative PCR and the 2-∆∆CT Method. Methods 25, 402–408 (2001). 24. Kern, D. et al. An enhanced-sensitivity branched-DNA assay for quantification of human immunodeficiency virus type 1 RNA in plasma. J. Clin. Microbiol. 34, 3196– 3202 (1996). 25. Wang, J. et al. Regulation of insulin preRNA splicing by glucose. Proc. Natl Acad. Sci. USA 94, 4360–4365 (1997). 26. Shippy, R. et al. Using RNA sample titrations to assess microarray platform performance and normalization techniques. Nat. Biotechnol. 24, 1123–1131 (2006). 27. Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006).



A N A LY S I S

Using RNA sample titrations to assess microarray platform performance and normalization techniques Richard Shippy1, Stephanie Fulmer-Smentek2, Roderick V Jensen3, Wendell D Jones4, Paul K Wolber2, Charles D Johnson5, P Scott Pine6, Cecilie Boysen7, Xu Guo8, Eugene Chudin9, Yongming Andrew Sun10, James C Willey11, Jean Thierry-Mieg12, Danielle Thierry-Mieg12, Robert A Setterquist13, Mike Wilson5, Anne Bergstrom Lucas2, Natalia Novoradovskaya14, Adam Papallo3, Yaron Turpaz8, Shawn C Baker9, Janet A Warrington8, Leming Shi15 & Damir Herman12 We have assessed the utility of RNA titration samples for evaluating microarray platform performance and the impact of different normalization methods on the results obtained. As part of the MicroArray Quality Control project, we investigated the performance of five commercial microarray platforms using two independent RNA samples and two titration mixtures of these samples. Focusing on 12,091 genes common across all platforms, we determined the ability of each platform to detect the correct titration response across the samples. Global deviations from the response predicted by the titration ratios were observed. These differences could be explained by variations in relative amounts of messenger RNA as a fraction of total RNA between the two independent samples. Overall, both the qualitative and quantitative correspondence across platforms was high. In summary, titration samples may be regarded as a valuable tool, not only for assessing microarray platform performance and different analysis methods, but also for determining some underlying biological features of the samples.

1GE Healthcare, 7700 S. River Pkwy., Suite #2603, Tempe, Arizona 85284, USA. 2Agilent Technologies, Inc., 5301 Stevens Creek Blvd., Santa Clara, California 95051, USA. 3University of Massachusetts-Boston, 100 Morrissey Blvd., Boston, Massachusetts 02125, USA. 4Expression Analysis, Inc., 2605 Meridian Pkwy., Durham, North Carolina 27713, USA. 5Asuragen, Inc., 2150 Woodward, Austin, Texas 78744, USA. 6Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, Maryland 20993, USA. 7ViaLogy, 2400 Lincoln Ave, Altadena, California 91001, USA. 8Affymetrix, Inc., 3420 Central Expressway, Santa Clara, California 95051, USA. 9Illumina, Inc., 9885 Towne Centre Dr., San Diego, California 92121, USA. 10Applied Biosystems, 850 Lincoln Centre Dr., Foster City, California 94404, USA. 11University of Toledo, Toledo, Ohio 43606, USA. 12National Center for Biotechnology Information, Bethesda, Maryland 20894, USA. 13Applied Biosystems, 2150 Woodward, Austin, Texas 78744, USA. 14Stratagene, 11011 N. Torrey Pines Rd., La Jolla, California 92037, USA. 15National Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Rd., Jefferson, Arizona 72079, USA. Correspondence should be addressed to R.S. ([email protected]).

Published online 8 September 2006; doi:10.1038/nbt1241


Microarrays are widely used to simultaneously measure the levels of thousands of RNA targets in a biological sample. Despite their widespread use, many in the community are concerned with the comparability of the results obtained using different microarray platforms and thus the biological relevance of the qualitative and quantitative results obtained. Microarray platform performance has been evaluated before on the criteria of sensitivity, specificity, dynamic range, precision and accuracy1–12. As part of the MicroArray Quality Control (MAQC) project, similar assessments have also been reported13,14. Other studies have used defined mixtures of RNA samples (titration samples) for interplatform2,15 and interlaboratory15 comparisons. Here we have investigated an alternative performance metric: the abilities of different microarray platforms to accurately detect a signal trend produced by mixing samples (titration trend) and the effects of normalization and other data analysis practices on this performance characteristic. Gene-expression levels were measured for two pure samples and two mixtures using five different commercial whole-genome platforms at three different test sites per platform. The five commercially available whole-genome platforms tested were Applied Biosystems (ABI), Affymetrix (AFX), Agilent Technologies (AG1), GE Healthcare (GEH) and Illumina (ILM). The level of accurate titration response was quantified by determining the number of probes for which the average signal response in the titration samples was consistent with the response in the independent, reference RNA samples. We analyzed every platform at each site, and here we present comparisons of the various platforms using various data processing and normalization techniques. To assess the titration response of as many genes as possible, an a priori expectation of differential expression of many transcripts was necessary. On the basis of results from pilot titration studies (data not shown), we elected to use two independent samples (A, Stratagene Universal RNA, and B, Ambion Human Brain RNA) that showed large, statistically significant differences in expression for a large number of transcripts to generate the two titration samples (C and D, consisting of 3:1 and 1:3 ratios of A to B, respectively; see Fig. 1). We defined the series of mean signals generated by a gene on a microarray platform across these samples as its titration response. For these analyses, we assumed

1123

A N A LY S I S B

A

Independent samples


C

D

Titration samples

75% A + 25% B

25% A + 75% B

Figure 1 RNA samples. We used expression measurements from two independent total RNA samples, A and B, and mixtures of these two samples at defined ratios of 3:1 (C) and 1:3 (D). The titration mixtures were generated once for all experiments, with samples A and B at equal total RNA concentrations as determined by A260.

that the expression measurement of a transcript in a titration sample follows a linear titration relationship: the signal of any given transcript in the two titration samples should be a linear combination of the signals produced by the two independent samples. From the signal intensities in the microarray titration experiments, we obtained the percentage of genes on each platform that showed a monotonic titration response and analyzed that percentage as a function of the magnitude of differential expression between A and B or as a function of the signal intensity. Many normalization methods have been developed that are commonly used for different microarray platforms16–24, including those methods that have been recommended by the array manufacturers for the MAQC project13 (see Methods). Differences in these methods significantly influence several aspects of microarray performance, including precision and sensitivity9,16–20,23,24. However, no clear consensus exists in the microarray community as to which method is best under a given set of circumstances. The optimal normalization or scaling methods for a given dataset may depend both on the experiment and on many attributes of that microarray dataset, including signal distribution and noise characteristics25. The experimental design used here is valuable for assessing the influence of different data processing techniques on the self-consistency of microarray data with regard to titration response. In addition, the different data processing techniques were also analyzed with respect to their impact on the statistical power of these platforms to distinguish between the independent and titration samples. The titration analysis presented here was applied to all commercial whole-genome microarray platforms tested in the MAQC project13, using various data processing techniques, to evaluate the self-consistency and statistical power of the resulting data. When assessing accuracy in experimental systems, the goal is to compare observed results to the expected ‘true’ values of the system. For most experiments measuring gene expression, the ‘true’ values are either unknown or difficult to measure independently. However, the titration response results presented here can provide some quantitative information about the relative accuracy of measurements of differential gene expression. Monotonicity in the titration response indicates a selfconsistent relationship among the expression measurements from the four samples. Because many inferences drawn from microarray experiments depend as much or more on the direction of expression changes

1124

as on their magnitudes, the consistency with which microarray assays determine direction of change is an important performance characteristic. The main advantages of our method are that titration responses can be assessed on a large scale, independent of a designated reference platform, and that it does not require substantial assumptions to be made about the data2,25. RESULTS The experimental design of the main MAQC study is described in detail elsewhere13. Briefly, two independent RNA samples were chosen for study and used to generate two titration samples. The gene-expression profiles of these samples, all split from a single pool, were measured on ten gene-expression measurement platforms. For each of the five whole-genome microarray platforms examined in this study, the samples were analyzed at three different test sites, each with ≤5 replicate assays per sample, for a total of 293 microarray hybridizations at 15 different sites. Data from all platforms were then processed using the recommended method from each array manufacturer, as represented in the main MAQC paper13, as well as one or more alternative normalization methods. Using probe sequence information, we identified 12,091 genes that were uniquely targeted by at least one probe for all five commercial whole-genome microarray platforms. For each platform, only the probe closest to the 3′ end of the gene was considered13. We chose to exclude genes that were not detected across all samples and focused on genes whose signals were above the noise level and therefore more reliable10. Each manufacturer provided quantitative detection calls characterizing the probability that a gene was detected in a given replicate13. For most analyses, only genes detected in at least three replicates for a given sample and site were considered. This detection-call protocol is the same as described in the main MAQC paper13. Measuring titration response as a function of fold change The chief advantage of an experiment that evaluates gene expression in a series of known mixtures of two samples is that the rank order of measured expression levels of any given gene across the series can be predicted from the relative expression levels in the two original samples. For the series described in this paper, if the true expression level (Ai) of any gene i in sample A is greater than the true expression level (Bi) of the same gene i in sample B, then Ai > Ci > Di > Bi, where Ci and Di are the true expression levels of gene i in samples C and D. If Bi > Ai, then Bi > Di > Ci > Ai. In our case, if we postulate Ai > Bi on the – basis of the observed sample mean of Ai (Ai) being significantly larger – (P < 0.001) than the observed sample mean of Bi (Bi), then we expect – – – – Ai > Ci > Di > Bi. Finally, if Ai ≈ Bi, then the order of observed means will be nearly random. In Figure 2, the percentage of genes in a 100-gene moving window that produce the expected titration response for each site and platform – – is plotted as a function of the average Ai / Bi ratio of those 100 genes, – – – – – – when Ai > Bi (left side of graph), or of the Bi / Ai ratio, when Bi > Ai (right – – – – side of graph). The x-axis origin of these graphs is at Ai / Bi = Bi / Ai = 1, the ratio at which the titration response changes direction. The overall shapes of all of the curves are similar: as expected from theory, they rise – – – – from a value near zero at Ai / Bi = Bi / Ai = 1 to an asymptote of 100% at – – – – larger values of Ai / Bi or Bi / Ai. Figure 2 also illustrates how alternative normalization methods (for AFX, alternative data reduction methods of the individual features) affect the quantitative outcome. For example, the data from the different test sites for AG1 show distinct behaviors under the standard normalization, but exhibit much more similar titration behaviors when normalized using the alternative method. In addition, for the AFX data, GCRMA processing26 (a modified version


differences resulting from alternative normalization techniques are also apparent in the results presented in Figure 2 and Table 1.

of robust multichip analysis (RMA) processing that models intensity of probe level data as a function of GC content) results in titration curves with a broader spread than those produced by probe logarithmic intensity error (PLIER)21 or RMA18. It should be noted that the different data processing techniques also yield different numbers of genes showing significant deviations in expression values between samples A and B (Fig. 2 and Table 1), which can also influence titration performance. The most striking differences resulting from normalization techniques are seen with the ILM data, where the alternative method, invariant scaling, resulted in many fewer significant genes on the left side of the panel as well as lower percentages of genes that titrate at lower-fold changes. The quantitative differences between the various curves shown in Figure 2 are listed in Table 1, which presents the ratios at which 50%, 75% or 90% of the detected genes show a monotonic titration response. The performances observed for different sites and platforms were similar but not identical (Table 1). Many different platforms and sites identified the correct ordering of the titration samples for more than 90% of genes with twofold difference between A and B (Table 1, rows 14 and 17), which suggests that the DNA microarrays can reliably distinguish very small-fold differences in the mixture samples. The

ABI - quantile

Measuring titration response as a function of signal intensity To further explore the impact of different normalization techniques, we assessed titration response as a function of signal intensity. In Figure 3, we plot the fraction of genes that titrate relative to the total number of genes in the given intensity range, as a function of the lowest signal in the monotonic titration trend. That is, for the monotonic trend – – – – – Ai > Ci > Di > Bi, we plotted this fraction against the signal intensity Bi – – – – – (solid lines), whereas for the opposite trend Bi > Di > Ci > Ai, we used – the intensity Ai (dashed lines). We observed that, in general, the fraction of genes that titrate is inversely proportional to the signal intensity. The signal plotted on the x-axis is the lowest signal in the series; therefore, when this signal is low, the probes are more likely to show the expected titration response, as the fold differences will tend to be larger. When the magnitude of this lowest signal increases, the possible fold difference between A and B will decrease. Differences in distribution among platforms and normalization methods are evident. For ABI, the fraction of genes that titrate follows the same trend as for the other platforms when A > B (Fig. 3, solid lines),

ABI - scaling

AG1 - median scaling

AG1 - 75th percentile scaling

100

100

100

100

90

90

90

90

80

80

80

80

70

70

70

60

60

60

50 40 30 20

A>C>D>B n = 2,806 n = 2,169 n = 2,740

50

B>D>C>A n = 2,240 n = 1,803 n = 2,198

40 30 20

0 4.0

A>C>D>B n = 2,960 n = 2,285 n = 2,807

50

B>D>C>A n = 2,355 n = 1,844 n = 2,312

40 30 20

10

10

Percentage of genes that titrate


A N A LY S I S

3.5

3.0

2.5

2.0

1.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 4.0

60 50

B>D>C>A n = 1,977 n = 2,168 n = 2,589

40 30 20

10 3.5

3.0

B/A

A/B

70

A>C>D>B n = 3,654 n = 3,435 n = 2,697

2.5

2.0

1.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 4.0

3.0

2.5

2.0

1.5

1.0

1.5

2.0

A/B

2.5

3.0

3.5

4.0

0 4.0

100

90

90

80

80

80

80

70

70

70

60

60

60

40 30 20

50

A>C>D>B n = 2,290 40 n = 2,772 30 n = 2,966 20

40 30 20

3.5

3.0

2.5

2.0

1.5

1.0

1.5

2.0

A/B

2.5

3.0

3.5

4.0

0 4.0

3.5

3.0

B/A

2.5

2.0

1.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 4.0

50

B>D>C>A n = 1,781 n = 2,020 n = 2,015

40 30 20

3.0

2.5

2.0

1.5

1.0

1.5

2.0

A/B

2.5

3.0

3.5

4.0

0 4.0

100

90

90

80

80

80

80

70

70

70

70

60

60

60

50

50

30 20

30 20

50

B>D>C>A n = 2,251 n = 2,326 n = 2,496

40 30 20

10

10 0 4.0

A>C>D>B n = 3,902 n = 3,977 n = 4,352

3.5

3.0

2.5

A/B

2.0

1.5

1.0

1.5

2.0

2.5

B/A

3.0

3.5

4.0

0 4.0

3.5

3.0

2.5

A/B

2.0

1.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

B/A

3.5

4.0

B>D>C>A n = 1,720 n = 1,931 n = 1,956

3.0

2.5

2.0

1.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

B/A

60

A>C>D>B n = 3,128 n = 3,002 n = 2,543

50

B>D>C>A n = 2,136 n = 2,038 n = 1,792

40 30 20

A>C>D>B n = 1,981 n = 1,755 n = 1,542

B>D>C>A n = 3,152 n = 2,882 n = 2,900

10

10 0 4.0

3.0

ILM - invariant scaling

100

90

40

3.5

ILM - quantile

100

B>D>C>A n = 1,918 n = 2,008 n = 2,091

2.5

B/A

A/B

90

A>C>D>B n = 3,809 n = 4,063 n = 4,034

2.0

A>C>D>B n = 2,581 n = 3,092 n = 3,227

B/A

100

40

1.5

10 3.5

GEH - quantile

GEH - median scaling

1.0

60

B/A

A/B

1.5

70

A>C>D>B n = 2,772 n = 3,305 n = 3,365

10

10

10 0 4.0

50

B>D>C>A n = 1,696 n = 1,834 n = 1,951

2.0

AFX - GCRMA

100

90

B>D>C>A n = 1,869 n = 2,038 n = 2,132

2.5

AFX - RMA

100

90

A>C>D>B n = 2,938 n = 3,520 n = 3,517

3.0

A/B

100

50

3.5

B/A

AFX - MAS5

AFX - PLIER

B>D>C>A n = 2,164 n = 2,256 n = 2,538

10 3.5

B/A

A/B

A>C>D>B n = 3,138 n = 3,048 n = 3,254

3.5

3.0

2.5

A/B

2.0

1.5

1.0

1.5

2.0

2.5

B/A

3.0

3.5

4.0

0 4.0

3.5

3.0

2.5

A/B

2.0

1.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

B/A

Average linear ratio – – – – – – –– – – – – – Figure 2 Percentage of genes showing the monotonic titration responses Ai > Ci > Di > Bi and Bi > Di > Ci > Ai plotted against the linear Ai / Bi and Bi / Ai ratios, respectively, for each commercial whole-genome microarray platform, using various normalization methods. All graphs were generated from the set of 12,091 genes common across whole-genome platforms, with outlier arrays excluded per manufacturer’s recommendations13. Genes detected across all four samples per site that were also significantly differentially expressed (P < 0.001) in independent samples A and B were used in the calculations (Table 1, rows 4 and 5). A two-sample t-test, with equal variance, was performed within each site on log2 expression values. For each platform, a 100-probe moving window, – – – – based on sorted Ai / Bi ratios (left side of plot) or Bi / Ai ratios (right side of plot), was used to calculate the percentage of self-consistent monotonic titration – – – – response genes (y-axis) as a function of the corresponding moving average of Ai / Bi or Bi / Ai ratios (x-axis) within each site. Graphs are plotted with a scale break between –1 and 1, with reassignment of the x-axis for clarity. Each graph contains six series of data points (three sites in two monotonic directions), which were smoothed using a distance-weighted least-squares method. Blue, site 1; red, site 2; gray, site 3. Total number of genes showing the monotonic – – – – – – – – –– – – – trend for each site are indicated in each graph, for both directions (Ai > Ci > Di > Bi for Ai / Bi ratios >1 and Bi > Di > Ci > Ai for Bi / Ai ratios >1), and are also listed in Table 1 (rows 4 and 5). The normalization methods highlighted in yellow for each platform represent the manufacturer’s recommended method used in the MAQC main paper13.


1125

A N A LY S I S Table 1 Gene counts for AFX and ABI (top) and AG1, GEH and ILM (bottom) for each normalization method


Quantile

Scaling

PLIER

MAS 5.0

RMA

GCRMA

Row

Condition

ABI_1 ABI_2 ABI_3 ABI_1 ABI_2 ABI_3 AFX_1 AFX_2 AFX_3 AFX_1 AFX_2 AFX_3 AFX_1 AFX_2 AFX_3 AFX_1 AFX_2 AFX_3

1

Detected in A · B · C · D

8,049 7,863 8,550 8,049 7,863 8,550 7,359 7,006 7,424 7,359 7,006 7,424 7,359 7,006 7,424 7,359 7,006 7,424

2

A>B

4,284 4,191 4,509 4,308 4,219 4,424 4,423 4,291 4,557 4,244 4,040 4,267 4,414 4,192 4,440 4,356 4,125 4,376

3

B>A

3,765 3,672 4,041 3,741 3,644 4,126 2,936 2,715 2,867 3,115 2,966 3,157 2,945 2,814 2,984 3,003 2,881 3,048

4

A > B and P < 0.001

3,144 2,298 3,046 3,143 2,376 3,037 3,723 3,632 3,848 2,982 2,934 3,168 3,559 3,491 3,670 3,420 3,273 3,490

5

B > A and P < 0.001

2,572 1,886 2,436 2,571 1,930 2,494 2,356 2,176 2,306 2,074 1,999 2,182 2,272 2,274 2,372 2,224 2,172 2,303

6

A>C>D>B

3,063 2,924 3,159 3,296 3,104 3,256 3,042 3,751 3,616 2,493 3,111 3,258 2,862 3,462 3,479 2,708 3,297 3,407

7

B>D>C>A

2,471 2,424 2,622 2,670 2,487 2,772 1,924 2,154 2,222 1,873 2,089 2,170 1,858 2,100 2,087 1,829 2,071 2,075

8

A > C > D > B and P < 0.001 2,806 2,169 2,740 2,960 2,285 2,807 2,938 3,520 3,517 2,290 2,772 2,966 2,772 3,305 3,365 2,581 3,092 3,227

9

B > D > C > A and P < 0.001 2,240 1,803 2,198 2,355 1,844 2,312 1,869 2,038 2,132 1,696 1,834 1,951 1,781 2,020 2,015 1,720 1,931 1,956

10

(A > C > D > B) / (A > B)

0.71

0.70

0.70

0.77

0.74

0.74

0.69

0.87

0.79

0.59

0.77

0.76

0.65

0.83

0.78

0.62

0.80

11

(B > D > C > A) / (B > A)

0.66

0.66

0.65

0.71

0.68

0.67

0.66

0.79

0.78

0.60

0.70

0.69

0.63

0.75

0.70

0.61

0.72

0.68

12

50% titrate when A/B =

1.35

1.35

1.36

1.28

1.32

1.32

1.30

1.13

1.20

1.52

1.28

1.30

1.40

1.18

1.25

1.60

1.28

1.32

13


1.58

1.65

1.65

1.45

1.60

1.60

1.65

1.20

1.30

1.98

1.45

1.50

1.70

1.32

1.42

2.05

1.47

1.58

14


1.80

1.98

1.99

1.68

1.90

1.94

2.10

1.30

1.52

3.00

1.67

1.78

2.10

1.42

1.61

2.80

1.68

1.85

15

50% titrate when B/A =

1.43

1.42

1.45

1.34

1.35

1.40

1.39

1.20

1.22

1.53

1.30

1.36

1.44

1.22

1.30

1.63

1.35

1.47

16


1.77

1.80

1.88

1.60

1.75

1.83

1.68

1.37

1.38

1.82

1.45

1.52

1.75

1.40

1.50

2.22

1.65

1.80

17


2.08

2.23

2.40

1.85

2.12

2.30

2.05

1.49

1.50

2.50

1.75

1.87

2.15

1.58

1.68

2.90

2.10

2.30

18

A/B > 2.00

1,794 1,664 1,830 1,813 1,718 1,808 1,703 1,602 1,832 1,759 1,548 1,756 1,693 1,468 1,702 2,178 2,062 2,255

19

B/A > 2.00

1,636 1,562 1,745 1,634 1,548 1,793 1,171 1,028 1,136 1,360 1,202 1,346 1,172 1,017 1,141 1,462 1,378 1,501

20

A/B > 2.00 (P < 0.001)

1,772 1,558 1,802 1,793 1,626 1,782 1,703 1,602 1,832 1,732 1,542 1,748 1,693 1,468 1,700 2,168 2,049 2,233

21

B/A > 2.00 (P < 0.001)

1,613 1,423 1,672 1,612 1,435 1,716 1,171 1,028 1,136 1,350 1,195 1,335 1,171 1,017 1,141 1,447 1,365 1,487

Row

Condition

AG1_1 AG1_2 AG1_3 AG1_1 AG1_2 AG1_3 GEH_1 GEH_2 GEH_3 GEH_1 GEH_2 GEH_3 ILM_1 ILM_2 ILM_3 ILM_1 ILM_2 ILM_3

1

Detected in A · B · C · D

8,322 8,468 9,121 8,322 8,468 9,121 10,416 10,505 10,289 10,416 10,505 10,289 7,995 7,761 7,555 7,995 7,761 7,555

2

A>B

5,046 4,922 5,051 4,624 4,705 5,027 6,324 6,537 6,161 6,173 6,275 6,123 4,505 4,349 4,221 3,670 3,512 3,009

3

B>A

3,276 3,546 4,070 3,698 3,763 4,094 4,092 3,968 4,128 4,243 4,230 4,166 3,490 3,412 3,334 4,325 4,249 4,546

4

A > B and P < 0.001

3,711 3,763 3,710 3,443 3,624 3,807 3,998 4,753 4,393 4,042 4,582 4,512 3,657 3,289 2,808 2,868 2,479 1,769

5

B > A and P < 0.001

2,057 2,439 2,839 2,447 2,707 2,958 2,238 2,352 2,632 2,409 2,586 2,772 2,713 2,473 2,051 3,384 3,068 2,960

6

A>C>D>B

4,249 3,714 2,923 3,430 3,218 3,460 4,413 4,314 4,381 4,637 4,308 4,917 3,204 3,170 2,924 2,097 1,945 1,989

7

B>D>C>A

2,304 2,357 2,848 2,384 2,377 2,703 2,167 2,230 2,258 2,718 2,653 2,833 2,198 2,153 2,059 3,426 3,221 3,697

Median scaling

75th % scaling

Median scaling

Quantile

Quantile

0.78

Invariant scaling

8

A > C > D > B and P < 0.001 3,654 3,435 2,697 3,138 3,048 3,254 3,809 4,063 4,034 3,902 3,977 4,352 3,128 3,002 2,543 1,981 1,755 1,542

9

B > D > C > A and P < 0.001 1,977 2,168 2,589 2,164 2,256 2,538 1,918 2,008 2,091 2,251 2,326 2,496 2,136 2,038 1,792 3,152 2,882 2,900

10

(A > C > D > B) / (A > B)

0.84

0.75

0.58

0.74

0.68

0.69

0.70

0.66

0.71

0.75

0.69

0.80

0.71

0.73

0.69

0.57

0.55

11

(B > D > C > A) / (B > A)

0.70

0.66

0.70

0.64

0.63

0.66

0.53

0.56

0.55

0.64

0.63

0.68

0.63

0.63

0.62

0.79

0.76

0.66 0.81

12


1.24

1.35

1.60

1.38

1.48

1.43

1.34

1.45

1.40

1.25

1.38

1.25

1.32

1.30

1.34

1.52

1.55

1.32

13


1.39

1.66

2.15

1.53

1.75

1.70

1.50

1.70

1.53

1.40

1.62

1.38

1.50

1.49

1.54

2.08

2.08

1.65

14


1.55

2.09

3.20

1.68

2.02

2.02

1.65

1.95

1.66

1.60

1.95

1.55

1.65

1.70

1.72

2.72

2.80

2.15

15


1.39

1.45

1.40

1.52

1.57

1.48

1.46

1.44

1.51

1.30

1.35

1.30

1.44

1.45

1.41

1.26

1.30

1.25

16


1.76

1.87

1.70

1.90

1.92

1.87

1.65

1.65

1.70

1.50

1.58

1.50

1.74

1.81

1.69

1.42

1.47

1.47

17


2.30

2.60

2.05

2.50

2.35

2.33

1.87

1.85

1.88

1.72

1.80

1.72

2.00

2.14

1.93

1.65

1.70

1.75

18

A/B > 2.00

2,570 2,435 2,284 2,179 2,236 2,262 2,363 2,772 2,640 2,216 2,522 2,570 1,620 1,602 1,446 1,377 1,298 1,063

19

B/A > 2.00

1,556 1,714 1,901 1,790 1,843 1,916 1,351 1,351 1,453 1,373 1,432 1,451 1,382 1,371 1,254 2,008 1,969 2,227

20

A/B > 2.00 (P < 0.001)

2,504 2,393 2,249 2,136 2,197 2,227 2,339 2,757 2,616 2,200 2,508 2,545 1,620 1,602 1,430 1,377 1,290 1,045

21

B/A > 2.00 (P < 0.001)

1,458 1,673 1,883 1,672 1,802 1,901 1,340 1,347 1,443 1,356 1,427 1,437 1,382 1,365 1,238 2,004 1,942 2,146

– – Row 1 lists the number of genes detected in all four samples for each platform, separated by site. Rows 2 and 3 represent the number of concordantly detected genes for A > B – – and B > A, respectively. The sum of rows 2 and 3 for each column is identical to the gene count in row 1. Rows 4 and 5 represent the number of concordantly detected, statistically – – – – – – – – – – –– – significant (P < 0.001) genes for A > B and B > A. Rows 6 and 7 represent the number of detected genes that show the monotonic titration trends A > C > D > B and B > D > C > A. – – – – – – –– – Rows 8 and 9 represent the number of statistically significant (P < 0.001), concordantly detected genes that show the monotonic titration trends A > C > D > B and B > D > C > A. The statistical test used was a two-sample t-test, using equal variance, calculated within each site and comparing log2 expression values between the independent samples A and B. The gene counts in rows 8 and 9 are also indicated in Figure 2 for each monotonic direction. Rows 10 and 11 translate the previous rows into percentages of genes showing the monotonic titration trend. Rows 12–17 summarize Figure 2 for three specific y-axis values (50%, 75% and 90% of genes titrate at the listed average fold changes). Rows 18 and – – – – 19 show the numbers of genes for which A / B > 2 and B / A > 2. Rows 20 and 21 show the numbers of statistically significant (P < 0.001) genes used to create the box plots in Figure 4. Columns highlighted in blue, for each platform, represent the manufacturer’s recommended normalization methods used in the main MAQC paper13. More detailed gene counts with cross-site intersections can be found in Supplementary Table 1 online.

1126


A N A LY S I S

Fraction of genes that titrate


ABI - quantile

ABI - scaling

AG1 - median scaling

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0 –5

0

5

10

15

20

0 –5

0

AFX - PLIER

5

10

15

0 –5

20

AFX - MAS5

0.1 0

5

10

15

0 –5

20

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

5

10

15

20

GEH - median scaling

0 –5

0

5

10

15

0 –5

20

GEH - quantile

5

10

15

0 –5

20

ILM - quantile

0.6

0.6 0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

5

10

15

20

0 –5

0

5

10

15

0 –5

20

15

20

15

20

0.1 0

0.5

0

10

0.6

0.6

0 –5

5

AFX - GCRMA

0.6

0

0

AFX - RMA

0.6

0 –5

AG1 - 75th percentile scaling

0.6

0

5

10

ILM - invariant scaling

0.1 0

5

10

15

0 –5

20

0

5

10

15

20

Average log2 signal A>C>D>B

Site 1

Site 2

Site 3

B>D>C>A

Site 1

Site 2

Site 3

Figure 3 Impact of normalization on the distributions of titrating genes as a function of signal intensity. Fractions of genes showing the monotonic titration – – – – – – –– – – – responses Ai > Ci > Di > Bi and Bi > Di > Ci > Ai are plotted against Bi (solid line) and Ai (dashed line), respectively. Histograms in each panel represent data from a different platform and normalization technique, separated by site and direction. Normalization methods highlighted in yellow for each platform are the manufacturer’s recommended method used in the MAQC study. Blue, site 1; red, site 2; gray, site 3. The data for these graphs were generated from the set of 12,091 genes common across the platforms that were significantly differentially expressed (P < 0.001) in samples A and B and detected in all four samples (Table 1, rows 4 and 5). All data are plotted on the same scale: the x-axis is normalized signal in log2 units and the y-axis shows the fraction of titrating probes relative to the total number of probes in the given intensity range. Bin centers are 0.5 apart on the log2 scale. To avoid spurious oscillations in the lowest and highest signal intensities, we plotted only bins with more than ten genes. Differences between normalization techniques are demonstrated by the differing signal ranges within a platform for the monotonic titration response. The normalization methods highlighted in yellow for each platform represent the manufacturer’s recommended method used in the MAQC main paper13.

but when B > A (dotted lines), these data show a sudden increase in that fraction at high intensity. This effect, although still present, is much less distinct for the scaled than for the quantile-normalized data. We saw improved reproducibility among sites and concordance between the two titration trends in the AG1 75th percentile scaling relative to the median scaling. For the AFX-PLIER data, the signal range across which a titration response is elicited is smaller than for the other platforms and normalization methods, possibly owing to the variance stabilization used in the PLIER method. In all cases, the AFX data show lower percentages for site 1, as in Figure 2. For the GEH data, median normalization results in a very clear distinction between the two different titration patterns; this distinction is moderated by quantile normalization. The data for the ILM rank invariant scaling indicate a larger number of – – – – – genes showing the titration response Bi > Di > Ci > Ai than showing the opposite trend, a result not seen for any other platform or normalization method. Unlike in Figure 2, the percentage of titrating genes never reaches 100% because, at all signal ranges, some genes show only very small differences in expression across the samples and are more likely to yield a near-random ordering in their titration responses. Analysis of titration mixtures An underlying assumption for this study was that the proportions of each mRNA in the mixture samples (C and D) from each of the original samples (A and B) are equivalent to the mixing proportions of the total RNA. For this assumption to be true, the fractions of each mRNA in the total RNA samples A and B had to be the same and had to be processed by the various biochemical systems with equal efficiencies. Using mathematical modeling, we investigated whether we could derive the relative mRNA contents of the two independent samples using the microarray data from the independent and titration samples (see Methods). Such


modeling defines the true fractions of mRNA derived from sample A in titration samples C and D as αC and αD, and the true fractions of mRNA derived from sample B in titration samples C and D as βC and βD (see Box 1 and Supplementary Fig. 5). Figure 4 shows the results of this modeling for all the platforms and normalization methods, with the y-axes representing the estimates of βC (bottom) and βD (top). The lower charts show median values of βC centered on 0.18 but usually – – – – larger for Ai > Bi (left) than for Bi > Ai (right), and the upper charts show median values of βD centered on 0.67. These deviations from the expected values of 0.25 and 0.75 based on the 3:1 mixtures of total RNA suggest that the mRNA concentrations of the A and B samples were not identical. From these results, we estimate the mRNA concentration in the B sample to be approximately two-thirds of the concentration in the A sample (see Box 1). An empirical evaluation of mRNA content in samples A and B is consistent with our estimates of 3% and 2%, respectively (see Methods). The values calculated from the different platforms and normalization methods are generally similar, with two clear exceptions. For ILM, invariant scaling results in much lower estimates for βC and βD than the other platforms and normalization methods when A > B (left side) but not when B > A. This difference is consistent with the results noted for the titration response (Figs. 2 and 3). For ABI, the estimates of βC and βD are consistent with the other platforms when A > B but lower than the other platforms when B > A. This result was seen with both normalization methods, although to different extents, and may be related to the differences noted in Figure 3. The deviations for βC and βD are particularly noteworthy because of the relatively small errors of the ABI data in this analysis. The individual microarray measurements for the titration coefficients shown in Figure 4 indicate that normalization and data-processing

1127

A N A LY S I S Box 1 Modeling of titration mixtures Ideally, the mRNA expression levels of each gene in samples C and D may be mathematically expressed as C = αCA + βCB and D = αDA + βDB,


where A and B are the measured mRNA abundances of the gene in samples A and B, respectively, and αC, βC, αD and βD are the mixture coefficients. If we impose the requirement that

for different platforms, sites and normalization methods can lead to deviations from these expected values (Fig. 4). For example, if the mRNA fractions for the A and B samples (termed a and b, respectively) are unequal (a ≠ b), then C = ((0.75a)A + (0.25b)B)/(0.75a + 0.25b) and D = ((0.25a)A + (0.75b)B)/(0.25a + 0.75b).

αC + βC =1 and αD + βD = 1 (if A = B, then C = A = B = D), then elementary algebra can be used to derive simple formulas for βC and βD: βC = (C – A)/(B – A) and βD = (D – A)/(B – A). If the mRNA fractions in samples A and B are identical and the normalization of samples A, B, C and D exactly the same, then the measured fraction should be centered on the ideal mixture fractions of βC = 0.25 and βD = 0.75 (implying αC = 0.75 and αD = 0.25). However, different mRNA concentrations in the A and B samples and differences in the normalization of the four samples

differences are not the primary cause for the deviations from the theoretical values. Differences in mRNA abundance contribute to these deviations and may not be circumvented with normalization alone. Additionally, further analysis of microarray measurements from these titration mixtures may provide greater-resolution observations of the global tendency (Fig. 4) of estimates of βC and βD to be larger for A > B than for B > A (see Supplementary Fig. 1 online). Effects of outlier data During execution and analysis of the MAQC study, the consortium identified one outlier site and multiple outlier arrays on the basis of objective criteria of data quality13. In some cases, we evaluated the effects of not censoring such data from the analysis. The results (data not shown) were as expected: inclusion of low-quality data degraded both intra- and intermethod reproducibility. This result, although predictable, is nonetheless noteworthy because microarray experiments are expensive and are sometimes used to analyze samples that are available in very limited quantities. Low-quality microarray data are discarded with great pain. It is therefore important that the community develop shared standards of microarray data quality to allow use and interpretation of less-thanperfect data while preventing overinterpretation. The well-characterized RNA samples and all of the data (including outliers) produced by the MAQC study are a good start on the road to such data-quality standards. In particular, the titration experimental design used in this work may prove to be an important tool for developing such standards, as the experiments can be interpreted using a small number of plausible assumptions. DISCUSSION The MAQC titration study was conceived as an experiment that could be implemented across several platforms, with a minimum of assumptions. One of the initial goals of the titration study was to assess relative accuracy by comparing observed expression in the titration samples with the expression expected on the basis of the known mixing ratios

1128

We can express the true ratios of the B to A mRNA fractions, b/a = 3βC/(1 – βC) = βD/3(1 – βD) (see Supplementary Fig. 5). Using the empirical measurements of βC and βD, we can then estimate these true mRNA fractions. For example, if the B fraction of sample C is βC ≈ 0.18, as indicated by microarray median values in Figure 4 (bottom), then we can deduce that the true ratio of mRNA fractions b/a is approximately 2:3. Moreover, these results predict that βD = 9βC/(1 + 8βC) ≈ 0.67, which is consistent with the empirical microarray results in Figure 4 (top).

of the two independent samples. This analysis proved to be more complex than originally anticipated, largely owing to the effects of different mRNA fractions in the two independent samples. However, the qualitative expectation of a particular signal ordering is still valid and provides a sensitive tool for differentiating microarray platform performance and normalization methods. As the measurement of titration response illustrates, different platforms and data analysis methods have slightly different performance optima: design and processing choices that increase the number of detected genes also tend to increase noise in the titration series. In addition to differences in the number of genes analyzed, the variations seen in Figure 2 and Table 1 can also result from differences in expression-ratio compression (leading to different ratios observed for any given gene) as well as levels of noise in each measurement. In general, the behaviors of various sites and platforms are quite similar. The analysis of the titration mixtures reveals some interesting observations about the data. These results show asymmetry in the titration responses (Figs. 2 and 3) and the estimates of the true fractions of mRNA in the titration samples (Fig. 4). This asymmetry may be caused in part by additional differences in the normalization of the A and B samples (Supplementary Fig. 1), may relate to more difficulty in distinguishing A and C at low signal or may be a consequence of nonlinearity in the signal response relative to the concentration amounts (Supplementary Fig. 2 online). In addition, the results presented here demonstrate that the mRNA content of the two independent samples is not equal. This conclusion is supported by additional lines of evidence. First, an apparent power analysis27–30 (Supplementary Figs. 3 and 4 online) is asymmetric between the sample pairings (A, C) and (B, D). This asymmetry is probably the result of the A sample being more similar to C than B is to D. Second, the slopes of the linear trends for the titration sample/independent sample ratios (Supplementary Fig. 1) suggest that the ratio of sample A to B in sample C differs from the expected value from the total RNA ratios. Third, external spike-in RNA controls were included for several platforms; these controls were amplified and labeled along with the sample RNA and indicate that the A sample contains a higher percentage


βD = (D – A)/(B – A)

of mRNA relative to the B sample31. Finally, a preliminary empirical analysis of mRNA content in the A and B samples (see Methods) confirmed that the mRNA content differs between the samples. The discovery of a difference in the mRNA content of samples A and B has important implications for the future use of these commercially available samples in method calibration, proficiency testing and other activities requiring well-characterized, complex RNA. As a result of the MAQC study, these samples are probably the best-characterized complex RNA preparations available. The RNA-measurement community should complete the characterization of these samples by more accurately measuring the fraction of mRNA in each preparation, so that the scientific community can make better use of this resource. The utility of the titration samples for assessing normalization and data preprocessing methods can be seen throughout the analyses presented here. Notably, for all platforms except AFX and ILM, the performance of the MAQC ‘standard’ normalization or data preprocessing method was slightly inferior to that of the secondary method, especially in the apparent power analysis (Supplementary Fig. 3). This result highlights the observation noted throughout this study that data processing methods determined to be optimal under one set of circumstances may not always prove appropriate under all conditions, particularly if primary assumptions underlying those data processing methods are violated. A great strength of the design presented here is that, despite the added complexities of varying mRNA content, the qualitative expectation of a particular signal ordering is still valid, provided that the different data sets are properly scaled relative to one another. Therefore, this design is very valuable for assessing microarray performance. Specifically, as we have shown here, the titration response can be used to distinguish between normalization methods that are sensitive to changes in mRNA fraction and methods that are robust despite such changes. One observation of this study is that the robustness of a normalization

method depends in part on the subset of data used to determine the scaling constant or function. Our results indicate a path toward objective optimization of this normalization set. The differences in gene expression among samples may be greater and the variability across replicates may be smaller in this study than in typical biological experiments; nonetheless, the lessons learned regarding the use of titration mixtures to evaluate the performance and normalization of large-scale gene-expression measurements may have widespread application in more realistic settings. In addition, the wide range of gene expression in these samples probably served to amplify data processing–derived differences that would have been more difficult to detect in analyses of more closely matched samples. Finally, it should be noted that the majority of genes considered here yielded very similar behavior across all platforms, in spite of the complications noted in this manuscript. Therefore, these results should be considered a testament to the underlying strength of all of the methods examined. Improvement of mRNA quantification methods remains an important objective, and the MAQC study has produced samples and data that will aid the community in making such improvements. The concordance of data presented here demonstrate that the methods used are sound and, when properly implemented and interpreted, can be used to measure expression levels of thousands of RNA targets simultaneously. METHODS Preparation of the RNA sample titrations. RNA samples are described in detail in the main MAQC paper13. Briefly, two commercially available total RNA solutions and 3:1 and 1:3 mixtures were chosen at the outset by the members of the MAQC project. For simplicity, these samples were designated as A, B, C and D. A and B are independent total RNA samples. A is derived from a collection of ten human cell lines and B from human brain tissue. Sample A is sold commercially under the name Universal Human Reference RNA (Catalog number 740000, Stratagene). Sample B is sold commercially under the name FirstChoice Human Brain Reference RNA (Catalog number 6050, Ambion).

A>B

B>A

1.0

1.0

0.9

0.9

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2 0.1

0.1

0

0

Quantile

Scaling

PLIER

ABI

βC = (C – A)/(B – A)


A N A LY S I S

MAS 5.0

RMA

GCRMA

AFX

Scaling Scaling 75

AG1

Scaling

Quantile

GEH

Quantile

Scaling

Scaling

Quantile

ILM

PLIER

ABI

1.0

1.0

0.9

0.9

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

MAS 5.0

RMA

GCRMA

AFX

Scaling

Scaling 75

AG1

Scaling

Quantile

GEH

Quantile

Scaling

ILM

0

0

Quantile

Scaling

ABI

PLIER

MAS 5.0

AFX

RMA

GCRMA

Scaling Scaling 75

AG1

Scaling

Quantile

GEH

Quantile

Scaling

Quantile

ILM Site 1 쐽

Scaling

PLIER

ABI Site 2 쐽

MAS 5.0

AFX

RMA

GCRMA

Scaling Scaling 75

AG1

Scaling

Quantile

GEH

Quantile

Scaling

ILM

Site 3 쐽

Figure 4 Titration-response concordance for each commercial whole-genome microarray platform, using different normalization methods, with data from each platform separated by site and fold-change direction. Data shown are from the 12,091 genes common across whole-genome platforms. Box plots were generated in cases where a gene was detected across all samples per site and had a statistically significant (P < 0.001) A/B ratio >2 in the direction indicated. A two-sample t-test, with equal variance, was performed within each site on log2 expression values. Data for each site were split by direction of fold change: left, genes where A/B > 2; right, genes where B/A > 2 (all differences significant, P < 0.001, for both directions). Number of genes used for each box plot is indicated by individual site counts in Table 1 (rows 20 and 21). Each box represents the interquartile range, with median marked by a horizontal black line and 10th and 90th percentiles marked by the outer whiskers. Blue, site 1; red, site 2; gray, site 3. The horizontal dashed black lines represent expected values assuming 3% and 2% mRNA abundance levels for samples A and B, respectively. In other words, when the mRNA/total RNA fraction in A is equal to 3% and in B is equal to 2%, then βC = (C – A)/(B – A) = 0.18 (bottom two charts) and βD = (D – A)/(B – A) = 0.67 (top two charts). Refer to Box 1 for further details. Normalization methods highlighted in yellow for each platform represent the manufacturer’s recommended method used in the MAQC main paper13.


1129


A N A LY S I S RNA titration samples were generated once for all MAQC experiments (Fig. 1), with samples A and B at equal concentrations as measured by A260. Sample C was made by mixing sample A with sample B at a volumetric ratio of 75:25, and sample D was made by mixing sample A with sample B at a volumetric ratio of 25:75.

negative control probes, which are thermodynamically equivalent to regular probes but do not have specific targets in the transcriptome. Gene signals were ranked relative to signals of negative controls, and the detection flag was set to present if gene signal exceeded 99% of signals of negative controls.

Normalization methods used in this study. For ABI, we used quantile normalization17 independently for each test site and 90% trim mean scaling. For trim mean scaling, the signals for highest 5% and lowest 5% are removed, and the remaining 90% of signals are used to calculate the mean. The mean of each array is scaled to the same level, and the scaling factor for each array is used to scale the signals. The trim mean scaling was calculated independently for each test site. For AG1, the data were transformed so that signal values below 5 were set to 5. After this transformation, each measurement was divided by the median of all detected measurements in that sample (for median scaling) or by the 75th percentile of all measurements in that sample (for 75th percentile scaling). For AFX data, we used PLIER21, MAS 5.0, RMA18 and GCRMA27 for data preprocessing and normalization. The PLIER method produces a summary value for a probe set by accounting for experimentally observed patterns in feature behavior and handling error appropriately at low and high abundance. PLIER accounts for the systematic differences between features by means of parameters termed feature responses, using one such parameter per feature (or pair of features, when using mismatch (MM) probes to estimate cross-hybridization signal intensities for background). Feature responses represent the relative differences in intensity between features hybridizing to a common target. PLIER produces a probe-set signal by using these feature responses to interpret intensity data, applying dynamic weighting by empirical feature performance and handling error appropriately across low and high abundances. Feature responses are calculated using experimental data across multiple arrays. PLIER also uses an error model that assumes error is proportional to the observed intensity rather than to the background-subtracted intensity. This ensures that the error model can adjust appropriately for relatively low and high abundances of target nucleic acids. Here, PLIER was run with the default options (quantile normalization and PM-MM) with the addition of a 16 offset to each expression value13. The AFX MAS 5.0 algorithm is a method for calculating probe-set signal values. The MAS 5.0 algorithm is implemented on a chip-by-chip basis and is not applied across an entire set of chips. The signal value is calculated from the background-adjusted PM and MM values of the probes in the set using a robust biweight estimator. Here, MAS 5.0 is implemented with default options, and global scaling (96% trim mean) is used for normalization. RMA18 fits a robust linear model to the probe-level data and conducts a multichip analysis. The algorithm includes a model-based background correction, quantile normalization and an iterative median polishing procedure to generate a single expression value for each probe set. GCRMA substantially refines the RMA algorithm by replacing the model for background correction with a more sophisticated computation that uses each probe’s sequence information to adjust the measured intensity for the effects of nonspecific binding, according to the different bond strengths of the two types of base pairs. It also takes into account the optical noise present in data acquisition. Both RMA and GCRMA were implemented using the ArrayAssist Lite package with default settings (Affymetrix; http://www.affymetrix.com/products/software/specific/arrayassist_lite.affx). For GEH data, we compared median scaling and quantile normalization. For the median-scaling approach, each measurement was divided by the median of all measurements within each array. Therefore, the median signal is scaled to 1 for each array. The quantile normalization approach16 was applied to log2-transformed expression values across all samples and replicates within each site. For ILM data, we compared quantile normalization16 with the addition of 15 counts of offset to each probe signal13 and normalization by a robust leastsquares fit of rank-invariant genes. For the latter normalization method, array data corresponding to sample A were averaged and used as a reference on each site independently. Signals from each array in the experiment were compared to the reference, and probes with relative rank changes of less than 5% (only probes ranked between the 50th and 90th percentiles were included) were considered to be rank invariant. Normalization coefficients were computed with iteratively reweighted linear least squares using the Tukey bisquare weight function. Background signal, estimated as the mean signal of negative controls, was subtracted before normalization. Each ILM array contains approximately 1,600

Purification of mRNA to empirically determine abundance in samples A and B. In a follow-up experiment, mRNA was isolated from 100 µg of samples A and B total RNA in duplicate using the Absolutely mRNA purification kit (Stratagene) according to the manufacturer’s protocol. Briefly, 50 µl of mRNA oligo (dT) magnetic particles were combined with 100 µl of total RNA and washed four times, and mRNA was eluted with 100 µl elution buffer. mRNA quantity and quality were evaluated by ND-1000 NanoDrop spectrophotometer (NanoDrop Technologies) and Agilent 2100 Bioanalyzer with RNA 6000 Nano LabChip Kit (Agilent Technologies). This empirical evaluation of mRNA content in each 100 ng of total RNA produced an average yield of 2.870 ± 0.095 ng for sample A and 2.003 ± 0.124 ng for sample B (mean ± s.d.).

1130

Note: Supplementary information is available on the Nature Biotechnology website. ACKNOWLEDGMENTS This study used a number of computing resources, including the high-performance computational capabilities of the Biowulf PC/Linux cluster at the US National Institutes of Health in Bethesda, Maryland (http://biowulf.nih.gov). This research was supported in part by the Intramural Research Program of the US National Institutes of Health, National Library of Medicine. DISCLAIMER This work includes contributions from, and was reviewed by, the FDA and the NIH. This work has been approved for publication by these agencies, but it does not necessarily reflect official agency policy. Certain commercial materials and equipment are identified in order to adequately specify experimental procedures. In no case does such identification imply recommendation or endorsement by the FDA or the NIH, nor does it imply that the items identified are necessarily the best available for the purpose. COMPETING INTERESTS STATEMENT The following authors declare competing financial interests (see the Nature Biotechnology website for details). Published online at http://www.nature.com/nbt/ Reprints and permissions information is available online at http://npg.nature.com/ reprintsandpermissions/

1. Barczak, A. et al. Spotted long oligonucleotide arrays for human gene expression analysis. Genome Res. 13, 1775–1785 (2003). 2. Barnes, M., Freudenberg, J., Thompson, S., Aronow, B. & Pavlidis, P. Experimental comparison and cross-validation of the Affymetrix and Illumina gene expression analysis platforms. Nucleic Acids Res. 33, 5914–5923 (2005). 3. Dobbin, K.K. et al. Interlaboratory comparability study of cancer gene expression analysis using oligonucleotide microarrays. Clin. Cancer Res. 11, 565–572 (2005). 4. Dorris, D.R. et al. Oligodeoxyribonucleotide probe accessibility on a three-dimensional DNA microarray surface and the effect of hybridization time on the accuracy of expression ratios. BMC Biotechnol. 3, 6 (2003). 5. Hughes, T.R. et al. Expression profiling using microarrays fabricated by an ink-jet oligonucleotide synthesizer. Nat. Biotechnol. 19, 342–347 (2001). 6. Irizarry, R.A. et al. Multiple-laboratory comparison of microarray platforms. Nat. Methods 2, 345–350 (2005). 7. Larkin, J.E., Frank, B.C., Gavras, H., Sultana, R. & Quackenbush, J. Independence and reproducibility across microarray platforms. Nat. Methods 2, 337–344 (2005). 8. Li, J., Pankratz, M. & Johnson, J.A. Differential gene expression patterns revealed by oligonucleotide versus long cDNA arrays. Toxicol. Sci. 69, 383–390 (2002). 9. Naef, F., Socci, N.D. & Magnasco, M. A study of accuracy and precision in oligonucleotide arrays: extracting more signal at large concentrations. Bioinformatics 19, 178–184 (2003). 10. Shippy, R. et al. Performance evaluation of commercial short-oligonucleotide microarrays and the impact of noise in making cross-platform correlations. BMC Genomics 5, 61 (2004). 11. Yuen, T., Wurmbach, E., Pfeffer, R.L., Ebersole, B.J. & Sealfon, S.C. Accuracy and calibration of commercial oligonucleotide and custom cDNA microarrays. Nucleic Acids Res. 30, e48 (2002). 12. Chudin, E. et al. Assessment of the relationship between signal intensities and transcript concentration for Affymetrix GeneChip arrays. Genome Biol. 3, RESEARCH0005 (2002). 13. MAQC Consortium. The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nat. Biotechnol. 24, 1151–1161 (2006).



A N A LY S I S 14. Shi, L. et al. Cross-platform comparability of microarray technology: intra-platform consistency and appropriate data analysis procedures are essential. BMC Bioinformatics 6 (Suppl.) S12 (2005). 15. Thompson, K.L. et al. Use of a mixed tissue RNA design for performance assessments on multiple microarray formats. Nucleic Acids Res. 33, e187 (2005). 16. Bolstad, B.M., Irizarry, R.A., Astrand, M. & Speed, T.P. A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics 19, 185–193 (2003). 17. Irizarry, R.A. et al. Summaries of Affymetrix GeneChip probe level data. Nucleic Acids Res. 31, e15 (2003). 18. Irizarry, R.A. et al. Exploration, normalization, and summaries of high density oligonucleotide array probe level data. Biostatistics 4, 249–264 (2003). 19. Irizarry, R.A., Wu, Z. & Jaffee, H.A. Comparison of Affymetrix GeneChip expression measures. Bioinformatics 22, 789–794 (2006). 20. Parrish, R.S. & Spencer, H.J. III. Effect of normalization on significance testing for oligonucleotide microarrays. J. Biopharm. Stat. 14, 575–589 (2004). 21. Guide to probe logarithmic intensity error (PLIER) estimation. Affymetrix Technical Note 22. Statistical algorithms description document. Affymetrix


23. Cope, L.M., Irizarry, R.A., Jaffee, H.A., Wu, Z. & Speed, T.P. A benchmark for Affymetrix GeneChip expression measures. Bioinformatics 20, 323–331 (2004). 24. Wu, Z. & Irizarry, R.A. Stochastic models inspired by hybridization theory for short oligonucleotide arrays. J. Comput. Biol. 12, 882–893 (2005). 25. Sendera, T.J. et al. Expression profiling with oligonucleotide arrays: technologies and applications for neurobiology. Neurochem. Res. 27, 1005–1026 (2002). 26. Wu, Z. , Irizarry, R.A., Gentleman, R., Martinez Murillo, F. & Spencer, F. A model based background adjustment for oligonucleotide expression arrays. J. Am. Stat. Assoc. 99, 909–917 (2004). 27. Seo, J., Gordish-Dressman, H. & Hoffman, E.P. An interactive power analysis tool for microarray hypothesis testing and generation. Bioinformatics 22, 808–814 (2006). 28. Hwang, D., Schmitt, W.A. & Stephanopoulos, G. Determinatoin of minimum sample size and discriminatory expression patterns in microarray data. Bioinformatics 18, 1184–1193 (2002). 29. Tibshirani, R. A simple method for assessing sample sizes in microarray experiments. BMC Bioinformatics 7, 106 (2006). 30. Page, G.P. et al. The PowerAtlas: a power and sample size atlas for microarray experimental design and research. BMC Bioinformatics 7, 84 (2006). 31. Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006).

1131


A N A LY S I S

Evaluation of external RNA controls for the assessment of microarray performance Weida Tong1, Anne Bergstrom Lucas 2, Richard Shippy3, Xiaohui Fan1,4, Hong Fang5, Huixiao Hong5, Michael S Orr6, Tzu-Ming Chu7, Xu Guo8, Patrick J Collins2, Yongming Andrew Sun9, Sue-Jane Wang6, Wenjun Bao7, Russell D Wolfinger7, Svetlana Shchegrova2, Lei Guo1, Janet A Warrington8 & Leming Shi1 External RNA controls (ERCs), although important for microarray assay performance assessment, have yet to be fully implemented in the research community. As part of the MicroArray Quality Control (MAQC) study, two types of ERCs were implemented and evaluated; one was added to the total RNA in the samples before amplification and labeling; the other was added to the copyRNAs (cRNAs) before hybridization. ERC concentration-response curves were used across multiple commercial microarray platforms to identify problematic assays and potential sources of variation in the analytical process. In addition, the behavior of different ERC types was investigated, resulting in several important observations, such as the sample-dependent attributes of performance and the potential of using these control RNAs in a combinatorial fashion. This multiplatform investigation of the behavior and utility of ERCs provides a basis for articulating specific recommendations for their future use in evaluating assay performance across multiple platforms. ERCs are synthetic or naturally occurring RNA species that are added to an RNA sample for the purpose of quality control of the assay. Most commercial microarray platforms contain probes specifically designed for interrogating ERC transcripts. These probes have been extensively prototyped and optimized for performance on each microarray platform. To provide an enhanced assessment of the analytical performance of the system during data collection, a variety of ERCs can be added to the sample in a range of concentrations spanning high to low abundance by

1National Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Rd., Jefferson, Arkansas 72079, USA. 2Agilent Technologies, Inc., 5301 Stevens Creek Blvd., Santa Clara, California 95051, USA. 3GE Healthcare, 7700 S. River Pkwy., Suite #2603, Tempe, Arizona 85284, USA.4Pharmaceutical Informatics Institute, Zhejiang University, Hangzhou 310027, China. 5Z-Tech Corporation, National Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Rd., Jefferson, Arkansas 72079, USA. 6Center for Drug Evaluation and Research, US Food and Drug Administration, 10903 New Hampshire Ave., Silver Spring, Maryland 20993, USA. 7SAS Institute Inc., SAS Campus Drive, Cary, North Carolina 27513, USA. 8Affymetrix, Inc., 3420 Central Expressway, Santa Clara, California 95051, USA. 9Applied Biosystems, 850 Lincoln Centre Dr., Foster City, California 94404, USA. Correspondence should be addressed to W.T. ([email protected]).

Published online 8 September 2006; doi:10.1038/nbt1237

1132

evaluating assay performance across the expected range of concentrations in the sample1. A well-constructed concentration-response series of ERCs is useful in many ways for assessing assay performance. Depending on the point in the assay the ERCs are added, they can be used to identify potentially failed steps during the assay process. Realizing the potential importance of ERCs for analytical performance assessment, the External RNA Control Consortium (ERCC) was established in 2003 with the objective of developing a set of ERC transcripts that could be used with various gene expression profiling technologies, including microarray platforms2. ERCs can also be useful for evaluating different data analysis methods3. The cRNA data set from Affymetrix, known as the Latin square data set (http://www.affymetrix.com/support/technical/sample_data/datasets. affx), consists of data from 42 cRNAs, which were prelabeled and added to a hybridization solution at various known concentrations. A similar data set is also provided by GeneLogic (http://www.genelogic.com/newsroom/ studies/index.cfm). Both data sets are freely available and have been widely used in the research community for comparative performance analysis of GeneChip-specific normalization and gene selection methods4–7. Recently, Choe et al.8,9 demonstrated the value of using a large number of cRNA transcripts at concentration ratios varying from one- to fourfold to compare the performance of different data analysis scenarios. The MAQC study10 provides a rich data resource to investigate various issues associated with DNA microarray platforms, including the performance of ERCs across various platforms. In this project, the probes for the ERC transcripts (Supplementary Methods online) are unique nonmammalian sequences selected to minimize cross-hybridization with transcripts from mammalian species such as human, mouse and rat. Seven microarray platforms were evaluated and ERCs were used in the following platforms: Applied Biosystems Genome Survey Microarray, Affymetrix GeneChip, both Agilent’s One-Color and Two-Color platforms, GE Healthcare CodeLink and Eppendorf (data not shown). With these data sets, the following questions were asked: (i) Do the ERCs behave in the expected manner? (ii) Can outlying assays be identified using ERCs? (iii) Can ERCs assess the accuracy of ratios between different samples? (iv) Can ERCs provide information other than assay quality? (v) How does the choice of normalization and data processing methods affect the ERCs data? RESULTS The utility and performance behavior of ERCs were investigated using two independent sets of data; the MAQC data set10 and rat toxicogenomics



A N A LY S I S (TGx) data set11. Because the results in this paper are derived from two independent experiments the following nomenclature is used to provide clarity. The subset of the MAQC data set used for the present analysis corresponds to four genome-wide commercial microarray platforms, Affymetrix GeneChip (AFX), Applied Biosystems Genome Survey Microarray (ABI) and Agilent One-Color (AG1) and Agilent Two-Color (AGL) microarrays. Data were generated for each of these platforms by three different test sites with five technical replicates for each of the four RNA samples (A, B, C and D10,12). Each data set is denoted by platform_site_replicate; for example, AG1_2_A1 denotes Agilent One-Color platform, test site 2, sample A and replicate 1. The rat TGx data set that is denoted by platform_RAT contains data from Affymetrix (AFX_Rat), Agilent One-Color microarray (AG1_Rat), Applied Biosystems (ABI_Rat) and GE Healthcare (GEH_Rat). This experiment was performed at one test site with six biological replicates for each of six different treatments. The nomenclature for the site, mentioned above, is therefore not applicable, yet it’s necessary to make a distinction between samples and that is provided in Methods and within the figures. Two types of ERCs were investigated. One type is added to the total RNA (called tERC hereafter) before initiating the cDNA synthesis and in vitro transcription steps of the RNA labeling procedure. When added in this manner, the tERC generally assesses the efficiency of the target preparation as well as the performance of the hybridization and scanner. The other type of ERC is added to the cRNA (called cERC hereafter) immediately before hybridization, which allows assessment of the assay performance from the hybridization onward. Applied Biosystems and Affymetrix platforms used both types of ERCs in their respective protocols, whereas Agilent used tERC and GE Healthcare used cERC only (Fig. 1). The concentration-response behavior of both tERCs and cERCs was evaluated using a linear regression analysis in an effort to identify microarray assays that show outlier behavior. This is a favorable approach as the analysis is self-contained within each microarray, and therefore, does not require replicates to assess outliers. The behavior of both ERC types was investigated further to determine if additional ERC-specific analysis methods could be useful for analytical performance assessment. External RNA control concentrationresponse curves The ERC transcripts span a range of concentrations in the Affymetrix, Agilent and GE Healthcare microarray platforms, making them suitable for concentration-response analyses. The Agilent One-Color platform has ten tERCs that span six logs of concentration and interrogate the lower and upper limits of assay signal detection (Supplementary Table 1 online). The Affymetrix platform has four tERCs that span one and a half logs of concentration and the GE Healthcare platform has six ERCs that span three logs of concentration. For the Applied Biosystems microarray platform, ERC controls are spiked at a single fixed concentration, rendering them unsuitable for a concentration-response analysis. Figure 2 depicts the concentration-response curves for AG1, AFX and GEH_RAT. In general, all platforms exhibited accurate concentrationresponse patterns. In addition, performance

differences are observed for tERCs relative to cERCs as seen in the data from AFX where the tERCs show decreased linear correlations compared to the cERC plots (Fig. 2, comparing the second and third rows of graphs for the AFX platform). This result is somewhat expected as the tERCs are introduced earlier in the assay process and are subject to multiple sources of variation introduced during sample amplification and labeling, more closely approximating the analytic manipulation. In contrast, the cERCs are added just before hybridization, and their more stable performance reflects fewer sample manipulations after these controls are added. Two assays generated by AG1 site 2 (AG1_2_D2 and AG1_2_A3) have noticeably higher signals for tERCs at the lowest concentrations, indicating potential assay outliers. However, the specific problematic step of the assay for these two data sets cannot be identified because the behavior of tERC reflects the performance associated with multiple steps of the experiment. The benefit of using both tERCs and cERCs is demonstrated with the AFX platform, where the combination was used to elucidate procedural problems in the assay. In this example the AFX cERC performance is stable and consistent across all three test sites, but tERCs in site 1 have lower y-intercepts as compared to the other two sites, indicating that for site 1 the target preparation yield or labeling efficiency differed from the other sites (Fig. 2). Concentration-response curves in one-color microarray assays In addition to visually inspecting the concentration-response curves to interrogate the performance over the dynamic range of an assay, we calculated the linear regression statistics of the linear portion of the curves for outlier identification, including R2 correlations and slopes. Figure 3 (Supplementary Table 2 online) plots the linear regression slope versus R2 correlations for AG1, AFX and GEH_Rat. Three outlying assays were identified for AG1 site 2 (Fig. 3a); AG1_2_D1 has a normal R2 with a low slope, whereas AG1_2_D2 has a normal slope with a low R2 and AG1_2_A3 has both low slope and R2. An assay with a concentration-response slope of one indicates no compression of the signal because values of x and y are identical across the regression fit. By inspecting the slopes in Figure 3, different degrees of compression in gene expression data are observed between three

Microarray assay process Total RNA samples

tERC added to

RT

cRNA Fragmentation Fragmented cRNA

Affymetrix: four poly-A controls Agilent: ten in vitro synthesized, polyadenylated transcripts for both one- and two-color arrays

cDNA IVT

Applied Biosystems: three IVT controls and three RT controls

cERC added to cERC added to

GE Healthcare: six positive controls Applied Biosystems: three hybridization controls Affymetrix: four hybridization controls

Hybridization

Figure 1 Overview of external RNA controls (ERCs) implemented in Affymetrix, Agilent, Applied Biosystems and GE Healthcare platforms. Two types of ERCs are implemented in these four commercial microarray platforms. The first type of ERC is added to the total RNA (tERC) before initiating the cDNA synthesis and IVT (in vitro transcription) steps of the RNA labeling procedure. The second type of ERC is added to the cRNA (cERC) just before the cRNA is placed into the hybridization mixture. Applied Biosystems and Affymetrix platforms use both types of ERCs in their respective protocols, whereas Agilent uses the tERC and GE Healthcare uses cERC in this study. RT, reverse transcription.


1133

AG1 site 2

Signal intensity

AG1 site 1

AG1 site 3

AG1_2_A3 AG1_2_D2 sample A sample B sample C sample D

1/1,000,000 1/100,000

1/10,000

1/1,000

1/100

1/1,000,000 1/100,000

tERC/poly-A molar ratio

1/1,000

1/100

1/1,000,000 1/100,000

AFX site 2

1/10,000

1/1,000

1/100


AFX site 3

Signal intensity

AFX site 1

1/10,000


sample A sample B sample C sample D

1/1,000,000

1/100,000

1/10,000

1/1,000


AFX site 1

1/1,000,000

1/100,000

1/10,000

1/1,000


1/1,000,000

1/100,000

1/10,000

1/1,000


AFX site 3

AFX site 2

Signal intensity

Figure 2 Concentration-response curves for ERCs on the Agilent, Affymetrix and GE Healthcare microarray platforms. Each concentrationresponse curve is generated from an individual microarray data set and represents the concentration of either the tERC (spiked poly-A molar ratio) or of cERC (spiked concentration in pM) on the x-axis as a function of normalized signal intensity on the y-axis. The amount of cERC added to the hybridization mixture is expressed in molar concentration based on the mass of the cERC transcript added to a specific volume of the hybridization mixture. The assumptions used to calculate the poly-A mass ratio for the different tERCs were that the average percentage of mRNA in total RNA is 2%, the average transcript length is 2,000 bases and the average molecular weight of a single base is 330 g/mol. The cERC concentration and tERC poly-A molar ratio used for this figure are summarized in Supplementary Table 1 online. The Agilent platform is presented in the first row where seven of the ten tERCs with the highest concentrations are plotted to better compare scales to the other platforms (the full concentration-response curve is presented in Supplementary Fig. 9 online). The Affymetrix platform is presented in the second and third rows and illustrates the combinatorial approach of using both tERCs (second row) and cERCs (third row). The GE Healthcare platform is presented in the fourth row illustrating the cERC concentration-response from the rat toxicogenomics study. This figure illustrates the different approaches each manufacturer employs for either tERC, cERC or both, when assessing assay quality using ERCs. Two microarrays from AG1 site 2 (AG1_2_D2 and AG1_2_A3) exhibit higher than expected signals for the tERCs with the lowest concentrations, indicating that these could be outlying assays. AA, aristolochic acid; RDL, riddelliine; CFY, comfrey. ‘L’ indicates samples isolated from livers and ‘K’ samples isolated from kidneys of treated rats. CTR, control (liver or kidney from untreated rats).

sample A sample B sample C sample D

cERC concentration (pM)



GEH Rat Signal intensity


A N A LY S I S

KAA

LCTR

LAA

LRDL KCTR LCFY platforms. The AG1 platform has very little compression with a slope close to 1 for tERCs. cERC concentration (pM) cERC concentration (pM) cERC concentration (pM) However, ERC data for AFX and GEH_RAT experiments appear compressed to a similar extent with slopes that are detectably 0 (shifted up) for Cy5(B)/Cy3(A) in all three sites and the y-intercept was 0.8). Two titration points (sample C and sample D) were plotted based on the amount of brain in the sample based on the volumetric mixing of samples A and B where C = 75%A + 25%B and D = 25%A + 75%B (Fig. 5). This plot is accurate if the percentage of mRNA is equal between sample A and sample B. However, the Agilent two-color tERC data indicate that the percentage of mRNA was higher in sample A compared to that in sample B (Fig. 4 and Supplementary Fig. 4 online). If we assume that sample A has 1.5-fold more mRNA as compared to sample B12, the percentage of brain RNA in sample C becomes 18% and for sample D becomes 67%. When these values are used in the x-axis of Supplementary Fig. 5 online, the correlation coefficients improve for all of the samples at all of the sites for three different microarray platforms, further supporting the hypothesis that the samples have different percentages of mRNA. The effect of the mRNA abundance differences between the four samples on cERC signal intensities was also investigated. Unlike tERC signal intensities, the cERC signal intensities across the four RNA samples for the ABI and AFX exhibited no significant difference (Supplementary Fig. 6 online), indicating that the cERCs added before hybridization are unaffected by the differences in the relative abundance of the sample mRNA tested in this set of experiments. The observation is also not affected by the choice of normalization (Supplementary Fig. 7 online). This result further supports the hypothesis that the differences between the biological samples occur at an earlier stage of target preparation. Additional analyses using external RNA controls For most assays identified as problematic, one or several ERCs behave differently from the others, which should be captured by an intensitybased unsupervised analysis, such as principal component analysis (PCA)17 or hierarchical cluster analysis (HCA)17. PCA based on tERC signal intensity identified AG1_1_D2, AG1_1_A1 and AG1_3_B3 as outliers, consistent with the PCA plot based on the entire microarray

1136

Sample_BB

AG1_1_B5

Observed log ratio


A N A LY S I S

Site 3

Expected log ratio

(Fig. 6a). Agilent’s Feature Extraction QC Report uses a different algorithm: the concentration-response curve fit to the linear portion is performed on a log-log plot after a parameterized sigmoidal curve fit of the data. The R2 correlations and slopes from the AG1 QC report are shown in Figure 6b. This type of sigmoidal curve fitting ignores the differences seen in the tERCs outside the linear range and results in identification of a different set of outlying assays than in the analysis shown in Fig. 3a, but with the same assays as identified in the PCA analysis (Fig. 6a). Results similar to those in Fig. 6a are also observed using HCA (Supplementary Fig. 8 online). These analyses, as well as approaches based on the concentration-response curve (Figs. 2 and 3) demonstrate the value of combining various ERC-specific approaches to enhance the capability of assay assessment. DISCUSSION A number of microarray manufacturers use ERCs to assess the technical performance of their gene expression assays. This study investigated the utility of ERCs, with emphasis on cERCs and tERCs, for assay assessment across five commercial microarray platforms using the MAQC data set10 and a rat toxicogenomic data set11. This study explores several different uses of ERCs for assay assessment. First, the observed ERC signal intensities were examined against the expected concentrations to visually detect potential outlying assays, which tend to deviate from the expected concentrationresponse curve trend. Second, the concentration-response curves were modeled for identification of potential outlying assays using output variables from linear regression analysis. These two approaches take advantage of the unique characteristic of ERCs spiked across a wide range of differing concentrations. However, for some platforms such as Applied Biosystems, ERCs are spiked in at a constant concentration, requiring analysis methods other than the concentration-response curve analysis. Thus, PCA and HCA were conducted based on the ERC signal intensity, and the ERC-identified outlying data sets are consistent with the analysis results based on the biological wholemicroarray data. These approaches are complimentary to each other and could be used in conjunction to enhance the discrimination of outlier identification.


A N A LY S I S Site 2

Site 3

Our key findings can be summarized as follows. The cERCs exhibit stable and consistent performance across both samples and sites. ABI tERC signal intensities increased and the biological probe signal intensity decreased in proportion to increasing amounts of brain RNA in the samples. When the tERC is added to total RNA samples, it is assumed that the tERC tranR = 0.9522 R = 0.8314 R = 0.8372 scripts are at different relative proportions to the pool of biological RNA transcripts. As the AGI abundance of mRNA is relatively higher in sample A as compared to the brain (sample B), the median signal of biological probes was found to be higher in sample A than in sample B, whereas the median tERC signal had the inverse relationR = 0.9350 R = 0.9788 R = 0.8325 ship. We further determined that different levels of compression in gene expression exist across commercial platforms, indicating that care must AFX be taken when conducting a cross-platform comparison with respect to making absolute fold-change assessments. And, finally, we also determined that quantile-based normalization Brain in the sample (%) approaches, such as those used in PLIER, RMA Figure 5 Illustration of the sample-dependent behavior of tERC signal across the MAQC samples. The and GCRMA for Affymetrix and for the Agilent ratio of the median tERC signal to the median biological signal is plotted against the percentages of One-Color and GE Healthcare platforms, reveal brain RNA in the different samples (0%, 25%, 75%, and 100% for A, C, D and B, respectively). In the variability of the concentration-response all nine groupings (three sites for each of three platforms), the slope was greater than zero with high slope estimates. This increase in variability may correlation coefficients, indicating that the tERC signal intensity is dependent on the abundance of mRNA or biological differences of the samples. Data used in creating this figure, along with the result from the differences in percentage mRNA statistical assessment, are summarized in Supplementary Table 4 online. between samples A and B. Although the mediannormalized signals of the tERCs and cERCs are relatively consistent, their relative ranks within ERCs added at different steps in the assay offer a quality control for samples A and B are different. Quantile normalization forces the distribudifferent steps of the assay process. cERCs are tolerant to differences tions of all data sets to be identical, moving the signals for the tERCs and in the mRNA abundance in the total RNA samples and provide the cERCs away from their original raw expression values. Because no single common standard set of external RNA controls advantage of being able to assess assay performance independent of the total RNA sample complexity (Fig. 2). A limitation of the cERCs is the using extended concentration range and a Latin square design are in inability to detect variability that may occur during target preparation. place for use across platforms in the microarray community, it is not Because tERCs are added into the assay process at a very early stage, they can reveal failures during sample collection, storage, labeling and amplification as well as hybridization, scanning and data collection. As a tERC probes Biological probes poor target quality is a common reason for aberrant assay results, there is 20 150 value in being able to use tERC to assess this independently, while using 15 100 cERCs to differentiate post-labeling sources of variation. Therefore, 10 50 these two types of ERCs are most valuable when used in combination. 5 0 0 This utility was demonstrated through the analysis of the AFX site 1 –50 –5 AG1_2_D2 data. The combination of tERC and cERC information assisted in the –100 –10 determination of sample amplification and labeling yields that differed AG1_1_A1 –150 –15 AG1_3_B3 AG1_1_A1 –200 from other sites and underlies the spread in the variability data. –20 AG1_2_D2 R 2 = 0.9789

R 2 = 0.8198

2

2

2

2

2

2

쐽 Sample A 쐽 Sample B 쐽 Sample C 쐽 Sample D

PC2

Ratio of median tERC signal divided by median biological signal

–25 –30 –20 –10

AG1_3_B3

0

10

20

30

40

50

60

–250 –200

70

–100

0

PC1

Figure 6 Alternative analysis using ERCs. (a) The Principal Component Analysis (PCA) of the Agilent tERC signal intensity is compared with the Agilent biological signal intensities. The graphs are colored by sample and shaped by site (site 1, triangle; site 2, square; site 3, circle). The same three assays (AG1_1_A1, AG1_2_D2, and AG1_3_B3) are potential outliers based on their shift in both the tERC and the biological signal. (b) Similar to Figure 3a, except that the parameterized sigmoidal curve–fitted linear regression data from the Agilent QC Report concentration-response curves was used to compare R2 correlation data (y-axis) and slope data (y-axis). The same three outlying assays identified in the PCA are shown as potential outliers in this analysis (circled in red) demonstrating identification of outlier agreement between two fairly different analyses.


100

200

300

PC1

b 1

1

1

0.995

0.995

0.995

0.99

0.99

0.985

0.985

R2


Site 1

R 2 = 0.9170

0.99 AG1_2_D2

0.985

0.98

0.98

0.98

0.975

0.975

0.975

AG1_1_A1

0.97 0.9

1

1.1

1.2

1.3

0.97 0.9

1

1.1

1.2

1.3

0.97 0.9

Sample A Sample B Sample C Sample D

1

AG1_3_B3

1.1

1.2

1.3

AG1 estimate of slope

1137

A N A LY S I S


Box 1 Recommendations for the implementation of external RNA controls • One key benefit of external RNA controls (ERCs) is the ability to get a qualitative assessment of assay performance. This benefit will be more fully realized when an extensive set of ERCs is available. • A comprehensive study is needed for modeling concentrationresponse behavior based on large data sets to determine the tolerance ranges for linear fit, slope and y-intercept for assay assessment, specifically in the context of false positives and false negatives. • The development of ERC-specific analysis approaches is encouraged. • ERCs that are added at both the total RNA level and cRNA level are valuable as they enable failure analysis for different steps of the assay. Using both types of ERCs in the same assay is beneficial for monitoring quality at multiple steps in the process.

yet possible to run the ideal set of external controls for a study of this nature1. Thus, the intent of this study was to identify key attributes of ERC performance that should be considered for designing better ERCs and associated analysis approaches in the future, which is one of the many important ERCC endeavors1. Based on the findings of this study, several points of consideration are summarized in Box 1. METHODS MAQC and TGx data sets. There are two types of data sets considered in this study; both are generated from the MAQC project. The difference between these two data sets is the nature of RNA samples used for generating the gene expression data. The MAQC data set used two calibrated RNA samples (A-Stratagene Universal Reference RNA and B-Ambion Brain reference RNA) and their two mixtures (C- 75%A/25%B and D-25%A/75%B). Applied Biosystems data (ABI), Affymetrix GeneChip data (AFX), and Agilent’s One-Color platform data (AG1) were generated using these four RNA samples. Each platform comprises a total of 60 microarrays, five technical replicates for each of four samples (A, B, C and D) for one test site (20 microarrays) and data from three test sites were used. In addition, Agilent Two-Color platform data (AGL) were also generated, but using only samples A and B. For AGL, four sets of assays were conducted with five replicates for each set, two dye swap experiments using brain-Cy5/UHRR-Cy3 (sample BA) and UHRR-Cy5/brain-Cy3 (sample AB) along two types of self-self hybridizations with brain-Cy5/brain-Cy3 (sample BB) and UHRR-Cy5/UHRR-Cy3 (sample AA), resulting in a total of 20 assays. The toxicogenomics (TGx) data set applied the RNA samples from rat livers in a TGx study. The detailed experimental protocol is described elsewhere11. Briefly, six-week-old Big Blue rats were treated with three compounds for 12 weeks and then killed. The compounds were aristolochic acid, a potent nephrotoxin and carcinogen that is present in plants used in herbal medicines, riddelliine, a carcinogenic pyrrolizidine alkaloid that contaminates various plants, and comfrey, a plant consumed by humans that is a rat liver carcinogen. RNA samples were isolated from livers of the rats treated with three compounds along with a liver control. In addition, RNA samples were also isolated from kidneys associated with treatment of aristolochic acid and a kidney control. Thus, there were a total of six types of rat RNA samples (four from liver and two from kidney). Six biological replicates (rats) were generated for each type of six RNA samples. The gene expression data were generated from four microarray platforms, Applied Biosystems (ABI_Rat), Affymetrix GeneChip (AFX_Rat), Agilent OneColor microarray (AG1_Rat), and GE Healthcare CodeLink (GEH_Rat). For each platform, 36 microarrays were generated, six for each of six groups. Applied Biosystems external RNA controls. These controls contains a suite of controls (>1,592 control probes) that can be used to check the quality of many aspects of an expression profiling experiment. These controls include

1138

the following: blank features, control ladders, hybridization controls, in vitro transcription (IVT) labeling controls, reverse transcription labeling controls, negative controls, spatial calibration controls and manufacturing quality controls. Among these controls, we used only the IVT and reverse transcription labeling controls and the hybridization controls, which are spiked at a single fixed concentration. For the hybridization controls, three unlabeled probes are spotted on the microarray: HYB_Control_1_Cp (60 replicates), HYB_Control_2_Cp (60 replicates) and HYB_Control_3_Cp (115 replicates). The hybridization cERCs consist of three digoxigenin-labeled 60-mer oligo control targets supplied with the chemiluminescence detection kit HYB_Control_1_Ct, HYB_Control_2_Ct and HYB_Control_3_Ct. The digoxigenin-labeled oligo targets (cDNA or cRNA) are added to the hybridization mixture. Presence of signal indicates hybridization occurrence and signal strength indicates hybridization stringency. IVT controls consist of three synthetic double-stranded cDNA with a T7 promoter and bacterial control gene sequences: bioB, 1,000-nt ds-cDNA; bioC, 750-nt dscDNA; bioD, 600-nt ds-cDNA. Five probes were used for each of three bacterial control genes, bioB, bioC and bioD targeting different regions of the control genes. This resulted in 15 probes and each probe is spotted eight times. Reverse transcription controls consist of three synthetic mRNAs with bacterial control gene sequences: lys, 1000-nt mRNA with poly(A) tail; phe, 1,400-nt mRNA with poly(A) tail; and dap, 1,900-nt mRNA with poly(A) tail. The synthetic mRNAs are added to the reverse transcription reaction with the RNA sample when using the reverse transcription labeling kit or the RT-IVT labeling kit. There are five control probes for each reverse transcription control gene targeting different regions on the gene, and each probe is spotted eight times with a total of 120 reverse transcription control probes. More detail on these controls can be found in http://docs.appliedbiosystems.com/pebiodocs/00113259.pdf and http://docs. appliedbiosystems.com/pebiodocs/04338853.pdf. Affymetrix external RNA controls. ERCs on GeneChip eukaryotic microarrays include poly-A controls (lys, phe, thr and dap) and hybridization controls (bioB, bioC, bioD and cre). Poly-A controls are Bacillus subtilis genes that are modified by the addition of poly-A tails, and then cloned into pBluescript vectors. The GeneChip Poly-A RNA Control Kit (P/N 900433) contains a presynthesized mixture of lys, phe, thr and dap. These poly A–tailed sense RNA samples can be spiked into isolated RNA samples as controls for the labeling and hybridization processes. Hybridization controls consists of bioB, bioC, bioD and cre. BioB, bioC and bioD represent genes in the biotin synthesis pathway of Escherichia coli; Cre is the recombinase gene from P1 bacteriophage. The GeneChip Eukaryotic Hybridization Control Kit (P/N 900299 and 900362) contains a mixture of biotin-labeled cRNA transcripts of bioB, bioC, bioD and cre. They can be spiked into the hybridization mixture, independent of RNA sample preparation, and used to evaluate sample hybridization efficiency. More detail can be found in GeneChip Expression Analysis Technical Manual (http://www.affymetrix.com/support/ technical/manual/expression_manual.affx) and GeneChip Expression Analysis Data Analysis Fundamentals (http://www.affymetrix.com/support/downloads/ manuals/data_analysis_fundamentals_manual.pdf). Agilent external RNA controls. The Agilent One-Color ERC Kit contains a mixture of ten in vitro synthesized, polyadenylated transcripts derived from the Adenovirus E1A gene. These transcripts are premixed at concentrations that span six logs and differ by one log or half-log increments (Supplementary Table 1 online). The ERC mixture is added to the total RNA, amplified and labeled with Cy3-dye. When the ERCs are used in processing Agilent OneColor microarray assays, the Agilent Feature Extraction (version 8.5) QC Report contains a number of tables and graphs providing information on system performance. These include an indication of the linear portion of the dynamic range of the microarray experiment, the high and low detection limits of the experiment and the reproducibility of the controls with coefficient of variation (CV) percentage calculations across the replicate probes for each of the ten ERCs. For more details, see http://www.chem.agilent.com/scripts/ literaturePDF.asp?iWHID=42629. The Agilent Two-Color ERC Kit contains the same ten tERC transcripts as used in the Agilent One-Color platform. Each transcript is premixed into two different ERC mixtures at known concentrations such that the ten transcripts are present in mass equivalents extending across 2.3 logs of concentration and represent ratios spanning from 1:10 to 10:1 (Supplementary Table 3 online). These two mixtures are spiked into


A N A LY S I S


either the Cy3 or Cy5 labeling reactions and colabeled with the total RNA. The Agilent Feature Extraction (version 8.5) QC Report contains a number of tables and graphs providing information on system performance. These include a measure of the expected versus observed log ratios that provide an indication of system accuracy, as well as a determination of the reproducibility of the controls with CV percentage calculations across the replicate probes for each of the ten ERCs. For more details, see http://www.chem.agilent.com/scripts/ literaturePDF.asp?iWHID=40485. GE Healthcare external RNA controls. Each CodeLink Whole Genome bioarray, from GE Healthcare, contains a set of positive-control probes designed against six E. coli genes. For each of the six bacterial genes there are five unique probe sequences represented in an 8× redundancy per rat bioarray. Therefore, there are a total of 240 positive-control probes within each bioarray, which are used to assess microarray quality by reporting dynamic range and sensitivity. Each of the six bacterial transcripts is supplied individually as poly-A(+) mRNA, ranging in size from 1,000 to 1,300 ribonucleotides. These control RNAs can be spiked at different concentrations into the total RNA starting material or labeled individually with biotin and spiked into the cRNA before hybridization. The cRNA spiking method, as used in this study, is the manufacturer’s recommendation for independently measuring bioarray quality because effects due to sample integrity and purity are circumvented. The positive-control poly-A(+) mRNAs supplied with the CodeLink Expression Assay Reagent Kit are araB, entF, fixB, hisB, gnd and leuB. These transcripts are reverse transcribed and amplified individually, incorporating biotin, and arranged in a dilution series from 50 fM to 50 pM, in fourfold concentration increments. The final hybridization concentrations of biotinylated spikes in the hybridization solution are araB (51.2pM), entF (12.8pM), fixB (3.2pM), hisB 0.80pM, gnd (0.20fM) and leuB (50.0fM). For more details, see http://www4. amershambiosciences.com/APTRIX/upp00919.nsf/Content/WD%3AExternal+ RNA+co%28274354027-B500%29?OpenDocument&hometitle=WebDocs. Microarray data preprocessing and normalization. Data preprocessing and normalization were performed in ArrayTrack, an FDA microarray data management, analysis and interpretation software18,19. For Affymetrix GeneChip, five different sets of normalized data were used, PLIER, MAS5, dChip, RMA and GCRMA. Present and Absent Calls were generated for each probe set. For the Agilent One-Color microarray, the raw data (gProcessedSignal data), Median Scaling data and Quantile normalized data were used. Negative values and ERCs were not included in the normalization. For the Two-Color microarray, only the dye-normalized Log Ratio data was used, without any further normalization. For the Applied Biosystems Microarray, signal intensity is associated with two measurements, signal/noise ratio and detection call (or flag). The spots having a ratio >3 and flag –1.

across known sources of variability (dye and site). Interestingly, log2 ratios appeared to be slightly less reproducible than log2 intensities for Agilent and TeleChem, but more reproducible for CapitalBio. This result could be driven by a larger microarray-to-microarray variability for CapitalBio or the performance of a manual channel balancing while scanning two-color, but not one-color CapitalBio microarrays. The overall lower correlation values for TeleChem appear to be driven by a nonlinear dye bias (data not shown). The intersite, one-color results for the Agilent sites are presented elsewhere15 and reveal that the Agilent data are very consistent between sites. To determine if the one-color and two-color designs are revealing the same biology, we compared the reproducibility of the lists of genes identified as differentially expressed by each approach within each platform. Common gene lists were generated comparing the number of differentially expressed genes for one-color and two-color data within each platform (Table 2). Comparisons are given for combinations of two P values (P < 0.05 and P < 0.01) and three fold-change (FC) thresholds (FC > 1.5, FC > 2.0 and FC > 4.0), with differentially expressed genes identified using a one-sample t-test of the sample B to sample A (B/A) ratio

1142

data including five replicates for each site. Concordances of differentially expressed genes are consistently >80% for all three Agilent sites, regardless of the P-value or fold-change criteria used. Similarly, the CapitalBio concordances are consistently ~70%. The TeleChem concordances are less consistent across P values and fold changes and are generally lower than those for the CapitalBio and Agilent data, which is in agreement with the lower overall correlation values for this platform. Specificity and sensitivity In addition to evaluating the reproducibility of the data from the oneand two-color assays, we also considered the sensitivity and specificity. Specificity defines the ability of an assay to determine differences only when they truly exist (that is, the true-negative rate). Sensitivity is the power to detect true differences (that is, the true-positive rate). Both of these measures make a tacit assumption that the truth is divided, which in this case means the mRNA levels derived from a gene are either the same for samples A and B or they are different. The actual truth is that they are likely to be always different, but this difference is small enough relative to technical noise that a substantial fraction


A N A LY S I S Table 2 Common gene list results for one- versus two-color microarray data based on differentially expressed genes Test site

Fold change

Number of differentially expressed genes

Number of differentially expressed genes

P < 0.05 Agilent 1


Agilent 2

Agilent 3

CapitalBio

TeleChem

P < 0.01

One color

Two color

Common genesa

One color

Two color

Common genesa

FC > 1.5

13,043

12,709

11,053 (86%)

11,771

12,506

10,175 (84%)

FC > 2

9,701

8,812

7,767 (84%)

9,273

8,678

7,467 (83%)

FC > 4

3,998

3,494

3,055 (82%)

3,979

3,447

3,029 (82%)

FC > 1.5

13,308

12,345

10,992 (86%)

12,673

11,410

9,940 (83%)

FC > 2

9,792

8,686

7,712 (83%)

9,526

8,043

7,071 (80%)

FC > 4

4,077

3,623

3,104 (81%)

4,042

3,261

2,886 (79%)

FC > 1.5

12,968

12,545

11,192 (88%)

12,537

12,056

10,580 (86%)

FC > 2

9,363

8,720

7,721 (85%)

9,266

8,373

7,397 (84%)

FC > 4

3,728

3,596

3,058 (84%)

3,716

3,399

2,987 (84%)

FC > 1.5

7,344

6,336

5,129 (75%)

6,238

6,098

4,529 (73%)

FC > 2

5,383

4,154

3,426 (72%)

5,004

4,078

3,203 (71%)

FC > 4

2,207

1,599

1,283 (67%)

2,081

1,580

1,187 (65%)

FC > 1.5

2,883

3,306

1,491 (48%)

1,079

3,305

760 (35%)

FC > 2

2,220

1,133

659 (39%)

997

1,133

458 (43%)

FC > 4

645

178

148 (36%)

475

178

140 (43%)

Values are presented using two different statistical comparisons (P < 0.05 or P < 0.01) and three different fold-change (FC > 1.5, 2 or 4) criteria. aThe values in parentheses represent the percentage of common genes based on the number of common genes identified as differentially expressed in both one- and two-color approaches divided by the total number of differentially expressed genes from both approaches combined.

of mRNA levels can be considered to be the same. When the binary truth is known, the trade-off between sensitivity and specificity is typically portrayed using a receiver operator characteristic (ROC) plot. However, here the truth is unknown with respect to A versus B gene expression, as is the case with most gene expression profiling experiments. Therefore, relative specificity and sensitivity is compared in terms of distributions of statistical modeling results. By using a P-value criterion to declare genes differentially expressed, the specificity (false-positive rate) can be controlled at the desired level. The accuracy of this control depends, at least in part, on the standard t-test assumptions, which can be shown to be approximately valid for these data. Once specificity is bounded, the total number of differentially expressed genes can be compared as a measure of sensitivity. To more rigorously assess sensitivity in this fashion, we fit and compared results from three different gene-by-gene ANOVA models (see Methods for details): Model 1: log2(Intensity) = Mean + Sample + Site + Error Model 2: log2(Intensity) = Mean + Sample + Dye + Sample*Dye + Site +Microarray + Error Model 3: log2(Ratio) = Mean + Dye + Site + Error Model 1 is applied to one-color data, model 2 is applied for intensity data directly without forming ratios for two-color data and model 3 is applied to ratios for two-color data. Direct modeling of intensities in models 1 and 2 enables a straightforward comparison between results for the one- and two-color data. Furthermore, the results from models 2 and 3 are quite similar, and so model 2 provides a bridge between models 1 and 3 that can be used for comparisons with ratio results that are commonly computed with two-color data. Before discussing primary results from these models, it should be noted that there is an imbalance in the number of samples hybridized for the one-color and two-color designs, which improves the sensitivity of the two-color results. More specifically, for each of the Agilent and CapitalBio sites, there are ten one-color microarrays and ten two-color microarrays, hence, there are twice as many samples hybridized on the two-color microarrays; that is, the one-color results effectively have half as much data, as only one sample was hybridized to each microarray.


This degree of unbalance is even greater in the TeleChem platform for which 20 two-color and only 10 one-color hybridizations were processed, resulting in four times as much two-color data. Subsequent results should be interpreted with this in mind. The three models were fit to the preprocessed Agilent, CapitalBio and TeleChem data and several output summary statistics were collected for each gene. Volcano plots (Fig. 1) compare the estimated log2 fold-change (x-axes) against its statistical significance (y-axes). Large numbers of genes are identified as differentially expressed as a result of the analyses of data from all three platforms, as is expected when comparing a brain sample to a tissue pool sample. All of the volcano plots visually have a similar distribution and range for the statistical significance values (y-axes) within each platform, except for model 1 for the TeleChem data (Fig. 1g), which has a substantially smaller range, that (as noted above) may be due to differences in the total number of microarrays processed for each approach. For all three platforms there is a tendency for the one-color data to exhibit larger fold changes but smaller significance scores (that is, the volcano plots are shorter and wider for one-color as compared to two-color). Figure 2 provides a more detailed depiction of the results from models 1, 2 and 3. Estimated log2 fold changes are compared in a scatter plot matrix for one-color intensities (model 1), two-color intensities (model 2) and two-color ratios (model 3) for the Agilent, CapitalBio and TeleChem data. The estimated fold changes are very consistent, especially between the two two-color methods (far right column). The fold changes estimated from the one-color data tend to be larger than those estimated by either model for the two-color data, as indicated by the slopes shown in Figure 2. The scatter plots in Figure 3 display negative log10 P-value comparisons from Agilent, CapitalBio and TeleChem data. Larger negative log10 P values mean more significant results. Therefore, when the negative log10 P values from different methods are compared graphically on different axes and the majority of the data points lie above the 45° reference line, it suggests that the method depicted on the y-axis is more sensitive than that depicted on the x-axis (or vice versa if the majority of points lie below the reference line). The scatter plots for the Agilent data suggest

1143

A N A LY S I S b 8

4

0

–4

8

4

0

–4

–8


R = 0.98 S = 1.12 Two-color intensity fold change (log2)

One-color intensity fold change (log2)

8

c

R = 0.98 S = 1.12 One-color intensity fold change (log2)

a

–8

–8

–4

0

4

8

R = 1.00 S = 1.00

4

0

–4

–8

–8

Two-color intensity fold change (log2)

–4

0

4

–8

8

Two-color ratio fold change (log2) R = 0.95 S = 1.25

8



R = 0.95 S = 1.25 One-color intensity fold change (log2)

0

4

8

f

e

d

–4

Two-color ratio fold change (log2) R = 1.00 S = 1.00

4

0

–4

–8

–8

–4

0

4

8

–8


–4

0

4

8

g

8


8

4

4

0

0

–8

–4

0

4


8

8

R = 1.00 S = 1.00

4

0

–8

–8

–8

4

–4

–4

–4

0

i R = 0.74 S = 1.18



R = 0.73 S = 1.18

–4

Two-color ratio fold change (log2)

h 8

–8


–8

–4

0

4


8

–8

–4

0

4

8


Figure 2 Comparison of log2 fold-change estimate results from three different modeling approaches for the three different platforms. (a–c) Agilent. (d–f) CapitalBio. (g–i) TeleChem. Columns correspond to log2 fold-change comparisons of one-color intensity versus two-color intensity, one-color intensity versus two-color ratio and two-color intensity versus two-color ratio. Each gray point represents a feature on the microarray. The red lines are 45° reference lines and the contours represent density levels for the points. Statistics for correlation (R) and slope (S) are inset in each graph.

that the two-color intensity-based analysis (model 2) has more power (sensitivity) than both the one-color intensity-based (model 1) and twocolor ratio-based analyses (model 3). The one-color analysis appears to have slightly more power than the two-color ratio analysis in the lower portion of the significance range, whereas the two-color ratio has more power in the upper range. When Figure 1a,c is also considered, the one-color data tend to exhibit larger fold changes, which explains why more differentially expressed genes were observed for the one-color data (Table 2). Figure 3 (row 1) shows that although the power between these two methods is similar, the relationship between them is nonlinear. For the CapitalBio data in Figure 3, both two-color models produce very similar results, and both appear to have more power (sensitivity) than the one-color-intensity analyses. For the TeleChem data, the difference is even more striking. As detailed above, these observed differences may be due to the differences in the amount of data for each approach, as twice as much data were obtained from a two-color assay compared

1144

to a one-color assay. Because of this inequity in the data, the power comparisons shown here are not a completely fair assessment of the sensitivity of one- versus two-color procedures, although they do help to demonstrate the effectiveness of increasing sample sizes, without also increasing the number of microarrays used. For example, from Table 2, when identical thresholds for significance are used, in most instances two-color ratio data produce fewer differentially expressed genes than one-color data, which indicates that either one-color platforms are more sensitive in identifying differentially expressed genes or that the fold changes reported by the one-color platform are less compressed than the two-color fold changes. The data modeled here suggest that the latter result is more likely. For two-color experimental designs, specificity can also be addressed by analysis of self-self hybridizations. In experimental designs that include a dye-swap design such as this, systematic errors are reduced by inclusion of the dye-flip control. One can, therefore, assess the false-positive rate



A N A LY S I S from self-self designs if one half of the self comparisons have the polarity reversed before calculation of significance. This analysis was performed for one of the Agilent test sites, for both pairs of self-self experiments. In this analysis, four of the self-self hybridizations were combined with two randomly chosen microarrays chosen for polarity reversal. For the A sample, 98 of 41,000 genes were detected as significantly differentially expressed (that is, false positives, P < 0.01). For the B sample, 61 of 41,000 genes were detected as significantly differentially expressed (P < 0.01). To further address the question of which design (one-color or twocolor) provides greater sensitivity, we examined correlations of one-color and two-color data for one of the Agilent test sites without any filtering based on detection calls (see Supplementary Fig. 2 online). Fold-change values correlated well between the two approaches across the entire intensity range, indicating that the approaches have similar levels of sensitivity. Furthermore, when thresholds for differential expression were applied (P < 0.01 and FC > 1.5) there was a 69% overlap of the genes identified by both approaches. Each approach uniquely identified 13–18% of the total number of differentially expressed genes and only a very small subset of the genes were found to be anticorrelated (18 or 0.09%). Accuracy Whereas specificity and sensitivity refer to a divided version of the truth, a more direct assessment of the accuracy of the platforms can be obtained when the truth is quantitative. Again, the true quantitative differences between the mRNA levels of samples A and B for each gene are unknown, but a well accepted surrogate can be obtained from orthogonal quantitative technologies (e.g., TaqMan assays). As detailed above, when data from one of the Agilent test sites were analyzed, ~31% of the total number of differentially expressed genes detected by one approach was not also identified by the other. To discern if these discordant data points are false positives on one or another of the approaches, we compared both to results generated using TaqMan assays. Genes were selected for measurement in these samples by TaqMan assays as part of the main MAQC study15. Most of the genes assayed by TaqMan were randomly selected from a set of RefSeq genes that were common to four commercial microarray platforms (Affymetrix, Agilent, GE Healthcare and Illumina). More details on the selection of these genes can be found elsewhere16. Figure 4 illustrates the comparison of the one-color, two-color and TaqMan assay data, and is colored based on the significance (P < 0.01 and FC > 1.5) of the ratio between B and A samples for the three different platforms (one-color, two-color and TaqMan assays). Data shown represent either all probes with TaqMan mapped data (Fig. 4a, N = 906) or only probes that were mapped as persistently detected in Agilent one- and two-color experiments (filtered as described in Methods) and detected in at least three of four replicates for both samples in the TaqMan assay data (Fig. 4b, N = 519). The results show a good overall correlation between the TaqMan assay data and both the one-color and two-color data. The 18 probes that were anticorrelated between one- and two-color data were not in the subset of genes assessed with TaqMan assays in this study. However, for those genes identified as discordant between the Agilent one- and two-color data, some were verified with TaqMan assays for each platform. A slightly higher percentage of probes found to be significant for only the two-color design were verified with TaqMan assays (51 of 85 or 60% for one-color, versus 39 of 55 or 71% for two-color; Fig. 4a), thereby indicating that both approaches have similar levels of accuracy. DISCUSSION Every aspect of microarray experimentation, including RNA isolation and purification, labeling and amplification, microarray fabrication, hybridization, data acquisition, analysis and statistical methods has


seen major advancements in the last several years. With the variety of platform choices available that have benefited from these advancements, a natural question arises regarding the characteristics of data generated from one-color and two-color assays. Results presented here describe a comprehensive study comparing one-color to two-color assays within three different platforms and across multiple test sites for one of the platforms, using two distinct RNA samples. Differential expression data from a pair of total RNA samples (Stratagene Universal Human Reference total RNA and Ambion Human Brain Reference total RNA) were generated using both one-color and two-color assays on different microarray platforms (Agilent, CapitalBio and TeleChem) and used to evaluate the relative reproducibility, specificity, sensitivity and accuracy of the two approaches. One of the strengths of this analysis is that comparison of the onecolor and two-color assays is not dependent on interplatform analysis, thus avoiding many of the complications inherent to such a comparison (including probe sequence issues as well as target labeling and detection technology differences). In addition, the filtered gene lists used for the analysis presented here are consistent between the two different design schemes on each platform, but are different between the different platforms, further complicating interplatform comparisons. Overall, the results between one- and two-color assays compare well, which aligns with expectations generated by numerous independent successes of one- and two-color microarray applications. Here we provide a statistical validation of this expectation. Reproducibility between the one-color and two-color assays is quite similar for each platform as demonstrated by the consistency of Pearson correlation values. When ratios are generated from the two distinct RNA samples, the differentially expressed gene lists are highly consistent across one- and two-color data when using widely accepted P-value and fold-change thresholds for significance. Just as important, the stability of the differentially expressed gene lists is consistent within individual platforms. Correlation coefficients in Table 1 are higher for the Agilent data, leading to greater overall concordance, but for all the platforms the one-color data and two-color data are comparable when assessing concordance using differentially expressed gene lists. Three ANOVA models are defined to provide a statistical framework for comparison of relative intraplatform specificity and sensitivity. Model 1 applies to one-color log2 intensities and model 3 to two-color log2 ratios. Model 2 handles two-color log2 intensities, and serves as a bridge between models 1 and 3. The use of these models avoids the problem of arbitrarily defining ratios for the one-color data, and enables adjustment for all known sources of variability. In addition, model 2 is shown to have slightly more sensitivity over model 3 for the Agilent data. Modeling two-color intensities directly as in model 2 is not common practice, but offers several advantages, including the ability to study sample-dye interactions. Overall, the relative specificity and sensitivity of the three platforms as determined by the three models is very similar between one- and two-color assays within each platform (Figs. 1–3). The results suggest that the two-color assays have a slight advantage with regard to power (sensitivity) and the detection of small fold changes (Figs. 1 and 3), especially when considering an equal number of microarrays. The one-color data do appear to be less compressed than two-color data as indicated by the slopes shown in Figures 2 and 4, which should be considered when using filtering rules that apply directly to estimated fold changes. In addressing the accuracy of the one-color and two-color assays using data from the Agilent platform, the results also show a good overall correlation with the TaqMan assay data. In some cases the TaqMan assay data have better agreement with the one-color data and in others the TaqMan assay data have better agreement with the two-color data. In

1145

reproducibility of the biology across the two approaches by comparing the concordance of differentially expressed gene lists, performance was approximately equal (Table 2 and Fig. 4). Cumulatively, these results indicate that data generated from both one- and twocolor assays are approximately equivalent and provide similar levels of biological insight. It should be noted that these results may not apply to microarray platforms for which manufacturing variability is high (such as may occur with some suboptimal, in-house, robotically spotted arrays, with poor quality control). All microarrays used in this study were obtained and processed at approximately the same time. Although in all three platforms multiple manufacturing lots of microarrays were used, no effort was made to control which manufacturing lots were grouped together in the study. Hence, the magnitude of the variance of the one-color and two-color results may differ from those presented here, if the data were specifically generated and assessed as individual groups across multiple manufacturing lots. In essence the variability due to manufacturing lot has not been addressed in this study since the array populations for each

many cases the differential expression results were consistent in direction between the one-color and two-color assays, but failed to meet the applied fold-change or significance criteria. In those cases when genes are reported as significantly differentially expressed by TaqMan assays, but not by either the one-color or two-color microarray assays, the differences may be attributable to the fact that the technologies are targeting and measuring different regions of a particular gene and/or splice variant. Also, most of the genes reported as significantly differentially expressed in the TaqMan assay, but not in the microarray data, are below the detection level of the microarray assay (Fig. 4) and may be indicative of the higher sensitivity of the PCR-based method. Finally, the significance of the microarray and TaqMan assays is not directly comparable, as a different level of replication was undertaken for the TaqMan assay data16. In summary, by presenting the experimental design and performance advantages of both modes, researchers are now provided insight and guidance for properly selecting the best approach (oneor two-color) to meet their research needs. When assessing the

50

40

40

30

30

20

20

10

10

0

0 0

10

20

30

40

0

10

One-color intensity P value (–log10)

12

10 8 6 4 2

30

40

4

6

8

10

12

8 6 4

2

4

6

8

10

12

12 10 8 6 4

2

14 16

Two-color intensity P value (–log10)

18

16

8 6 4

0 12

4

2

4

6

8

10

12

14

18

10

0 10

6

0

R = 0.42 S = 0.17

12

2

8

8

Two-color ratio P value (–log10)

14

2

50

10

14



14

40

0 0

16

30

R = 0.97 S = 0.99

12

18

6

20


R = 0.41 S = 0.16

4

10

14

10

14

18

2

10


R = 0.76 S = 0.58


0

20

0

0 2

30

50

2

0

16

20


14

R = 0.69 S = 0.60

0

40


14 12

R = 0.94 S = 0.98

0

50



50

R = 0.78 S = 0.68


R = 0.81 S = 0.69



50



A N A LY S I S

14

R = 0.99 S = 0.99

12 10 8 6 4 2 0

0

2

4

6

8

10

12


14 16

18

0

2

4

6

8

10

12

14

16

18


Figure 3 Comparison of negative log10 P-value estimate results from three different modeling approaches for the three different platforms. (a–c) Agilent. (d–f) CapitalBio. (g–i) TeleChem. Columns correspond to negative log10 P-value estimates of one-color intensity versus two-color intensity, one-color intensity versus two-color ratio and two-color intensity versus two-color ratio. Each gray point represents a feature on the microarray. The red lines are 45° reference lines and the contours represent density levels for the points. Statistics for correlation (R) and slope (S) are inset in each graph.

1146


A N A LY S I S b

a R = 0.876 m = 0.733

R = 0.947 m = 0.913

5

5

0

0

–5

–5

R = 0.968 m = 0.842

R = 0.906 m = 0.761

–10

–10 –5

0

5

One-color B/A log2 ratio

One color 34

Two color

38

16

439 51

39 129

160

–10

10 10


–10

–5

0

5


R = 0.876 m = 0.807

One color

5

30 0

Two color

28

14

289 29

23 19

–5

87

10 10


Two-color B/A log2 ratio


10

10

R = 0.916 m = 0.924

5

0

–5

Taq

Taq

–10

–10 –10

–5

0

5

10

Taq B/A log2 ratio

–10

–5

0

5

10

Taq B/A log2 ratio

Figure 4 Comparison of Agilent one-color and two-color data with TaqMan assay data. The figure illustrates the comparison of the one-color, two-color and TaqMan assay data, and is colored based on the significance of the ratio between B and A samples for the three different sets of data as illustrated. Significance was based on a P < 0.01 and a 1.5 fold change. Data shown represent either of two possibilities. (a) All probes with TaqMan mapped data (N = 906). (b) Only probes that were mapped as persistently detected in Agilent one- and two-color experiments (filtering as described in Methods) and that were detected in at least three of four replicates for both samples in the TaqMan assay data (N = 519). The numbers in gray refer to the number of genes that are not detected as significantly differentially expressed (based on given FC and P-value criteria) by any of the three assays. Lines shown represent the orthogonal fit to the data with slope (m) and correlation (R) as shown in the inset.

platform were heterogeneous in terms of batch specificity. Ultimately the decision to use either a one-color or two-color approach will be determined by cost, experimental design considerations and personal preference. METHODS Hybridization. Three independent test sites were used for the Agilent platform and one test site each was used for the CapitalBio and TeleChem platforms (five total test sites). All test sites received the same lot numbers of two different total RNA samples (Stratagene Universal Human Reference total RNA (SUHRR, sample A) and Ambion Human Brain Reference total RNA (AHBRR, sample B). The hybridization-dye pairings and RNA descriptions were as follows: twocolor hybridization: a, SUHRR-Cy3 versus SUHRR-Cy5; b, AHBRR-Cy3 versus AHBRR-Cy5; c, SUHRR-Cy3 versus AHBRR-Cy5; d, AHBRR-Cy3 versus SUHRRCy5; one-color hybridization: e, SUHRR-Cy3; f, AHBRR-Cy3. The two-color self-self hybridizations (codes a and b) provide information about the reproducibility and specificity of the two-color hybridizations, but are not used for a majority of the analyses described in this paper because of space constraints and to more evenly balance the comparisons between the one- and two-color results within a platform. However, they are included in the available data set. For each of the Agilent and CapitalBio sites, 5 microarrays were used for each of the RNA codes c, d, e and f, for a total of 20 microarrays (10 two-color and 10 one-color) at each of these sites. For the TeleChem site, 10 microarrays were used for RNA codes c and d, and 5 microarrays for codes e and f, for a total of 30 microarrays (20 two-color and 10 one-color). Across all five sites, a total of 110 microarrays were hybridized (60 two-color and 50 one-color), which assayed a total of 170 samples.


RNA quantification and purity assessment. RNA samples were quantified using a NanoDrop ND-100 UV-VIS spectrophotometer. Each test site performed three replicate measurements for each sample using 1.5 µl and reported the values as average ± s.d. RNA intactness assessment. SUHRR and AHBRR (200 ng) were run on the Agilent Bioanalyzer 2100 in triplicate (all samples on one chip) by each test site. rRNA ratio (28S/18S) and RNA Integrity Numbers (RIN) are reported as average ± s.d. Acceptable values were defined as: A260/A280 ratio in the range of 1.8–2.2, rRNA ratio (28S/18S) > 0.9 and RIN value > 8.0. Labeling and hybridizations on the Agilent platform. Five hundred nanograms of total RNA was converted into labeled cRNA with nucleotides coupled to a fluorescent dye (either Cy3 or Cy5) using the Low RNA Input Fluorescent Linear Amplification Kit (version 4.0 protocol) (Agilent Technologies). The quality and quantity of the resulting labeled cRNA was assessed using a NanoDrop ND-1000 spectrophotometer (NanoDrop Technologies) and an Agilent 2100 Bioanalyzer. Individually labeled cRNAs were not pooled before hybridization. Equal amounts of Cy3 and Cy5-labeled cRNA (1.5 µg) from two different samples (for the twocolor protocol) or only from one Cy3-labeled cRNA (for the one-color protocol) were hybridized (see hybridization configurations above) to Agilent Human Whole Genome Oligo Microarrays (G4112A) for 17 h at 65 °C. The hybridized microarrays were then washed using manufacturers’ recommended conditions and scanned using an Agilent G2565BA scanner. Data were extracted from the scanned image using Agilent Technologies’ Feature Extraction software version 8.5 (FE8.5). All data columns present in the extracted data files are described in detail in the Agilent G2567AA FE8.5 Software Reference Guide (http://www. chem.agilent.com/scripts/LiteraturePDF.asp?iWHID=41954).

1147


A N A LY S I S Labeling and hybridizations on the CapitalBio platform. The human genomewide long oligonucleotide microarray was constructed in-house at CapitalBio Corporation. Briefly, 5′-amino-modified 70-mer probes representing 21,329 H. sapiens genes from the Human Genome Oligo Set Version 2.1 (Qiagen), and internal and external controls, were printed on amino silaned glass slides using a SmartArray microarrayer (CapitalBio Corp.). Fluorescent-labeled DNA (Cy3 and Cy5-dCTP) was produced through Eberwine’s linear RNA amplification method and subsequent enzymatic reaction. This procedure has been previously described in detail17. Briefly, double-stranded cDNA containing T7 RNA polymerase promoter sequence was synthesized with 5 µg of total RNA using Reverse Transcription System, RNase H, DNA polymerase I and T4 DNA polymerase, according to the manufacturer’s recommended protocol (Promega). The resulting labeled DNA (labeled control and test samples) was quantitatively adjusted based on the efficiency of Cy-dye incorporation and mixed into 80-µl hybridization solution (3× SSC, 0.2% SDS, 25% formamide and 5× Denhart’s). Individually labeled cRNAs were not pooled before hybridization. Hybridization on a microarray (see hybridization configurations above) was performed under LifterSlip (Erie Company). The hybridization chamber was laid on a three-phase Tiling Agitator (CapitalBio Corp.) to facilitate the microfluidic circulation under the coverslip. The microarray was hybridized at 42 °C overnight and washed with two consecutive washing solutions (0.2% SDS, 2× SSC at 42 °C for 5 min, and 0.2% SSC for 5 min at 22 °C) before scanning with a confocal LuxScan scanner (CapitalBio Corp.). For two-color microarrays, the scanning setting for the Cy3 and Cy5 channels was manually balanced by visual inspection of the external control spots. The data from the obtained images were extracted with SpotData software (CapitalBio Corp.). Labeling and hybridizations on the TeleChem H25K platform. Two micrograms of each sample was amplified using a Genisphere SenseAmp Plus Amplification kit (generating amplified poly A–tailed senseRNA), according to the manufacturer’s recommended protocol. The resulting tailed senseRNA was reverse transcribed with amino-allyl indirect labeling using a SuperScript Indirect cDNA Labeling Kit (Invitrogen) with slight modifications. Each first strand cDNA generation reaction used 5 µg of senseRNA with Superscript II and aa-dUTP at 42 °C for 2 h. cDNA was purified using a MinElute PCR Purification Kit and conjugated with mono-functional Cy3 or Cy5 dye aliquot (GE Healthcare) for 1 h at 22 °C in the dark. Dye-conjugated cDNA was purified with a MinElute PCR Purification Kit. Dye:base labeling efficiency was determined at this point for all dye-conjugated cDNA. Hybridization was done manually in TeleChem Hybridization cassettes using LifterSlip (Erie Company). cRNAs were labeled independently and not pooled before hybridization. In one-color experiments, Cy3-labeled cDNA samples were denatured independently and one sample applied to each microarray. For two-color experiments, Cy3- and Cy5-paired cDNA samples were combined and denatured before applying to individual microarrays (see hybridization configurations above). Hybridization mixes (55 µl total volume) consisted of 38.5 µl labeled cDNA, 5.5 µl 2% SDS, 7.0 µl 20× SSC, 3.0 µl poly dA (5 µg/µl) and 1.0 µl Cotl DNA (1 µg/µl). Hybridization cassettes and slides were pre-heated to 55 °C before samples were added and 3× SSC was added into humidity grooves in the cassette. Samples were applied to the microarrays and hybridized for 16 h at 55 °C in a water bath. After hybridization, slides were washed (10 min, 2× SSC/0.1% SDS at 42 °C; 10 min, 0.2× SSC/0.1% SDS at 42 °C; 10 min, 0.2× SSC at 22 °C twice) before centrifugation in 50-ml conical tubes at 201g for 5 min to dry. Scanning was performed on Axon 4200A or 4200B instruments at a PMT yielding 1% or less saturated spots. Agilent data preprocessing, normalization and filtering. For one-color experiments, gProcessedSignal values from Agilent’s Feature Extraction software were used as input into experimental analyses. This ProcessedSignal is generated after background subtraction and includes correction for multiplicative surface trends. Features were marked as Absent (A) when the processed signal intensity was less than twofold the value of the processed signal error value (these features were transformed by setting their processed intensity value to that of the processed signal error value). Features were marked as Marginal (M) when the measured intensity was at a saturated value or if there was a substantial amount of variation in the signal intensity within the pixels of a particular feature. Features not considered Absent or Marginal were marked Present (P).

1148

For the two-color microarrays, raw data signals were preprocessed in a similar fashion as those for one-color microarrays, but did not include a surface-trend correction and did include additional preprocessing to adjust for possible dye bias within a microarray. Data used in the two-color analyses was either the red and green ProcessedSignal or LogRatio values from Agilent’s Feature Extraction software. Dye normalization of two-color Agilent microarrays includes both linear scaling and Lowess normalization to a rank invariant set of microarray features. For some of the analyses (see Table 2, Fig. 4 and Supplementary Fig. 2), LogRatio values, which are calculated from the ProcessedSignals by Agilent’s Feature Extraction software, were used. When LogRatio was used for the twocolor data, the sign on LogRatio was changed for half of the RNA comparisons to accommodate the dye swap. Generation of a filtered feature list for Agilent one- and two-color data was conducted as follows: (i) Agilent flagging rules were applied, setting all absent and marginal features to missing. (ii) To derive a reliable common gene set across both one- and two-color data, features with fewer than 50% present genes across all microarrays were filtered. (iii) Features with fewer than five present calls from each sample group (A or B) across sites for one-color or fewer than five present calls across sites for two-color were also filtered. (iv) This filtering results in 19,802 genes in the final common gene set that was used for much of the statistical analysis presented, from a total of 41,000 non-control probes on the microarray. For the analysis presented in Figure 4 and Supplementary Fig. 2, all 41,000 noncontrol probes were included in the analysis. Further details on the data processing steps used to generate the Agilent oneand two-color output columns can be found in the Agilent G2567AA FE8.5 Software Reference Guide (http://www.chem.agilent.com/scripts/LiteraturePDF. asp?iWHID=41954). Data were median normalized for the statistical analyses in Figures 1–3 and Supplementary Figure 1 through JMP Genomics software (http://www.jmp. com/). For the remainder of the analyses, normalization of the Agilent one-color data was performed in GeneSpring GX as follows: (i) Values below 5.0 were set to 5.0. (ii) Each measurement was divided by the 50th percentile of all measurements in that sample. The percentile was calculated using only genes marked present. For analyses presented in Figure 4 and Supplementary Figure 2 only, specific samples were normalized to one another. All samples were normalized against the median of the control samples (A). Each measurement for each gene in those specific samples was divided by the median of that gene’s measurements in the corresponding control samples. CapitalBio data preprocessing, normalization and filtering. All one-color and two-color images were analyzed using SpotData software (CapitalBio Corp.) and raw data were provided in the form of tab-delimited text files for each microarray. A spot-exclusion method was adopted to filter faint spots18,19. The average log2 intensity of each gene across all replicates of both samples (A and B) was calculated and sorted. Genes with average intensity in the lowest 50% were excluded from further analysis. A subset of 11,735 genes from a total of 23,231 spots (including controls) remained for analysis. Local median and background subtraction was applied for one-color and twocolor intensity. For two-color data, an additional linear Lowess normalization was applied to the background-subtracted data. This was performed by scaling each channel to a median intensity of 100 and then normalized. For one-color data, each microarray was scaled to a median intensity of 1,000. TeleChem data preprocessing, normalization and filtering. All one-color and two-color images were analyzed using Axon GenePix Pro 5.0 software, and raw data were provided in the form of one tab-delimited text (.gpr) file per microarray. Features automatically marked as Absent (A) had a numerical value of –75 and corresponded to features in the Axon (.gal) file that show ID ‘empty’. Features marked Not Found (NF) had a numerical value of –50 and were defined as features with less than 6 pixels, or the feature diameter was greater than the lesser of three nominal diameters set in Block Properties of the (.gal) file, or the diameter that would cause it to overlap an adjacent feature of nominal diameter, or the feature was found at a position that would overlap an adjacent feature. Features marked Bad (B) had a numerical value of –100 and were defined by visual inspection during spot finding as having major noise associated with either the spot or background signal. All probes with a value less than 0 on at least one microarray were removed across all microarrays. Features marked Present (P) had



A N A LY S I S a numerical value of 0 and were considered acceptable for further analysis. The common filtered genes between one- and two-color microarrays were retained. The subset data was based on the list of 12,453 genes from a total of 27,648 spots (including controls) on the microarray. Analysis was based on intensity values: (F532_Median)-B532 intensity was used for one- and two-color data, in addition to (F635_Median)-B635 for two-color data. Lowess normalization of intensities was applied within individual two-color microarrays and median normalization across the microarrays. The aforementioned preprocessing and normalization methods for all three platforms followed manufacturers’ recommendations to reflect what will most likely occur in common practice. The methods differ somewhat across platforms but are consistent within platforms in order to make intraplatform one-and twocolor comparisons fair. The primary difference in normalization techniques between the three platforms is found in the TeleChem two-color data. For all the other data analyses, median scaling was applied to the data before Lowess normalization. However, with the TeleChem two-color data, this process was reversed. To compare all the data using the same normalization work flow, we applied median scaling to the TeleChem two-color data before Lowess normalization and compared to the original normalization process (Lowess before median scaling). This comparison is shown in Supplementary Figure 3 online. These additional data confirm that the minor differences in normalization procedure have very little impact on the data. Outlier assays. For the Agilent data set, microarrays identified as outlier microarrays based on single microarray quality metrics (AG1_1_A1, AG1_2_A3, AG1_ 3_B3, AGL_1_B5, AGL_1_D1, AGL_2_A1, AGL_2_C4) were not removed from analysis for the majority of the analysis presented here. The analysis presented in Figure 4 and Supplementary Figure 2 did exclude outlier microarrays. Generation of common differentially expressed gene lists. Data used for the generation of the common differentially expressed gene lists (Table 2) were from the genes that passed data preprocessing and filtering criteria for each platform and included 19,802 genes for Agilent, 11,735 genes for CapitalBio and 12,453 genes for TeleChem. Data normalization for the Agilent data was performed as described above. For both CapitalBio and TeleChem, ArrayTrack20 median scaling was used for one-color data and Linear & Lowess for two-color data (default median target intensity = 1,000). Significant differentially expressed genes were identified with a one sample t-test of log2 (B/A) ratio of five replicates that differ from 0. For two-color data, the dye swap result was averaged before doing the t-test. For both one-color and two-color data, all combinations of P-values of 0.05 and 0.01 and fold-changes of 1.5, 2.0 and 4.0 were calculated to determine the percentage of common differentially expressed genes. The percentage of common genes was calculated by dividing the number of common genes identified as differentially expressed in both one- and two-color approaches by the total number of differentially expressed genes from both approaches combined. The common manufacturer ID was used to identify the common genes from the gene lists. ANOVA models. Several analyses are based on fitting three different models to the preprocessed and normalized data: Model 1: log2(Intensity) = Mean + Sample + Site + Error Model 2: log2(Intensity) = Mean + Sample + Dye + Sample*Dye + Site +Microarray + Error Model 3: log2(Ratio) = Mean + Dye + Site + Error Separate models are fitted to the data from each feature within each platform. Model 1 is used for the one-color data and models 2 and 3 are used for the twocolor data. In these models, Intensity refers to a particular intensity value for one gene; Ratio refers to a particular ratio value for one gene; Mean indicates an overall mean value, which corresponds to mean log2(Intensity) for models 1 and 2, and mean log2(Ratio) for model 3; Sample indicates whether the intensity measurement is from sample A or B (this term is not needed in model 3 because ratios between A and B are being modeled); Site indicates the site (included for Agilent data only because CapitalBio and TeleChem data only had one site); Dye indicates the dye effect in model 2 and the dye-swap configuration in model 3; Sample*Dye refers to an interaction effect between samples and dyes; Microarray indicates the microarray from which the data were measured; Error indicates


random error, which is assumed to be normally distributed with mean zero and variance specific for each gene. Along with the Error term, the Site and Microarray effects are also assumed to be normally distributed with mean zero and constant variance. This arises from an assumption that effects of Site and Microarray can be assumed to be drawn from a normal population. They are so-called random effects, and estimates of their variances are known as variance components. All other effects are assumed to be fixed, that is, they have a finite number of levels, the mean value of which is estimated during the model fitting process. Model 2 is obviously the most complex of the three models but is easily fitted to two-color data using standard mixed models software. The random Microarray effect is critical, as it models the correlation between pairs of intensities observed on the same microarray. This model enables a more refined analysis of two-color results than that from model 3 by including estimates of overall mean intensity and the Sample*Dye interaction. Model 2 and its variants have been used successfully for the past five years in a variety of microarray applications21–23. For each feature on each platform, an estimate of the log2 fold change between samples A and B is computed in models 1 and 2 as the difference between the two levels of the estimated Sample effect. The ANOVA model output also includes a standard error and degrees of freedom for this difference, from which a –log10 P value is computed using a t-distribution. For model 3, the estimate of the Mean effect represents the estimated log2-scaled fold change (B/A) because the ratios were computed by dividing the B-intensity by the A-intensity, and a –log10 P value is computed in the same way as in models 1 and 2. Statistical results for all three models are based on mixed model theory21–24. Comparison of Agilent one-color and two-color data with TaqMan assay data. One-color data were normalized in Agilent GeneSpring GX as described above including the normalization of specific samples to each other (Fig. 4). Two-color data were analyzed using the following scheme. The processed signal data from the Agilent Feature Extraction software were loaded into Agilent’s GeneSpring GX software. To account for dye swap, we reversed the signal channel and control channel measurements for all d microarrays. Each gene’s measured intensity was divided by its control channel value in each sample. TaqMan assay data were generated as part of the MAQC study as described elsewhere16. TaqMan assay data were imported into Agilent’s GeneSpring GX from the data file provided by the MAQC after splitting it into individual files for each sample. For the TaqMan assay comparisons, the mapping from the final 12,091 genes was used for cross comparison between the Agilent probes and TaqMan assays15. The processed (‘intensity like’) TaqMan assay data were imported into GeneSpring GX based on the mapping, and ratios were calculated as follows: each measurement for each gene in those specific samples was divided by the median of that gene’s measurements in the corresponding control (A, SUHRR) samples. P values were calculated for the Agilent and TaqMan assay data using a onesample t-test with the appropriate number of replicates (four or five for the microarray assays, depending on the comparison, and four for the TaqMan assays), with the mean intensity value (as calculated above) compared to 1. Note: Supplementary information is available on the Nature Biotechnology website. ACKNOWLEDGMENTS The authors thank the MicroArray Quality Control (MAQC) consortium for generating the large data sets used in this study. E.K.L. and P.H. acknowledge the Advanced Technology Program of the National Institute of Standards and Technology, whose generous support provided partial funding of this research (70NANB2H3009). DISCLAIMER This work includes contributions from, and was reviewed by, the FDA and the NIH. This work has been approved for publication by these agencies, but it does not necessarily reflect official agency policy. Certain commercial materials and equipment are identified in order to adequately specify experimental procedures. In no case does such identification imply recommendation or endorsement by the FDA or the NIH, nor does it imply that the items identified are necessarily the best available for the purpose.

1149

A N A LY S I S COMPETING INTERESTS STATEMENT The authors declare competing financial interests (see the Nature Biotechnology website for details).


Published online at http://www.nature.com/naturebiotechnology/ Reprints and permissions information is available online at http://npg.nature.com/ reprintsandpermissions/ 1. Fodor, S.P. et al. Light-directed, spatially addressable parallel chemical synthesis. Science 251, 767–773 (1991). 2. Fodor, S.P. et al. Multiplexed biochemical assays with biological chips. Nature 364, 555–556 (1993). 3. Schena, M. et al. Quantitative monitoring of gene expression patterns with a complementary DNA microarray. Science 270, 467–470 (1995). 4. Churchill, G.A. Fundamentals of experimental design for cDNA microarrays. Nat. Genet. 32, Suppl. 490–494 (2002). 5. Li, J., Pankratz, M. & Johnson, J. Differential gene expression patterns revealed by oligonucleotide versus long cDNA arrays. Toxicol. Sci. 69, 383–390 (2002). 6. Tan, P. et al. Evaluation of gene expression measurements from commercial platforms. Nucleic Acids Res. 31, 5676–5684 (2003). 7. Dobbin, K.K. et al. Interlaboratory comparability study of cancer gene expression analysis using oligonucleotide microarrays. Clin. Cancer Res. 11, 565–572 (2005). 8. Irizarry, R.A. et al. Multiple-laboratory comparison of microarray platforms. Nat. Methods 2, 345–349 (2005). 9. Larkin, J.E., Frank, B.C., Gavras, H., Sultana, R. & Quackenbush, J. Independence and reproducibility across microarray platforms. Nat. Methods 2, 337–343 (2005). 10. Kuo, W.P. et al. A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies. Nat. Biotechnol. 24, 832–840 (2006). 11. Järvinen, A-K. et al. Are data from different gene expression microarray platforms comparable? Genomics 83, 1164–1168 (2004).

1150

12. de Reynies, A. et al. Comparison of the latest commercial short and long oligonucleotide microarray technologies. BMC Genomics 7, 51 (2006). 13. Wang, Y. et al. Large scale real-time PCR validation on gene expression measurements from two commercial long-oligonucleotide microarrays. BMC Genomics 7, 59 (2006). 14. Bammler, T. et al. Standardizing global gene expression analysis between laboratories and across platforms. Nat. Methods 2, 351–356 (2005). 15. MAQC Consortium. The MicroArray Quality Control (MAQC) project shows inter- and intramolecular reproducibility of gene expression measurements. Nat. Biotechnol. 24, 1151–1161 (2006). 16. Canales, R.D. et al. Evaluation of DNA microarray results with quantitative gene expression platforms. Nat. Biotechnol. 24, 1115–1122 (2006). 17. Guo, Y. et al. Genomic analysis of anti-hepatitis B virus (HBV) activity by small interfering RNA and lamivudine in stable HBV-producing cells. J. Virol. 79, 14392–14403 (2005). 18. Barczak, A. Spotted long oligonucleotide arrays for human gene expression analysis. Genome Res. 13, 1175–1785 (2003). 19. Shi, L. et al. Cross-platform comparability of microarray technology: intra-platform consistency and appropriate data analysis procedures are essential. BMC Bioinformatics 6 Suppl. 2, S12 (2005). 20. Tong, W. et al. Development of public toxicogenomics software for microarray data management and analysis. Mutat. Res. 549, 241–253 (2004). 21. Wolfinger, R.D. et al. Assessing gene significance from cDNA microarray data via mixed models. J. Comput. Biol. 8, 625–637 (2001). 22. Jin, W., Riley, R., Wolfinger, R.D., White, K.P, Passador-Gurgel, G. & Gibson G. Contributions of sex, genotype and age to transcriptional variance in Drosophila melanogaster. Nat. Genet. 29, 389–395 (2001). 23. Chu, T-M., Deng, S., Wolfinger, R.D., Paules, R.S. & Hamadeh, H.K. Cross-site comparison of gene expression data reveals high similarity. Environ. Health Perspect. 112, 449–455 (2004). 24. Chu, T-M., Deng, S. & Wolfinger, R.D. Modeling Affymetrix data at the probe level. in DNA microarray and statistical genomics techniques: Design, analysis, and interpretation of experiment. (eds. Edwards, J.W., Beasley, T.M., Page, G.P. and Allison, D.B.) 197–222 (Chapman & Hall/CRC, Taylor & Francis Group, Boca Raton, FL, 2006).



ARTICLES

The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements MAQC Consortium* Over the last decade, the introduction of microarray technology has had a profound impact on gene expression research. The publication of studies with dissimilar or altogether contradictory results, obtained using different microarray platforms to analyze identical RNA samples, has raised concerns about the reliability of this technology. The MicroArray Quality Control (MAQC) project was initiated to address these concerns, as well as other performance and data analysis issues. Expression data on four titration pools from two distinct reference RNA samples were generated at multiple test sites using a variety of microarray-based and alternative technology platforms. Here we describe the experimental design and probe mapping efforts behind the MAQC project. We show intraplatform consistency across test sites as well as a high level of interplatform concordance in terms of genes identified as differentially expressed. This study provides a resource that represents an important first step toward establishing a framework for the use of microarrays in clinical and regulatory settings.

Recently, pharmacogenomics and toxicogenomics have been identified both by the US Food and Drug Administration (FDA) and the US Environmental Protection Agency (EPA) as key opportunities in advancing personalized medicine1,2 and environmental risk assessment3. These agencies have issued guidance documents to encourage scientific progress and to facilitate the use of these data in drug development, medical diagnostics and risk assessment (http://www. fda.gov/oc/initiatives/criticalpath/; http://www.fda.gov/cder/guidance/ 6400fnl.pdf; http://www.fda.gov/cdrh/oivd/guidance/1549.pdf; http:// www.epa.gov/osa/genomics.htm). However, although DNA microarrays represent one of the core technologies for this purpose, concerns have been raised regarding the reliability and consistency, and hence potential application of microarray technology in the clinical and regulatory settings. For example, a widely cited study reported little overlap among lists of differentially expressed genes derived from three commercial microarray platforms when the same set of RNA samples was analyzed4. Similar low levels of overlap have been reported in other interplatform and/or cross-laboratory microarray studies5–8. Although similar results continue to appear in peer-reviewed journals9,10, raising doubts about the repeatability, reproducibility and comparability of microarray technology11–13, several studies have also been recently published showing increased reproducibility of microarray data generated at different test sites and/or using different platforms14–18. It follows that before this technology can be applied in clinical practice and regulatory decision making, microarray standards, quality measures and consensus on data analysis methods need to be developed2,19–21.

Here we describe the MAQC project, a community-wide effort initiated and led by FDA scientists involving 137 participants from 51 organizations. In this project, gene expression levels were measured from two high-quality, distinct RNA samples in four titration pools on seven microarray platforms in addition to three alternative expression methodologies. Each microarray platform was deployed at three independent test sites and five replicates were assayed at each site. This experimental design and the resulting data set provide a unique opportunity to assess the repeatability of gene expression microarray data within a specific site, the reproducibility across multiple sites and the comparability across multiple platforms. Objective assessment of these technical metrics is an important step towards understanding the appropriate use of microarray technology in clinical and regulatory settings. This study also addresses many other needs of the scientific community pertaining to the use and analysis of microarray data (see MAQC goals in Supplementary Data online). The MAQC project has generated a rich data set that, when appropriately analyzed, reveals promising results regarding the consistency of microarray data between laboratories and across platforms. In this article, we detail the study design, describe its implementation and summarize the key findings of the MAQC main study. The accompanying set of articles22–26 provides additional analyses and related data sets. Although the sample types used in this study are not directly representative of a relevant biological study, the study provides technical insights into the capabilities and limitations of microarray technology. Similar levels of concordance in cross-laboratory and interplatform comparisons have been independently reported using a toxicogenomics study26.

*A list of authors and their affiliations appears at the end of the paper. Correspondence and requests for materials should be addressed to L.S. ([email protected]). Received 6 June; accepted 31 July; published online 8 September 2006; doi:10.1038/nbt1239

NATURE BIOTECHNOLOGY

VOLUME 24

NUMBER 9

SEPTEMBER 2006

1151


ARTICLES RESULTS Experimental design The MAQC project (http://www.fda.gov/nctr/science/centers/ toxicoinformatics/maqc/) repeatedly assayed four pools comprised of two RNA sample types on a variety of gene expression platforms and at multiple test sites. The two RNA sample types used were a Universal Human Reference RNA (UHRR) from Stratagene and a Human Brain Reference RNA (HBRR) from Ambion. The four pools included the two reference RNA samples as well as two mixtures of the original samples: Sample A, 100% UHRR; Sample B, 100% HBRR; Sample C, 75% UHRR:25% HBRR; and Sample D, 25% UHRR:75% HBRR. This combination of biologically different RNA sources and known titration differences provides a method for assessing the relative accuracy of each platform based on the differentially expressed genes detected. A unique feature of the MAQC project is that both sample type A and sample type B are commercially available to the community for a few years to come in the exact batches as those used by the MAQC project. Six commercially available microarray platforms were tested: Applied Biosystems (ABI); Affymetrix (AFX); Agilent Technologies (AGL for two-color and AG1 for one-color); GE Healthcare (GEH); Illumina (ILM) and Eppendorf (EPP). In addition, scientists at the National Cancer Institute (NCI) generated spotted microarrays using oligonucleotides obtained from Operon. The RNA sample types were also tested on three alternative gene expression platforms: TaqMan Gene Expression Assays from Applied Biosystems (TAQ TaqMan is a registered trademark of Roche Molecular Systems, Inc.); StaRT-PCR from Gene Express (GEX) and QuantiGene assays from Panomics (QGN). Each microarray platform provider selected three sites for testing. In most cases, five replicate assays for each of the four sample types were processed at each of the test sites. Six of the microarray providers used one-color protocols where one labeled RNA sample was hybridized to each microarray (Table 1). The Agilent two-color and NCI microarrays were tested using a two-color protocol so that two differently labeled RNA samples were simultaneously hybridized to the same microarray. The Eppendorf assay contained two identical microarrays on one glass slide, which were independently hybridized to two samples. Although only a single fluorescent dye was used, the Eppendorf data are presented in a ratio format.

Each microarray provider used its own software to generate a quantitative signal value and a qualitative detection call for each probe on the microarray. This attention to the qualitative calls of each platform resulted in our using a potentially different number of genes in each calculation. It also had an impact on data analysis, because some, but not all, of the platforms removed suspect or low intensity data. In addition, 11 hybridizations were removed from further analysis due to quality issues. Table 1 notes the final number of hybridizations used in the final data analysis for each microarray platform. Further details are presented in Methods and Tables S1-S4 in Supplementary Data online. Pre-hybridization and posthybridization quality information of samples is available as Supplementary Table 1 online. A direct comparison of results across platforms was challenging because of inherent differences in protocols, number of data points per platform and data preprocessing methods. Whenever possible, all platforms were included in any comparisons, but occasionally results from one or two platforms were excluded from an analysis because the data comparison was untenable and forced contrivance that was ultimately uninformative. Although some data from the alternative platforms are presented in this article, a more thorough discussion is included elsewhere22. Probe mapping Microarray experiments generally rely on a hybridization intensity measurement for an individual probe to infer a transcript abundance level for a specific gene. This relationship raises several difficult issues, including: which gene corresponds to which probe, and how sensitive and specific is the probe. Previous publications have suggested that some of the variability in cross-platform studies was due to annotation problems that made it difficult to reconcile which genes were measured by specific probes27–30. Despite the fact that the human genome sequence is complete, the final list of actual genes has yet to be determined. All identifiers are moving targets, and even the NCBI hand-curated reference sequences are often modified. Another issue is that a gene expression assay designed to measure a given RNA target may unknowingly detect multiple alternatively spliced transcripts, which may have different functions and expression patterns. Thus, the number of genes or transcripts

Table 1 Gene expression platforms and data analyzed in the MAQC main study Number of Number of Number of Number of Total number of Manufacturer

probesa

test sites

samples

replicates

microarraysb

Code

Protocol

Platform

Applied Biosystems

ABI

One-color microarray

Human Genome Survey Microarray v2.0

32,878

3

4

5

58

Affymetrix Agilent

AFX One-color microarray HG-U133 Plus 2.0 GeneChip AGL Two-color microarrayc Whole Human Genome Oligo Microarray, G4112A

54,675 43,931

3 3

4 2

5 10

60 56

Eppendorf

AG1 EPP

One-color microarray Whole Human Genome Oligo Microarray, G4112A One-color microarray DualChip Microarray

43,931 294

3 3

4 4

5 5

56 60

GE Healthcare Illumina

GEH ILM

One-color microarray One-color microarray

CodeLink Human Whole Genome, 300026 Human-6 BeadChip, 48K v1.0

54,359 47,293

3 3

4 4

5 5

60 59

NCI_Operon Applied Biosystems

NCI TAQ

Two-color microarray TaqMan assays

Operon Human Oligo Set v3 4200,000 assays available

37,632 1,004

2 1

4 4

5 4

33 N/A

Panomics Gene Express

QGN GEX

QuantiGene assays StaRT-PCR assays

B2,600 assays available B1,000 assays available

245 207

1 1

4 4

3 3

N/A N/A

Total

442

aA

global definition of probes is used to include individual probes, probe sets or primer pairs depending on the gene expression platform. The numbers listed in this table are derived from product literature and may include some platform duplication. Alternative figures for the number of probes analyzed are provided as Table S5 in Supplementary Data online. bMaximum number of microarrays per one-color protocol is 60 (3 sites 4 sample types 5 replicates). As described in the text, replacement hybridizations but not outlier hybridizations are included in the main study data analysis. Only data from 386 microarrays were analyzed in this article. Additional data sets are described in Table S4 in Supplementary Data online. cAlthough not presented in this paper, the Agilent two-color data (56 microarrays) are discussed elsewhere24. In the remaining figures, test sites and sample types are referenced using the following nomenclature: ‘‘platform code_test site_ sample ID’’. Sample A ¼ 100% UHRR; Sample B ¼ 100% HBRR; Sample C ¼ 75% UHRR: 25% HBRR; and Sample D ¼ 25% UHRR: 75% HBRR.

1152

VOLUME 24

NUMBER 9

SEPTEMBER 2006


Figure 1 Repeatability of expression signal 12,000 60 within test sites. For the one-color platforms, the CV of the expression signal values between site replicates of the same sample type was 10,000 50 calculated for all generally detected genes. The distributions of these replicate CVs are presented in a series of twelve box and whiskers plots for 8,000 40 each microarray platform: one for each of the four sample types at the three test sites. The plots are highlighted to distinguish the sample 6,000 30 replicates: sample A (white), sample B (light blue), sample C (light purple) and sample D (dark blue). The twelve plots showing results from the platforms with three test sites are presented in 4,000 20 the following order from left to right: A1, A2, A3, B1, B2, B3, C1, C2, C3, D1, D2 and D3. For the two-color NCI platform, the CV of the expression 2,000 10 Cy3/Cy5 ratios between site replicates of the same sample type was similarly calculated. The distributions of these replicate CVs are presented A B C D A B C D A B C D A B C D A B C D A B C D A B C D 0 0 in a series of eight box and whiskers plots from ABI AFX AG1 EPP GEH ILM NCI the two NCI test sites in the following order from Platform-sample left to right: A1, A2, B1, B2, C1, C2, D1, and D2. The median (gap), interquartile range as well as the 10th and 90th percentile values are indicated in each plot. Only genes from the 12,091 common set that were detected in at least three of the replicates were included in the box plots and CV calculations. This number varies by platform/sample/test site and is noted as the line plot with the secondary axis and as Table S6 in Supplementary Data online. The platforms and sample types are labeled according to the nomenclature presented in Table 1. CV (%)

No. of detected genes


ARTICLES

detected with a gene expression platform is inherently difficult to define and quantify. A unique advantage of the MAQC project is that most of the sequence information for the probes used in each gene expression technology was provided by the manufacturers. We mapped the probes (see Supplementary Methods online and Supplementary Notes online) to the Ref Seq human mRNA database31 (http:// www.ncbi.nlm.nih.gov/Ref Seq) and to the AceView database32 (http://www.ncbi.nlm.nih.gov/IEB/Research/Acembly), a less curated but more comprehensive database, which includes all the Ref Seq, GenBank and dbEST human cDNA sequences. Although the total number of probes varied across platforms, the six high-density microarray platforms assayed similar numbers of Entrez genes (15,429–16,990) and had similar percentages of probes (68–84%) that aligned to AceView transcripts (see Table S5 in Supplementary Data online). We found that 23,971 of the 24,157 Ref Seq NM Accessions from the March 8, 2006 release were assayed by at least one platform (Supplementary Table 2 online) and that 15,615 Accessions were assayed by all high-density microarray platforms used in the MAQC study. Because of alternative splicing, each platform mapped to roughly four Ref Seq transcripts per three Entrez genes. To simplify the interplatform comparison, we condensed the complex probe-target relationships to a ‘one-probe-to-one-gene’ list. The 15,615 Ref Seq entries on all of the high-density microarray platforms represented 12,091 Entrez genes. For each gene, we selected a single Ref Seq entry (Supplementary Table 4 online), primarily the one annotated by TaqMan assays, or secondarily the one targeted by the majority of platforms. When a platform contained multiple probes matching the same Ref Seq entry, only the probe closest to the 3¢ end was included in the common set (Supplementary Table 3 online). In this way, we selected for each high-density platform 12,091 probes matching a common set of 12,091 reference sequences from 12,091 different genes (Supplementary Table 5 online).


VOLUME 24

NUMBER 9

SEPTEMBER 2006

Intraplatform data repeatability and reproducibility We examined microarray data for consistency within each platform by reviewing both the intrasite repeatability and the intersite reproducibility at two levels: the quantitative signal values and the qualitative detection calls. Only genes that were detected in at least three of the five sample replicates (or generally detected genes) were included in most of these calculations. This filter accounts for the different manner in which the microarray platforms identified genes below their quality thresholds, and directs our research away from the less confident, noisy results. The number of generally detected genes for each sample type at each site varied from 8,000 to 12,000 for the highdensity microarray platforms, but was relatively consistent between test sites using the same platform (Fig. 1). The coefficient of variation (CV) of the quantitative signal values between the intrasite replicates was calculated using the generally detected subset from the 12,091 common genes for each sample type at every test site. The distribution of the replicate CV measures across the set of detected genes is displayed in a series of box and whiskers plots in Figure 1. Most of the one-color microarray platforms and test sites demonstrated similar replicate CV median values of 5–15%, although the distributions of replicate CV results differed between platforms. For the two-color NCI microarrays, the replicate CVs were calculated using the Cy3/Cy5 ratios. (Sample type A was used as the Cy5 reference in all NCI hybridizations.) These values were only slightly larger than the one-color signals for the same sample type. We next examined the total CV of the quantitative signal, which included both the intrasite repeatability as well as variation due to intersite differences. By definition, the total CV measure (n r 15) will be larger than the replicate CV measures (n r 5). Median values for the total CV distribution and the average of three replicate CV medians for each platform are presented in Figure 2. Overall, the total CV median was very consistent across all platforms, ranging from 10% to just over 20% and not dramatically higher than the replicate CV median values. In general, the total CV median was up to twice as large as the replicate CV median, but this result is not

1153

12,000

25

10,000

20

8,000

15

6,000

10

4,000

5

2,000

0

0 A B CD ABI

A B CD AFX

A B CD AG1

A B CD EPP

A B CD GEH

A B CD ILM

Platform-sample

Figure 2 Signal variation within and between test sites. For each of the four sample types, the replicate CV of signal within a test site (blue bar) and the total CV of signal across and within sites (red bar) are presented. As in Figure 1, genes detected in at least three of the replicates of a sample type at a single test site are included in the replicate CV calculation. Genes present in the intersection of these gene lists are included in the total CV calculation. (These gene lists are therefore slightly different than those in Figure 1.) The number of such genes within each platform and sample type is noted by blue dots connected by lines and is read on the secondary axis. It is also reported as Table S6 in Supplementary Data online. Intrasite normalization was performed according to default settings for each manufacturer, and intersite normalization was performed by scaling between sites (see main text). The NCI platform is omitted because data from only two test sites was available in the main study so intersite reproducibility measures may not be representative. The platforms and sample types are labeled according to the nomenclature presented in Table 1.

unexpected and simply implies that site-related effects should be taken into account when combining data from multiple sites using the same platform. To assess variation in the qualitative measures, the percentage of the 12,091 common genes with concordant detection calls between replicates of the same sample type was calculated for each of the four sample types on each platform (Fig. 3). These figures include

either all sample replicates at a single site (n r 5) or all sample replicates across the test sites (n r 15). Most one-color test sites demonstrated 80–95% concordance in the qualitative calls for the sample replicates within their facility. The value dropped to 70–85% concordance for the reproducibility of the qualitative calls across all three test sites. It is not surprising that platforms with more detected calls (Fig. 1) generally had higher concordance percentages. For example, the NCI microarrays detected almost all of the 12,091 common genes and had concordance percentages near 100% between test sites. Microarray platforms that had lower numbers of detected genes generally had reduced concordance percentages. Interestingly, the GE Healthcare platform had both a large number of genes detected (B11,000 per hybridization) and approximately 85% concordance between test sites. Interplatform data comparability Expression values generated on different platforms cannot be directly compared because unique labeling methods and probe sequences will result in variable signals for probes that hybridize to the same target. Alternatively, the relative expression between a pair of sample types should be maintained across platforms. For this reason, we examined the microarray data for comparability between platforms by reviewing sample type B relative to sample type A expression values with three different metrics: differential gene list overlap, log ratio compression and log ratio rank correlation. For log ratio compression and rank correlation, only generally detected genes from the common 12,091 gene list were included in the analysis. For the gene list overlap, all 12,091 common genes were considered. A list of differentially expressed genes was generated for each test site and compared to lists from other test sites using the same platform and those using a different platform. A percent score was calculated to indicate the number of genes in common between each pair of test site lists. The percentage of overlap for each comparison is displayed in Figure 4. Note the graphic comparisons are asymmetrical indicating the analysis is performed in two directions. That is, the percentage of test site Y genes on the list from test site X can be different from the

100

Figure 3 Concordance of detection calls within and between test sites. For the 12,091 common 90 genes, detection calls within each platform were categorized as either ‘detected’ or ‘not detected.’ 80 For each sample type within each platform, the percentage of genes with calls that were perfectly 70 concordant as ‘detected’ within the replicates 60 for a given site is plotted as blue dots, and the corresponding percentage of genes with calls 50 perfectly concordant as ‘detected’ across all sites are plotted as the blue bars. The total percentage 40 of genes with perfectly concordant calls (detected and not detected) within a site is plotted as the 30 yellow dots, and the corresponding percentage of genes with calls perfectly concordant across all 20 sites is plotted as the top of the yellow bars. The bars are split between perfectly detected genes 10 123 123 123123 123 123123123 123 123123123 123 123123123 123123123 123 12 12 1212 (blue portion) and perfectly not detected genes A B C D A B C D A B C D A B C D A B C D A B C D (yellow portion) across all test sites. It is not 0 AFX AG1 GEH ILM NCI ABI expected that detected genes are concordant Platform-sample-site across sample types. The number of perfectly detected genes for each test site is provided as Table S6 in Supplementary Data online. As described in the main text, the stringency with which individual platforms determine that the data for a gene is sufficiently reliable to be called detected has different manufacturer defaults, leading to altered concordance percentages. Changes in the settings for sensitivity/specificity may shift the proportion of the bar assigned to each detection category. Because reliability depends on platform-specific details, detected calls do not correspond directly to relative abundance and may vary between platforms. Note: as some platforms have removed outlier hybridizations, the number of replicates within (n r 5) and between sites (n r 15) varies for determining concordance. Genes with consistent calls (%)


30

No. of detected genes across sites

Median CV (%)

ARTICLES

1154

VOLUME 24

NUMBER 9

SEPTEMBER 2006


ARTICLES ABI_1

Overlap between test site pairs (%)

ABI_2

80 70 60 50 40 30 20 10 0

ABI_3 AFX_1 AFX_2 Test site Y gene list

AG1_2 AG1_3 GEH_1 GEH_2 GEH_3 ILM_1 ILM_2 ILM_3 NCI_1 NCI_2

ILM_3

NCI_1

ILM_2

ILM_1

GEH_3

GEH_2

AG1_3

GEH_1

AG1_2

AFX_3

AG1_1

AFX_2

ABI_3

AFX_1

ABI_2

ABI_1

NCI_2

Test site X gene list

percentage of test site X genes on the test site Y list. For all but the NCI test sites, the gene list overlap is at least 60% for each test site comparison (both directions) with many site pairings achieving 80% or more between platforms and 90% within platforms. Typically, the genes that the NCI microarray platform identified as differentially expressed were also identified on the other platforms, suggesting a low false positive rate for this platform. However, the converse was

a

ABI_1 ABI_2 ABI_3 AFX_1 AFX_2 AFX_3 AG1_1 AG1_2 AG1_3 EPP_1 EPP_2 EPP_3 GEH_1 GEH_2 GEH_3 ILM_1 ILM_2 ILM_3 NCI_1 NCI_2 GEX_1 QGN_1 TAQ_1

Compression/expansion difference between site pairs 0.60 0.50 0.40 0.30 0.20 0.10 0.00 –0.10 –0.20 –0.30 –0.40 –0.50 –0.60

not necessarily true, most likely due to more log ratio compression observed in the NCI platform and the use of a stringent P-value threshold. Each microarray platform has a defined background correction method and dynamic range of signal detection, which can lead to over- or underestimates of log ratios and fold changes in expression between sample types. To examine the level of compression or ABI_1 b ABI_2

Rank correlation of log ratios between site pairs

ABI_3 AFX_1 AFX_2 AFX_3 AG1_1 AG1_2 AG1_3 EPP_1 EPP_2 EPP_3 GEH_1 GEH_2 GEH_3 ILM_1 ILM_2 ILM_3 NCI_1 NCI_2 GEX_1 QGN_1 TAQ_1

1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00



Test site Y


AFX_3 AG1_1

Figure 4 Agreement of gene lists. This graph indicates the concordance of genes identified as differentially expressed for pairs of test sites, labeled as X and Y. A list of differentially expressed genes between sample type A replicates versus sample type B replicates was generated for each test site (using the 12,091 common genes with Z twofold change and P o 0.001 thresholds) and compared for commonality to other test sites. The size of these gene lists is reported as Table S7 in Supplementary Data online. No filtering related to the qualitative detection call was performed. The color of the square in the matrix reflects the percent overlap of genes on the list for the test site Y (listed in row) that are also present on the list for the test site X (listed in column). A light-colored square indicates a high percent overlap between the gene lists at both test sites. A dark-colored square indicates a low percent overlap, suggesting that most genes identified in site Y were not identified in site X. Numerical values for the percent overlap are presented as Table S9 in Supplementary Data online. Note: the graph is asymmetric and not complementary. Only the six high-density microarray platforms are presented. As described in the text, data from some platforms were omitted from these calculations because of quality issues. The platforms and sample types are labeled according to the nomenclature presented in Table 1. The _1, _2 and _3 suffixes refer to test site location.

Test site X

Test site X

Figure 5 Agreement of log ratios across platforms and test sites. (a) Log ratio compression/expansion. This graph indicates the percent difference from equivalency between platform/sites (corresponding to a slope value 1 for the best fitted line using orthogonal regression) of the log ratio differential expression using A and B replicates. A dark spot implies equivalency (slope ¼ 1 - percent difference ¼ 0). A positive percent difference in slope from the ideal line (aqua) indicates compression of log signal for test site Y relative to test site X. A negative percent difference in the ideal line (magenta) indicates expansion. Read as ‘‘What is the difference from equivalence in slope (m ¼ 1) for the test site Y versus test site X ?’’ Only genes detected by both test sites in at least three replicates of sample type A and three replicates of sample type B are included in the calculation, and the number for each pair is reported as Table S8 in Supplementary Data online. Numerical values for the percent difference are presented as Table S10 in Supplementary Data online. Note: the graph is asymmetric, but approximately complementary. As described in the text, data from some platforms were omitted from these calculations due to quality issues. The platforms and sample types are labeled according to the nomenclature presented in Table 1. The _1, _2 and _3 suffixes refer to test site location. (b) Rank correlation of log ratios. This graph indicates the correlation of the log ratio differential expression values (using A versus B replicates) when we examine their rank. Large positive log ratio values would be ranked high and large negative log ratio values would be ranked low. Read as ‘‘What is the correlation of the rank log ratio values between the test site Y and the test site X?’’ Only genes generally detected in both sample types A and B and by both test sites are included in the calculation, and the number for each pair is reported as Table S8 in Supplementary Data online. Numerical values for the rank correlation are presented as Table S11 in Supplementary Data online. Note: the graph is symmetric. As described in the text, data from some platforms were omitted from these calculations due to quality issues. The platforms and sample types are labeled according to the nomenclature presented in Table 1. The _1, _2 and _3 suffixes refer to test site location.


VOLUME 24

NUMBER 9

SEPTEMBER 2006

1155

ARTICLES

b

12

d

12

12 10

8

8

8

8

6

6

6

6

4 2 0 –2 –4

4 2 0 –2 –4 –6

–6

–12

Site1: n = 528, r = 0.86 Site2: n = 523, r = 0.85 Site3: n = 567, r = 0.84 –12 –10 –8 –6 –4 –2 0

2

4

6

–12

–12 –10 –8 –6 –4 –2 0

TaqMan log ratio

0 –2 –4

2

4

6

–12

8 10 12

f

12

–12 –10 –8 –6 –4 –2 0

2

4

g

12

10

8

8

6

6

6

0 –2 –4 –6 –8 –10 –12

4 2 0 –2 –4 –6

Site1: n = 670, r = 0.84 Site2: n = 680, r = 0.86 Site3: n = 660, r = 0.84 –12 –10 –8 –6 –4 –2 0

2

4

6

TaqMan log ratio

8 10 12

0 –4

6

Site1: n = 53, r = 0.91 Site2: n = 64, r = 0.79 Site3: n = 84, r = 0.83

–8 –10 –12

8 10 12

–12 –10 –8 –6 –4 –2 0

2

4

6

8 10 12

TaqMan log ratio

12

10

8

2

2 –2

TaqMan log ratio

10

4

4

–6


–8 –10

TaqMan log ratio

ILM log ratio

GEH log ratio

e

2

–6


–8 –10

8 10 12

4

NCI log ratio

–8

EPP log ratio

10

AG1 log ratio

10

–10


c

12

10

AFX log ratio

ABI log ratio

a

4 2 0 –2 –4 –6


–8 –10 –12

–12 –10 –8 –6 –4 –2 0

2

4

6

8 10 12

Site1: n = 769, r = 0.82 Site2: n = 740, r = 0.83

–8 –10 –12

–12 –10 –8 –6 –4 –2 0

TaqMan log ratio

2

4

6

8 10 12

TaqMan log ratio

Figure 6 Correlation between microarray and TaqMan data. The scatter plots compare the log ratio differential expression values (using A versus B replicates) from each microarray platform relative to values obtained by TaqMan assays. Each point represents a gene that was measured on both the microarray and TaqMan assays. The spot coloring indicates whether the data were generated in test site 1 (black), test site 2 (blue) or test site 3 (red) for the microarray platform. Only genes that were generally detected in sample type A replicates and sample type B replicates were used in the comparisons. The exact number of probes analyzed for each test site and its correlation to TaqMan assays are listed in the bottom right corner of each plot. As described in the text, data from some platforms were omitted from these calculations because of quality issues. The platforms and sample types are labeled according to the nomenclature presented in Table 1. The line shown is the ideal 451 line.

expansion in log ratios, we determined the best fitted line for the log ratio estimates between pairs of test sites. The percent difference of the slope for each comparison is displayed in Figure 5a. An ideal slope of 1 would result in a percent difference of 0; negative or positive percent differences in the slope of the ideal line indicate compression or expansion of the log ratios in one test site relative to the other. For each commercial one-color platform, good agreement was observed between its three test sites. Most of the interplatform test site comparisons also showed little compression or expansion. Test site 1 for the NCI microarrays produced consistently different results from the other test sites, both within and between platforms. The comparability of results across platforms was also examined using a rank correlation metric. Log ratios for the differential expression observed between sample B replicates and sample A replicates were calculated for the generally detected common genes and then compared between test sites and across platforms. The rank correlations of the log ratios are displayed visually in Figure 5b. Good agreement was observed between all sites, even those using different microarray platforms. In fact, the median rank correlation was 0.87 and the smallest rank correlation value was 0.69 between the microarray platforms. Assessing relative accuracy The relative accuracy of the microarray platforms can be assessed using either the titrated mixtures of the RNA samples23 or gene abundance measurements collected with alternative technologies22. Figure 5, as well as Tables S12 and S13 in Supplementary Data online, illustrate the relative rank correlation and compression/expansion values for log (B/A) between microarray-based and alternative gene expression technologies. Further comparisons between each microarray platform relative to the TaqMan assays are presented as scatter plots in Figure 6.

1156

The log ratios of sample type B to sample type A expression detected on the TaqMan assays were compared to the log ratios obtained for the same genes on the microarray assays. Only genes that were generally detected in both sample A and B replicates on the TaqMan assays and on the microarray were included in this analysis. The relative accuracy of each high-density platform to the TaqMan assay data was generally higher for those microarray platforms with fewer genes detected as indicated by number and magnitude of deviations from the ideal 451 line indicated in Figure 5a and Figure 6. Correlation with alternative platforms Similarly, the Affymetrix, Agilent, and Illumina platforms displayed high correlation values of 0.90 or higher with TaqMan assays based on comparisons of B450–550 genes, whereas the GE Healthcare and NCI platforms had a reduced average correlation of 0.84, but included almost 30% more genes in the data comparisons. These additional genes were not identified as ‘not detected’ during the data review process, but may represent less confident results due to lower signals exhibiting greater variance. Thus, much of the difference in comparability metrics may be a reflection of the algorithm used to assign detection calls. Similar correlation values for the microarray platforms were observed relative to each of the other alternative platforms, StaRT-PCR, and QuantiGene22. DISCUSSION The results of the MAQC project provide a framework for assessing the potential of microarray technologies as a tool to provide reliable gene expression data for clinical and regulatory purposes. All onecolor microarray platforms had a median CV of 5–15% for the quantitative signal (Fig. 1) and a concordance rate of 80–95% for the qualitative detection call (Fig. 3) between sample replicates. This variation increased when data from different test sites using

VOLUME 24

NUMBER 9

SEPTEMBER 2006



ARTICLES the same platform were included (Figs. 2 and 3). However, lists of differentially expressed genes averaged B89% overlap between test sites using the same platform and B74% overlap across onecolor microarray platforms (Fig. 4). Importantly, the ranks of log ratios were highly correlated among the microarrays (minimum R ¼ 0.69; Fig. 5b), indicating that all platforms were detecting similar changes in gene abundance. These results indicate that, for these sample types and these laboratories, microarray results were generally repeatable within a test site, reproducible between test sites and comparable across platforms, even when the platforms used probes with sequence differences as well as unique protocols for labeling and expression detection. Within the MAQC study, there were notable differences in various dimensions of performance between microarray platforms. Some platforms had better intrasite repeatability overall (e.g., Illumina), better intersite reproducibility (e.g., Affymetrix), or more consistency in the detection calls (e.g., GE Healthcare). Likewise, some platforms were more comparable to TaqMan assays (e.g., Applied Biosystems and Agilent one-color), whereas others demonstrated signal compression (e.g., NCI_Operon). Some of these differences were manifest in the apparent power analyses (see Figure S1 in Supplementary Data online) as test sites with smaller CV values (Fig. 1) typically had more power to discriminate differences between groups, as would be expected. Other differences might have been related to the platform’s signal-to-analyte response characteristics22. It is important to note that 11 (2.4%) of the 453 microarray hybridizations were removed from the analysis due to quality issues (listed as Table S1 in Supplementary Data online). The relative performance of some platforms might have been altered if this data filter had not been applied. Each microarray platform has made different trade-offs with respect to repeatability, sensitivity, specificity and ratio compression. One interesting result was that platforms with divergent approaches to measuring expression often generated comparable results. For example, data from Affymetrix test sites, which use multiple short oligonucleotide probes per target with perfect match and mismatch sequences, and Illumina test sites, which use plasma-etched silicon wafers containing beads with long oligonucleotide probes, were remarkably similar in the numbers of genes detected and the detection call consistency, gene list overlap and ratio compression analyses. In other words, the expression patterns generated were reflective of biology regardless of the differences in technology. Some of the results were affected by differences in data analysis and detection call algorithms. This effect is most noticeable in the foldchange compression observed in the two-color results from the NCI microarrays, which generally included low intensity probes resulting in over 95% detection call rate. The comparability of the NCI microarrays relative to the other platforms improves when background is based on ‘alien’ or negative control sequences. This alternative method reduces the detection call rate to 60–70%, while generally increasing the absolute fold changes in up- and downregulated genes (E.S.K., unpublished data). Interestingly, the NCI platform had lower intrasite repeatability (Fig. 1), but demonstrated comparable rankings in log ratios when compared to the other platforms (Fig. 5b). Additional analyses of the MAQC data are provided in the accompanying articles. For example, the microarray platforms detected known differences in gene abundance between defined RNA mixtures23 and generated differential expression results that were comparable with other gene expression platforms22,24. The comparability of the gene expression results increased when the microarrays and other methodologies analyzed overlapping


VOLUME 24

NUMBER 9

SEPTEMBER 2006

sequences from the same gene22. Furthermore, external RNA controls included in some microarray platforms were useful predictors of technical performance25. Direct comparison of different microarray platforms is neither a new nor an original idea in the realm of high-throughput biology. However, the data set generated by the MAQC project is unique in both its size and content. The main study compares seven different microarray platforms and includes B60 hybridizations per platform using well-characterized, commercially available RNA sample types. Including the reagents used in the two pilot studies and the toxicogenomics validation study26, 1,327 microarrays have been used for this project (see Table S4 in Supplementary Data online). Moreover, the availability of the probe sequences in the MAQC project enabled us to approach the interplatform comparisons with greater scientific rigor. We performed detailed probe mapping to confirm identity and reveal potential sequence- or target-based differences between the gene expression platforms. This analysis confirmed that the great majority of probes were very carefully chosen and of high quality. Most of the results in this report are based on a set of 12,091 common genes that are represented on six high-density microarray platforms, but which generally use different probe sequences for detection. Our probe selection procedure may have introduced a bias in the study because the imposed criteria neither reflect the platform design philosophies nor does it account for the very rich underlying biology. More than one probe per target can be a highly desirable feature on microarray platforms because a single probe may not capture all tissue-specific effects. We also found a number of probes that were not gene specific, suggesting a strategy of targeting multigene families. The MAQC data set captures intrasite, intersite and interplatform differences. However, it does not address protocol, time or other technical variables within a test site because all test sites used the same protocol and generated replicate data at approximately the same time (except as noted under data filtering). The effect and levels of these sources of variation have been described in other studies15,33. Furthermore, our analysis does not include performance metrics based on ‘biology’ (e.g., Gene Ontology terms or pathways)26. Though a relatively high level of concordance of differentially expressed gene lists were observed in this study, it is possible that a higher level of agreement would be detected using these other methods of gene list concordance34, or that a lower level would be observed with sample types that were more realistically similar. It should be noted that the results presented in this paper in terms of log ratios and overlap of lists of differentially expressed genes were derived from comparing sample types A and B, which exhibited the greatest differences among the four sample types used in the MAQC project. In practical applications, the expected differences between sample types (e.g., treated versus control animals) are usually much smaller compared to those seen between sample types A and B. Therefore, the comparability of microarray data reported in this paper does not necessarily mean that the same level of consistency would be achieved in toxicogenomic or pharmacogenomic applications. This difference can be seen from the relatively lower power and smaller overlap of gene lists (see Figures S1-S2 in Supplementary Data online) when comparing sample types C and D, where the maximum fold change is three. The MAQC data set can be used to compare normalization methods23 and data analysis algorithms26 (see Figure S2 in Supplementary Data online), similar to a currently available website (http:// affycomp.biostat.jhsph.edu) which illustrates the impact of the different data analysis methods on expression results30,34. It is our hope that

1157


ARTICLES future studies will add to the MAQC data set. For example, microarray providers could submit gene expression results from new microarrays with updated probe content and then use the MAQC data set to confirm consistency with older versions of the microarray. In an effort to equally represent all platforms and to present results in a timely manner, this publication analyzed only 386 microarray hybridizations from 20 test sites. However, additional data sets from the MAQC main study are available (listed as Tables S1-S4 in Supplementary Data online). Although most sites generated quality results, some differences were detected between test sites using the same platform. Thus, microarray studies need unified metrics and standards, which can be used to identify suboptimal results and monitor performance in microarray facilities. Previous reports have relied heavily on the statistical significance (P value) rather than on the actual measured quantity of differential expression (fold change or ratio) in identifying differentially expressed genes. This strict reliance on P values alone has resulted in the apparent lack of agreement between sites and microarray platforms20,26. Our results from analyzing the MAQC human data sets (see Figure S2 in Supplementary Data online) and the rat toxicogenomics data set26 indicate that a straightforward approach of foldchange ranking plus a nonstringent P cutoff can be successful in identifying reproducible gene lists, whereas ranking and selecting differentially expressed genes solely by the t-test statistic predestine a poor concordance in results, in particular for shorter gene lists, due to the relatively unstable nature of the variance (noise) estimate in the t-statistic measure. More robust methods such as ranking using the test statistic from the Significance Analysis of Microarrays (SAM)35 did not generate more reproducible results compared to fold-change ranking in our cross-laboratory and interplatform comparisons. Our results are consistent with previously published data20. Furthermore, the impact of normalization methods on the reproducibility of gene lists becomes minimal when the fold change, instead of the P value, is used as the ranking criterion for gene selection24,26. Two initiatives for microarray reference materials are currently in progress. A group led by FDA’s Center for Drug Evaluation and Research (CDER) developed two mixed-tissue RNA pools with known differences in tissue-selective genes that can be used as rat reference materials36, whereas the External RNA Controls Consortium (ERCC) is testing polyadenylated transcripts that can be added to each RNA sample before processing to monitor the technical performance of the assay37. The MAQC project complements these efforts by establishing several commercially available human reference RNA samples, and an accompanying large data set, which can be used by the scientific community to compare results generated in their own laboratories for quality control and performance validation efforts. In fact, the commercial availability of the MAQC reference sample types allowed several laboratories to generate and submit additional gene expression data to the MAQC project after the official deadline (listed as Table S4 in Supplementary Data online). Repeated intersite comparisons, such as a proficiency testing, are required three times a year for many Clinical Laboratory Improvement Amendments (CLIA) assays and may also be useful in microarray facilities to monitor the comparability and consistency of data sets generated over time38. For example, a proficiency testing program evaluated the performance over a 9-month period of 18 different laboratories by repeatedly hybridizing three replicates of the same two RNA sample types to Affymetrix microarrays (L.H.R. and W.D.J., unpublished results). This study revealed the range of quality metrics and the impact of protocol differences on the microarray results. The MAQC human reference RNA

1158

sample types could be used in this kind of intersite proficiency testing program. In summary, the technical performance of microarrays as assessed in the MAQC project supports their continued use for gene expression profiling in basic and applied research and may lead to their use as a clinical diagnostic tool as well. International organizations such as ERCC37, the Microarray Gene Expression Data Society39 and this MAQC project are providing the microarray community with standardization of data reporting, common analysis tools and useful controls that can help provide confidence in the consistency and reliability of these gene expression platforms. METHODS Probe mapping. Affymetrix, Agilent, GE Healthcare, Illumina and Operon oligonucleotides used by the NCI provide publicly available probe sequences for their microarray platforms in a spreadsheet format (websites listed in Supplementary Data online). The probe sequences for Applied Biosystems microarrays can be individually obtained through the Panther database (http:// www.pantherdb.org) and the sequences of the intended regions for QuantiGene (Panomics) assays are available upon request. Probe sequences for Eppendorf microarrays are not yet publicly available, but were provided to the MAQC project for confidential analysis. Gene Express provided annotation and approximate forward and reverse primer locations for the StaRT-PCR assays, which were sufficient to localize the intended target. For TaqMan assays, Applied Biosystems provided Assay ID, amplicon size, assay location on the Ref Seq and a context sequence (exact 25-nt sequence that includes the TaqMan assay detection probe). The MAQC probe mapping (Supplementary Methods online and Supplementary Notes online) used the March 8, 2006 Ref Seq release containing 24,000 curated accessions to which we subjectively added 157 entries that were recently either withdrawn or retired from the NCBI curation. AceView comparisons were based on the August 2005 database32. An exact match of the sequence of the probe to the database entry was required. Probes matching only the reverse strand of a transcript were excluded as well as probes matching more than one gene. An exact match of 80% of the probes within a probe set (usually 9 probes out of 11) was required for Affymetrix. The results based on these stringent criteria are provided as Supplementary Tables 2–5 online and summarized as Table S5 in Supplementary Data online. The counts for the StaRT-PCR and TaqMan assays were based on the annotation provided by Gene Express and Applied Biosystems. In the AceView analysis, the mapping was tolerant to low levels of noncentral mismatches, but applied a stringent gene-specific filter so that probes which potentially cross-hybridize were removed even if they had a single exact match. RNA preparation. The total RNA sources were tested and selected based on the results of 160 microarrays from Pilot Project I (data not shown). The Universal Human Reference RNA (catalog no. 740000) and Human Brain Reference RNA (catalog no. 6050) were generously donated by Stratagene and Ambion, respectively. The four titration mixtures of the samples were selected based on the results of 254 microarrays from Pilot Project II (data not shown) and prepared as described elsewhere23. The titration pools were mixed at the same time at one site using a documented protocol (MAQC_RNA_Preparation_ SOP.doc) available at the MAQC website (http://www.fda.gov/nctr/science/ centers/toxicoinformatics/maqc/). Each test site received 50-mg aliquots of the four sample types and confirmed the RNA quality using a Bioanalyzer (Agilent) before initiating target preparation. Target preparation and quality assessments. Every test site was provided with instructions (MAQC_Sample_Processing_Overview_SOP.doc) on the processing of RNA samples, conducting quality assessment of RNA reference samples, target preparations and replication guidelines, standardized nomenclature for referencing samples and a template for reporting quality assessment data (MAQC_RNA_Quality_Report_Template.xls). The gene expression vendors generously provided all reagents to the test sites. Each microarray test site assessed cRNA yields using a spectrophotometer and determined the median transcript sizes using a Bioanalyzer (Agilent). Pre-hybridization and posthybridization quality metrics are presented as Supplementary Table 1 online.

VOLUME 24

NUMBER 9

SEPTEMBER 2006



ARTICLES Some statistically significant differences were observed in these quality metrics between sites (data not shown). Affymetrix, Agilent, Applied Biosystems and Eppendorf test sites added platform-specific external RNA controls to the samples before processing25. Data were submitted to the FDA’s National Center for Toxicological Research (FDA/NCTR) directly from each test site and distributed to the eleven official analysis sites for review. Lists of the gene expression test sites and data analysis centers are available as Tables S1 and S2 in Supplementary Data online. All test sites for one vendor used the same target preparation protocols and processed all replicates at approximately the same time, with two exceptions: (i) Microarray slides at the NCI test sites were scanned at 100% laser power, but the photomultiplier setting varied from slide to slide and (ii) some outlier hybridizations were repeated at a later date as described below. Exact protocols for sample processing are available at the MAQC website (http://www.fda.gov/ nctr/science/centers/toxicoinformatics/maqc/) and are briefly described in Supplementary Data online. Data filters. Outlier hybridizations were repeated or removed from the analysis after the original data submission deadline in October 2005. One site each for the NCI and GE Healthcare platforms repeated all sample types in the MAQC study (NCI_2 and GEH_2) due to protocol issues. One Illumina site (ILM_2) repeated two samples in the MAQC study due to low cRNA yield, and another Illumina site (ILM_1) did not hybridize one sample replicate due to the same reason. Data quality from 11 hybridizations at seven test sites (ABI_2, ABI_3, AG1_1, AG1_2, AG1_3, AGL_1 and AGL_2) was not satisfactory. More details are provided as Table S3 in Supplementary Data online. Data processing. The platform-specific methods used for background subtraction, data normalization and the optional incorporation of offset values are described in Supplementary Data online. Each test site submitted its data (including image files) to the FDA/NCTR. All data were imported into the ArrayTrack database system40,41 and preprocessed and normalized according to the manufacturer’s suggested procedures. Each gene was reviewed for quality and marked with a detection call, using the manufacturer’s protocol. Data in a uniform format were distributed to all test sites and official data analysis sites for independent study. Data analysis. Data analyses were performed on either all of the 12,091 common genes or a subset of this group based on the qualitative detection call reported for each hybridization. The size of these subsets in each of the test sites for each sample type is reported as Table S6 in Supplementary Data online. Signal repeatability and reproducibility. The coefficient of variation (CV) of the signal or Cy3/Cy5 values (not log transformed) between the intrasite replicates (n r 5) was calculated for genes that were detected in at least three replicates of the same sample type within a test site. The distributions of these replicate CV values are displayed in Figure 1. The replicate CV medians from three test sites are included in Figure 2. A total CV (Fig. 2) of the signal values was calculated for all replicates across three test sites (n r 15) using the intersection of the generally detected gene lists (that is, genes detected in at least three replicates at all three sites). A global scaling normalization is inherently applied to data from the GE Healthcare and Agilent platforms, but is not part of data extraction and normalization on the Applied Biosystems, Affymetrix (using PLIER+16) and Illumina platforms. To account for these differences, Applied Biosystems, Affymetrix and Illumina provided scaling factors for each test site that were included when measuring the total CV. Concordance of detection call. Analyses were performed on all 12,091 common genes using the feature quality metrics provided by the manufacturers. All calls were resolved to a Detected or Not Detected status. Details on each platform’s method of determining qualitative calls are provided in Supplementary Data online. In general, the results are provided regarding the consistency of the resolved detection calls. If the call was missing because the microarray was absent, then the detection value was not considered. Otherwise, the qualitative call was considered, including those cases where the signal value was missing.


VOLUME 24

NUMBER 9

SEPTEMBER 2006

Gene list agreement. A list of differentially expressed genes was identified for each test site using the usual two group t-test that assumes equal variances between groups resulting in a pooled estimate of variance. This calculation is based on log signal. The criteria were P value o 0.001 and a mean difference greater than or equal to twofold. No filtering related to gene detection was performed. For each pair of test sites, the number of genes in both lists was identified. Percent overlap (Fig. 4) was calculated as the number of genes in common divided by number of genes on the list from one test site. For example, the agreement score for test site Y relative to test site X equals the number of genes on both lists divided by the number of genes on the test site Y list. Log ratio comparability. The log ratio of each gene is defined as the average of log signal for all sample B replicates minus the average of log signal of all sample A replicates. (This value is the equivalent of the log of the ratio of the geometric average of signal for all sample A replicates to the geometric average of signal for all sample B replicates.) Only genes that were detected in at least three sample A replicates and detected in at least three sample B replicates for both test sites were included. To detect compression or expansion (Fig. 5a), the slope (m) was calculated for each pair of test sites using orthogonal regression due to the potential measurement error in both sites. This analysis is based on the formula y ¼ mx + b, where y is the log ratio from test site Y and x is the log ratio from test site X. As the ideal slope is 1, the percent difference from ideal is simply m – 1. Comparability between a pair of test sites was also examined using Spearman rank correlations of the log ratios (Fig. 5b). This value compares the relative position of a gene in the test site X rank order of the log ratio (fold change) values against its position in test site Y rank order. Scatter plots of the log ratios from all sites against the log ratios generated with the TaqMan assays are presented in Figure 6. Accession numbers. All data are available through GEO (series accession number: GSE5350), ArrayExpress (accession number: E-TABM-132), ArrayTrack (http://www.fda.gov/nctr/science/centers/toxicoinformatics/Array Track/), and the MAQC web site (http://www.fda.gov/nctr/science/centers/ toxicoinformatics/maqc/). Note: Supplementary information is available on the Nature Biotechnology website.

ACKNOWLEDGMENTS All MAQC participants freely donated their time and reagents for the completion and analysis of the MAQC project. Participants from the National Institutes of Health (NIH) were supported by the Intramural Research Program of NIH, Bethesda, Maryland. D.H. thanks Ian Korf for BLAST discussions. This study utilized a number of computing resources, including the highperformance computational capabilities of the Biowulf PC/Linux cluster at the NIH (http://biowulf.nih.gov/) as well as resources at the analysis sites. DISCLAIMER This work includes contributions from, and was reviewed by, the FDA, the EPA and the NIH. This work has been approved for publication by these agencies, but it does not necessarily reflect official agency policy. Certain commercial materials and equipment are identified in order to adequately specify experimental procedures. In no case does such identification imply recommendation or endorsement by the FDA, the EPA or the NIH, nor does it imply that the items identified are necessarily the best available for the purpose. COMPETING INTERESTS STATEMENT The authors declare competing financial interests (see the Nature Biotechnology website for details). Published online at http://www.nature.com/naturebiotechnology/ Reprints and permissions information is available online at http://npg.nature.com/ reprintsandpermissions/ 1. Lesko, L.J. & Woodcock, J. Translation of pharmacogenomics and pharmacogenetics: a regulatory perspective. Nat. Rev. Drug Discov. 3, 763–769 (2004). 2. Frueh, F.W. Impact of microarray data quality on genomic data submissions to the FDA. Nat. Biotechnol. 24, 1105–1107 (2006). 3. Dix, D.J. et al. A framework for the use of genomics data at the EPA. Nat. Biotechnol. 24, 1108–1111 (2006).

1159


ARTICLES 4. Tan, P.K. et al. Evaluation of gene expression measurements from commercial microarray platforms. Nucleic Acids Res. 31, 5676–5684 (2003). 5. Ramalho-Santos, M., Yoon, S., Matsuzaki, Y., Mulligan, R.C. & Melton, D.A. ‘‘Stemness’’: transcriptional profiling of embryonic and adult stem cells. Science 298, 597–600 (2002). 6. Ivanova, N.B. et al. A stem cell molecular signature. Science 298, 601–604 (2002). 7. Miller, R.M. et al. Dysregulation of gene expression in the 1-methyl-4-phenyl-1,2,3,6tetrahydropyridine-lesioned mouse substantia nigra. J. Neurosci. 24, 7445–7454 (2004). 8. Fortunel, N.O. et al. Comment on ‘‘‘Stemness’: transcriptional profiling of embryonic and adult stem cells’’ and ‘‘a stem cell molecular signature’’. Science 302, 393 author reply 393 (2003). 9. Miklos, G.L. & Maleszka, R. Microarray reality checks in the context of a complex disease. Nat. Biotechnol. 22, 615–621 (2004). 10. Frantz, S. An array of problems. Nat. Rev. Drug Discov. 4, 362–363 (2005). 11. Marshall, E. Getting the noise out of gene arrays. Science 306, 630–631 (2004). 12. Michiels, S., Koscielny, S. & Hill, C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 365, 488–492 (2005). 13. Ein-Dor, L., Zuk, O. & Domany, E. Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer. Proc. Natl. Acad. Sci. USA 103, 5923–5928 (2006). 14. Petersen, D. et al. Three microarray platforms: an analysis of their concordance in profiling gene expression. BMC Genomics 6, 63 (2005). 15. Dobbin, K.K. et al. Interlaboratory comparability study of cancer gene expression analysis using oligonucleotide microarrays. Clin. Cancer Res. 11, 565–572 (2005). 16. Irizarry, R.A. et al. Multiple-laboratory comparison of microarray platforms. Nat. Methods 2, 345–350 (2005). 17. Larkin, J.E., Frank, B.C., Gavras, H., Sultana, R. & Quackenbush, J. Independence and reproducibility across microarray platforms. Nat. Methods 2, 337–344 (2005). 18. Kuo, W.P. et al. A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies. Nat. Biotechnol. 24, 832–840 (2006). 19. Shi, L. et al. QA/QC: challenges and pitfalls facing the microarray community and regulatory agencies. Expert Rev. Mol. Diagn. 4, 761–777 (2004). 20. Shi, L. et al. Cross-platform comparability of microarray technology: intra-platform consistency and appropriate data analysis procedures are essential. BMC Bioinformatics 6 Suppl. 2, S12 (2005). 21. Ji, H. & Davis, R.W. Data quality in genomics and microarrays. Nat. Biotechnol. 24, 1112–1113 (2006). 22. Canales, R.D. et al. Evaluation of DNA microarray results with quantitative gene expression platforms. Nat. Biotechnol. 24, 1115–1122 (2006). 23. Shippy, R. et al. Using RNA sample titrations to assess microarray platform performance and normalization techniques. Nat. Biotechnol. 24, 1123–1131 (2006).

24. Patterson, T.A. et al. Performance comparison of one-color and two-color platforms within the MicroArray Quality Control (MAQC) project. Nat. Biotechnol. 24, 1140– 1150 (2006). 25. Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006). 26. Guo, L. et al. Rat toxicogenomic study reveals analytical consistency across microarray platforms. Nat. Biotechnol. 24, 1162–1169 (2006). 27. Mecham, B.H. et al. Sequence-matched probes produce increased cross-platform consistency and more reproducible biological results in microarray-based gene expression measurements. Nucleic Acids Res. 32, e74 (2004). 28. Carter, S.L., Eklund, A.C., Mecham, B.H., Kohane, I.S. & Szallasi, Z. Redefinition of Affymetrix probe sets by sequence overlap with cDNA microarray probes reduces crossplatform inconsistencies in cancer-associated gene expression measurements. BMC Bioinformatics 6, 107 (2005). 29. Draghici, S., Khatri, P., Eklund, A.C. & Szallasi, Z. Reliability and reproducibility issues in DNA microarray measurements. Trends Genet. 22, 101–109 (2006). 30. Irizarry, R.A., Wu, Z. & Jaffee, H.A. Comparison of Affymetrix GeneChip expression measures. Bioinformatics 22, 789–794 (2006). 31. Pruitt, K.D., Tatusova, T. & Maglott, D.R. NCBI Reference Sequence (RefSeq): a curated non-redundant sequence database of genomes, transcripts and proteins. Nucleic Acids Res. 33, D501–D504 (2005). 32. Thierry-Mieg, D.& J, T.-M. AceView: a comprehensive cDNA-supported gene and transcripts annotation. Genome Biology 7, Suppl. 1, S12 (2006). 33. Bammler, T. et al. Standardizing global gene expression analysis between laboratories and across platforms. Nat. Methods 2, 351–356 (2005). 34. Harr, B. & Schlotterer, C. Comparison of algorithms for the analysis of Affymetrix microarray data as evaluated by co-expression of genes in known operons. Nucleic Acids Res. 34, e8 (2006). 35. Tusher, V.G., Tibshirani, R. & Chu, G. Significance analysis of microarrays applied to the ionizing radiation response. Proc. Natl. Acad. Sci. USA 98, 5116–5121 (2001). 36. Thompson, K.L. et al. Use of a mixed tissue RNA design for performance assessments on multiple microarray formats. Nucleic Acids Res. 33, e187 (2005). 37. Baker, S.C. et al. The External RNA Controls Consortium: a progress report. Nat. Methods 2, 731–734 (2005). 38. Reid, L.H. The value of a proficiency testing program to monitor performance in microarray laboratories. Pharm. Discov. 5, 20–25 (2005). 39. Ball, C.A. et al. Standards for microarray data. Science 298, 539 (2002). 40. Tong, W. et al. ArrayTrack–supporting toxicogenomic research at the U.S. Food and Drug Administration National Center for Toxicological Research. Environ. Health Perspect. 111, 1819–1826 (2003). 41. Tong, W. et al. Development of public toxicogenomics software for microarray data management and analysis. Mutat. Res. 549, 241–253 (2004).

AUTHORS The following authors contributed to project leadership: Leming Shi1, Laura H Reid2, Wendell D Jones2, Richard Shippy3, Janet A Warrington4, Shawn C Baker5, Patrick J Collins6, Francoise de Longueville7, Ernest S Kawasaki8, Kathleen Y Lee9, Yuling Luo10, Yongming Andrew Sun9, James C Willey11, Robert A Setterquist12, Gavin M Fischer13, Weida Tong1, Yvonne P Dragan1, David J Dix14, Felix W Frueh15, Federico M Goodsaid15, Damir Herman16, Roderick V Jensen17, Charles D Johnson18, Edward K Lobenhofer19, Raj K Puri20, Uwe Scherf21, Jean Thierry-Mieg16, Charles Wang22, Mike Wilson12,18, Paul K Wolber6, Lu Zhang9,23, William Slikker, Jr1, Leming Shi1, Laura H Reid2 Project leader: Leming Shi1 Manuscript preparation team leader: Laura H Reid2 MAQC Consortium: Leming Shi1, Laura H Reid2, Wendell D Jones2, Richard Shippy3, Janet A Warrington4, Shawn C Baker5, Patrick J Collins6, Francoise de Longueville7, Ernest S Kawasaki8, Kathleen Y Lee9, Yuling Luo10, Yongming Andrew Sun9, James C Willey11, Robert A Setterquist12, Gavin M Fischer13, Weida Tong1, Yvonne P Dragan1, David J Dix14, Felix W Frueh15, Federico M Goodsaid15, Damir Herman16, Roderick V Jensen17, Charles D Johnson18, Edward K Lobenhofer19, Raj K Puri20, Uwe Scherf21, Jean Thierry-Mieg16, Charles Wang22, Mike Wilson12,18, Paul K Wolber6, Lu Zhang9,23, Shashi Amur15, Wenjun Bao24, Catalin C Barbacioru9, Anne Bergstrom Lucas6, Vincent Bertholet7, Cecilie Boysen25, Bud Bromley25, Donna Brown26, Alan Brunner3, Roger Canales9, Xiaoxi Megan Cao27, Thomas A Cebula28, James J Chen1, Jing Cheng29, Tzu-Ming Chu24, Eugene Chudin5, John Corson6, J Christopher Corton14, Lisa J Croner30, Christopher Davies4, Timothy S Davison18, Glenda Delenstarr6, Xutao Deng22, David Dorris12, Aron C Eklund17, Xiao-hui Fan1, Hong Fang27, Stephanie Fulmer-Smentek6, James C Fuscoe1, Kathryn Gallagher31, Weigong Ge1, Lei Guo1, Xu Guo4, Janet Hager32, Paul K Haje33, Jing Han20, Tao Han1, Heather C Harbottle34, Stephen C Harris1, Eli Hatchwell35, Craig A Hauser36, Susan Hester14, Huixiao Hong27, Patrick Hurban19, Scott A Jackson28, Hanlee Ji37, Charles R Knight38, Winston P Kuo39, J Eugene LeClerc28, Shawn Levy40, Quan-Zhen Li41, Chunmei Liu4, Ying Liu42, Michael J Lombardi17, Yunqing Ma10, Scott R Magnuson43, Botoul Maqsodi10, Tim McDaniel4, Nan Mei1, Ola Myklebost44, Baitang Ning1, Natalia Novoradovskaya13, Michael S Orr15, Terry W Osborn38, Adam Papallo17, Tucker A Patterson1, Roger G Perkins27, Elizabeth H Peters38, Ron Peterson45, Kenneth L Philips19, P Scott Pine15, Lajos Pusztai46, Feng Qian27, Hongzu Ren14, Mitch Rosen14, Barry A Rosenzweig15, Raymond R Samaha9, Mark Schena33, Gary P Schroth23, Svetlana Shchegrova6, Dave D Smith47, Frank Staedtler45, Zhenqiang Su1, Hongmei Sun27, Zoltan Szallasi48, Zivana Tezak21, Danielle Thierry-Mieg16, Karol L Thompson15, Irina Tikhonova32, Yaron Turpaz4, Beena Vallanat14, Christophe Van7,

1160

VOLUME 24

NUMBER 9

SEPTEMBER 2006


ARTICLES Stephen J Walker49, Sue Jane Wang15, Yonghong Wang8, Russ Wolfinger24, Alex Wong6, Jie Wu27, Chunlin Xiao9, Qian Xie27, Jun Xu22, Wen Yang10, Liang Zhang29, Sheng Zhong50, Yaping Zong51, William Slikker, Jr1 Scientific management (National Center for Toxicological Research, US Food and Drug Administration): Leming Shi, Weida Tong, Yvonne P. Dragan, William Slikker, Jr. Affiliations:


1National

Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Road, Jefferson, Arkansas 72079, USA; 2Expression Analysis, Inc., 2605 Meridian Parkway, Durham, North Carolina 27713, USA; 3GE Healthcare, 7700 S. River Parkway, Suite 2603, Tempe, AZ 85284, USA; 4Affymetrix, Inc., 3420 Central Expressway, Santa Clara, California 95051, USA; 5Illumina,Inc. 9885 Towne Centre Drive, San Diego, California 92121, USA; 6Agilent Technologies, Inc., 5301 Stevens Creek Blvd., Santa Clara, California 95051, USA; 7Eppendorf Array Technologies, rue du Se´minaire 20a, 5000 Namur, Belgium; 8NCI Advanced Technology Center, 8717 Grovemont Circle, Bethesda, Maryland 20892, USA; 9Applied Biosystems, 850 Lincoln Centre Drive, Foster City, California 94404, USA; 10Panomics, Inc., 6519 Dumbarton Circle, Fremont, California 94555, USA; 11Medical University of Ohio, 3000 Arlington Avenue, Toledo, Ohio 43614, USA; 12Ambion, An Applied Biosystems Business, 2130 Woodward Street, Austin, Texas 78744, USA; 13Stratagene Corp., 11011 North Torrey Pines Road, La Jolla, California 92130, USA; 14Office of Research and Development, US Environmental Protection Agency, 109 TW Alexander Drive, Research Triangle Park, North Carolina 27711, USA; 15Center for Drug Evaluation and Research, US Food and Drug Administration, 10903 New Hampshire Avenue, Silver Spring, Maryland 20993, USA; 16National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, Maryland 20894, USA; 17University of Massachusetts-Boston, 100 Morrissey Boulevard, Boston, Massachusetts 02125, USA; 18Asuragen, Inc., 2150 Woodward, Austin, Texas 78744, USA; 19CogenicsTM, A Division of Clinical Data, Inc., 100 Perimeter Park Drive, Suite C, Morrisville, North Carolina 27560, USA; 20Center for Biologics Evaluation and Research, US Food and Drug Administration, 29 Lincoln Drive, Bethesda, Maryland 20892, USA; 21Center for Devices and Radiological Health, US Food and Drug Administration, 2098 Gaither Road, Rockville, Maryland 20850, USA; 22UCLA David Geffen School of Medicine, Transcriptional Genomics Core, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048, USA; 23Solexa, Inc., 25861 Industrial Boulevard, Hayward, California 94545, USA; 24SAS Institute, Inc., 100 SAS Campus Drive, Cary, North Carolina 27513, USA; 25Vialogy Corp., 2400 Lincoln Avenue, Altadena, California 91001, USA; 26Operon Biotechnologies, 2211 Seminole Drive, Huntsville, Alabama 35805, USA; 27Z-Tech Corp., 3900 NCTR Road, Jefferson, Arkansas 72079, USA; 28Center for Food Safety and Applied Nutrition, US Food and Drug Administration, 8401 Muirkirk Road, Laurel, Maryland 20708, USA; 29CapitalBio Corp., 18 Life Science Parkway, Changping District, Beijing 102206, China; 30Biogen Idec, 5200 Research Place, San Diego, California 92122, USA; 31US Environmental Protection Agency, Office of the Science Advisor, 1200 Pennsylvania Avenue, NW, Washington, DC 20460, USA; 32Yale University, W.M. Keck Biotechnology Resource Laboratory, Microarray Resource, 300 George Street, New Haven, Connecticut 06511, USA; 33TeleChem ArrayIt, 524 E. Weddell Drive, Sunnyvale, California 94089, USA; 34Center for Veterinary Medicine, US Food and Drug Administration, 8401 Muirkirk Road, Laurel, Maryland 20708, USA; 35Cold Spring Harbor Laboratory, 500 Sunnyside Boulevard, Woodbury, New York 11797, USA; 36Burnham Institute, 10901 North Torrey Pines Road, La Jolla, California 92037, USA; 37Stanford University School of Medicine, 318 Campus Drive, Stanford, California 94305, USA; 38Gene Express, Inc., 975 Research Drive, Toledo, Ohio 43614, USA; 39Harvard School of Dental Medicine, Department of Developmental Biology, 188 Longwood Avenue, Boston, Massachusetts 02115, USA; 40Vanderbilt University, 465 21st Avenue South, Nashville, Tennessee 37232, USA; 41University Texas Southwestern Medical Center, 6000 Harry Hines Boulevard/ND6.504, Dallas, Texas 75390, USA; 42University of Texas at Dallas, Department of Computer Science, MS EC31 Richardson, Texas 75083, USA; 43GenUs BioSystems, Inc., 1808 Janke Drive Unit M, Northbrook, Illinois 60062, USA; 44Norwegian Microarray Consortium, Rikshospitalet - Radiumhospitalet Health Centre, Montebello, N0310 Oslo, Norway; 45Novartis, 250 Massachusetts Avenue, Cambridge, Massachusetts 02139, USA; 46MD Anderson Cancer Center, Breast Medical Oncology Department-Unit 1354, 1155 Pressler Street, Houston, Texas 77230, USA; 47Luminex Corp., 12212 Technology Boulevard, Austin, Texas 78727, USA; 48Harvard Medical School, Children’s Hospital Informatics Program at the Harvard-MIT Division of Health Sciences and Technology (CHIP@HST), Boston, Massachusetts 02115, USA; 49Wake Forest University School of Medicine, Department of Physiology and Pharmacology, Medical Center Boulevard, Winston-Salem, North Carolina 27157, USA; 50University of Illinois at Urbana-Champaign, Department of Bioengineering, 1304 W. Springfield Avenue, Urbana, Illinois 61801, USA; 51Full Moon Biosystems, Inc., 754 N. Pastoria Avenue, Sunnyvale, California 94085, USA.


VOLUME 24

NUMBER 9

SEPTEMBER 2006

1161


ARTICLES

Rat toxicogenomic study reveals analytical consistency across microarray platforms Lei Guo1, Edward K Lobenhofer2, Charles Wang3, Richard Shippy4, Stephen C Harris1, Lu Zhang5, Nan Mei1, Tao Chen1, Damir Herman6, Federico M Goodsaid7, Patrick Hurban2, Kenneth L Phillips2, Jun Xu3, Xutao Deng3, Yongming Andrew Sun8, Weida Tong1, Yvonne P Dragan1 & Leming Shi1 To validate and extend the findings of the MicroArray Quality Control (MAQC) project, a biologically relevant toxicogenomics data set was generated using 36 RNA samples from rats treated with three chemicals (aristolochic acid, riddelliine and comfrey) and each sample was hybridized to four microarray platforms. The MAQC project assessed concordance in intersite and cross-platform comparisons and the impact of gene selection methods on the reproducibility of profiling data in terms of differentially expressed genes using distinct reference RNA samples. The real-world toxicogenomic data set reported here showed high concordance in intersite and cross-platform comparisons. Further, gene lists generated by fold-change ranking were more reproducible than those obtained by t-test P value or Significance Analysis of Microarrays. Finally, gene lists generated by fold-change ranking with a nonstringent P-value cutoff showed increased consistency in Gene Ontology terms and pathways, and hence the biological impact of chemical exposure could be reliably deduced from all platforms analyzed.

To validate and extend the findings of the MAQC project1, described elsewhere in this issue, we generated a toxicogenomics data set using a rat chemical exposure study. One of the objectives of the MAQC project was to assess the reproducibility of gene expression profiling data across laboratories and platforms. Analysis of the MAQC data set shows the high reproducibility of microarray data under well-controlled conditions and further indicates that the criteria used to define differentially expressed genes can have a dramatic impact on the overlap of the resulting gene lists. In particular, lists of differentially expressed genes generated using fold change, rather than t-test P value for gene selection have been previously proposed to be more reproducible1,2. The two RNA samples used in the MAQC project were reference samples with no explicit biological connection: the Stratagene Universal Human Reference RNA (comprised of RNA from ten different cell lines) and Ambion Human Brain Reference RNA1. The availability of these data provides an invaluable resource for benchmarking laboratory performance and for testing and validating new procedures, equipment and reagents, for example. Although data from these reference samples address technical performance and reproducibility of results from microarray technology, they cannot address whether microarray data from different laboratories or platforms would result in the same biological interpretation of real-world

samples. We therefore sought to apply the findings of the MAQC study to a set of experimental toxicogenomic data to validate the approach. Several recent publications have investigated the genotoxicity of three botanical carcinogens: aristolochic acid, riddelliine and comfrey3–6. In the present study, 36 RNAs were isolated from the kidney and/or liver of rats exposed to one of these compounds or a control group. To corroborate the findings of the MAQC project and to determine whether the same biological interpretations would result from cross-platform comparisons, we hybridized these samples to four commercially available platforms (Affymetrix, Agilent, Applied Biosystems and GE Healthcare). To address intersite performance, we used the Affymetrix platform at two different test sites. The results from this study are consistent with those of the MAQC project in that good concordance is found between data generated at different sites, as well as from different platforms. Furthermore, when fold-change ranking is used as the primary criterion for selecting differentially expressed genes, the overlap between gene lists from different laboratories using either the same or different platforms is high. In contrast, when a t-statistic (P-value) ranking is used as the primary criterion the cross-site or cross-platform overlap is substantially lower1,2. The selection criteria for differential expression can thus affect both the apparent reproducibility of microarray data, as well as

1National Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Road, Jefferson, Arkansas 72079, USA; 2Cogenics, A Division of Clinical Data, 100 Perimeter Park Drive, Suite C, Morrisville, North Carolina 27560, USA; 3UCLA David Geffen School of Medicine, Transcriptional Genomics Core, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048, USA; 4GE Healthcare, 7700 S. River Parkway, Suite #2603, Tempe, Arizona 85284, USA; 5Solexa, 25861 Industrial Boulevard, Hayward, California 94545, USA; 6National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, Maryland 20894, USA; 7Center for Drug Evaluation and Research, US Food and Drug Administration, 10903 New Hampshire Avenue, Silver Spring, Maryland 20993, USA; 8Applied Biosystems, 850 Lincoln Centre Drive, Foster City, California 94404, USA. Correspondence should be addressed to L.G. ([email protected]) or L.S. ([email protected]).

Received 5 June; accepted 18 July; published online 8 September 2006; doi:10.1038/nbt1238

1162

VOLUME 24

NUMBER 9

SEPTEMBER 2006


ARTICLES

b

d

AG1

GEH

GEH_K_AA_4 GEH_K_AA_6 GEH_K_AA_3 GEH_K_AA_5 GEH_K_AA_1 GEH_K_AA_2 GEH_K_CTR_4 GEH_K_CTR_6 GEH_K_CTR_2 GEH_K_CTR_3 GEH_K_CTR_1 GEH_K_CTR_5 GEH_L_RDL_1 GEH_L_RDL_6 GEH_L_RDL_3 GEH_L_RDL_5 GEH_L_RDL_2 GEH_L_RDL_4 GEH_L_CTR_4 GEH_L_AA_4 GEH_L_AA_3 GEH_L_AA_1 GEH_L_AA_2 GEH_L_AA_6 GEH_L_CTR_1 GEH_L_CTR_2 GEH_L_CTR_3 GEH_L_CTR_5 GEH_L_CTR_6 GEH_L_AA_5 GEH_L_CFY_5 GEH_L_CFY_4 GEH_L_CFY_1 GEH_L_CFY_3 GEH_L_CFY_2 GEH_L_CFY_6

c

AFX

AFX_L_CFY_4 AFX_L_CFY_6 AFX_L_CFY_2 AFX_L_CFY_3 AFX_L_CFY_1 AFX_L_CFY_5 AFX_L_AA_5 AFX_L_CTR_6 AFX_L_AA_4 AFX_L_AA_1 AFX_L_AA_3 AFX_L_AA_2 AFX_L_AA_6 AFX_L_CTR_1 AFX_L_CTR_3 AFX_L_CTR_4 AFX_L_CTR_2 AFX_L_CTR_5 AFX_L_RDL_1 AFX_L_RDL_2 AFX_L_RDL_4 AFX_L_RDL_3 AFX_L_RDL_5 F AFX_L_RDL_6 AFX_K_AA_4 AFX_K_CTR_4 AFX_K_CTR_2 AFX_K_CTR_1 AFX_K_CTR_3 AFX_K_CTR_5 AFX_K_CTR_6 AFX_K_AA_6 AFX_K_AA_3 AFX_K_AA_5 AFX_K_AA_1 AFX_K_AA_2

ABI_K_AA_4 ABI_K_AA_1 ABI_K_AA_2 ABI_K_AA_3 ABI_K_AA_6 ABI_K_AA_5 ABI_K_CTR_2 ABI_K_CTR_3 ABI_K_CTR_1 ABI_K_CTR_5 ABI_K_CTR_4 ABI_K_CTR_6 ABI_L_CFY_4 ABI_L_CFY_1 ABI_L_CFY_5 ABI_L_CFY_6 ABI_L_CFY_2 ABI_L_CFY_3 ABI_L_CTR_6 ABI_L_RDL_1 ABI_L_RDL_2 ABI_L_RDL_4 ABI_L_RDL_5 ABI_L_RDL_3 ABI_L_RDL_6 ABI_L_CTR_5 ABI_L_CTR_2 ABI_L_CTR_1 ABI_L_CTR_3 ABI_L_CTR_4 ABI_L_AA_6 ABI_L_AA_2 ABI_L_AA_1 ABI_L_AA_3 ABI_L_AA_4 ABI_L_AA_5

ABI

AG1_K_AA_6 AG1_K_CTR_3 AG1_K_CTR_4 AG1_K_CTR_5 AG1_K_CTR_1 AG1_K_CTR_2 AG1_K_CTR_6 AG1_K_AA_4 AG1_K_AA_3 AG1_K_AA_5 AG1_K_AA_1 AG1_K_AA_2 AG1_L_CFY_1 AG1_L_CFY_2 AG1_L_CFY_4 AG1_L_CFY_5 AG1_L_CFY_3 AG1_L_CFY_6 AG1_L_AA_2 AG1_L_RDL_2 AG1_L_RDL_6 AG1_L_RDL_3 AG1_L_RDL_4 AG1_L_RDL_1 AG1_L_RDL_5 AG1_L_AA_5 AG1_L_AA_6 AG1_L_CTR_5 AG1_L_AA_4 AG1_L_CTR_2 AG1_L_CTR_4 AG1_L_AA_1 AG1_L_AA_3 AG1_L_CTR_6 AG1_L_CTR_1 AG1_L_CTR_3


a

Figure 1 Hierarchical clustering of platform-specific microarray data separates samples by tissue and treatment. For each platform, the log2 intensity data from all 36 microarrays after filtering for genes flagged as below the detection level were hierarchically clustered using an average linkage algorithm and Euclidean distance as the distance metric. (a) Data from the Applied Biosystems platform (ABI). (b) Affymetrix site 1(AFX). (c) Agilent (AG1). (d) GE Healthcare (GEH). The sample labels are colored based on treatment/tissue group. Black, control kidney; purple, aristolochic acid–treated kidney; blue, control liver; red, aristolochic acid–treated liver; orange, riddelliine-treated liver; green, comfrey-treated liver.

the biological interpretation of the data. By using fold-change ranking plus a nonstringent P-value cutoff, the overlap of differentially expressed gene lists is increased, leading to improved agreement of the biological interpretation of the data in terms of enriched Gene Ontology (GO) nodes and pathways. Furthermore, data generated by this approach led to novel biological findings concerning chemical exposure. These findings are reproducible across laboratories and platforms when the preferred gene selection criteria are used. Together, these results further support the findings of the MAQC project, highlight the importance of appropriate data analysis procedures and demonstrate that microarray data generated from different platforms not only result in similar biological interpretation, but also reveal novel findings. RESULTS RNA was isolated from the target organs of rats exposed to aristolochic acid, riddelliine or comfrey, from studies that have been detailed previously3–6. In total there were six treatment/tissue groups: kidney from aristolochic acid–treated rats, kidney from vehicle

control, liver from aristolochic acid–treated rats, liver from riddelliine-treated rats, liver from comfrey-treated rats and liver from vehicle control. Within each treatment/tissue group there were six biological replicates. Aliquots of these samples were prepared and distributed to each of the test sites for gene expression profiling using microarrays from four different platforms. Laboratory procedures were identical to those in the MAQC project1. Unless otherwise stated, the platform manufacturer’s recommendations were used for data processing. Hierarchical clustering analysis To assess the overall reproducibility of microarray data from the four platforms, we performed hierarchical clustering analyses for each platform. Within each platform, samples were largely clustered first by tissue type and then by treatment (Fig. 1). Within each platform there are individual samples that did not cluster with the other members of their respective treatment/tissue group; however, the only sample that was consistently different across all platforms was sample no. 4 from the aristolochic acid–treated kidney

Table 1 Average Pearson correlation coefficients of log2-normalized intensity data for each treatment/tissue group No. of Probe(set)s

Aristolochic acid kidneya

Control kidney

Aristolochic acid liver

Comfrey liver

Control liver

Riddelliine liver

Applied Biosystems (ABI)

26,857

0.9586 (0.9623)

0.9742

0.9636

0.9737

0.9634

0.9705

Affymetrix no. 1 (AFX) Affymetrix no. 2 (AFX2)

31,099 31,099

0.9748 (0.9828) 0.9736 (0.9818)

0.9881 0.9879

0.9871 0.9860

0.9861 0.9827

0.9876 0.9862

0.9867 0.9836

Agilent (AG1) GE Healthcare (GEH)

41,071 35,129

0.9610 (0.9711) 0.9697 (0.9739)

0.9701 0.9761

0.9642 0.9690

0.9659 0.9690

0.9740 0.9687

0.9675 0.9734

Test site

aNumbers

in parentheses represent data after excluding sample no. 4.


VOLUME 24

NUMBER 9

SEPTEMBER 2006

1163

ARTICLES

Kidney, aristolochic acid Liver, comfrey Liver, aristolochic acid Liver, riddelliine


group. Similar results were obtained using principal components analysis (Supplementary Fig. 1 online). DNA adduct data indicate that this sample had 50% fewer DNA adducts compared to the other five animals in the same treatment group (Mei, N. et al., ABI-K-AA-1 ABI-K-AA-4 AFX-K-AA-4 AFX2-K-AA-4 AG1-K-AA-4 GEH-K-AA-4 AFX-K-AA-1 AFX2-K-AA-1 AFX-K-AA-2 AFX2-K-AA-2 AFX-K-AA-3 AFX2-K-AA-3 AFX-K-AA-5 AFX2-K-AA-5 AFX-K-AA-6 AFX2-K-AA-6 AG1-K-AA-1 AG1-K-AA-2 AG1-K-AA-3 AG1-K-AA-5 GEH-K-AA-1 GEH-K-AA-2 GEH-K-AA-3 GEH-K-AA-5 GEH-K-AA-6 AGI-K-AA-6 ABI-K-AA-2 ABI-K-AA-3 ABI-K-AA-5 ABI-K-AA-6 ABI-CFY-1 ABI-CFY-2 ABI-CFY-3 ABI-CFY-6 ABI-CFY-4 ABI-CFY-5 AFX-CFY-1 AFX-CFY-2 AFX-CFY-3 AFX-CFY-6 AFX2-CFY-6 AFX2-CFY-3 AFX2-CFY-1 AFX2-CFY-2 AFX-CFY-5 AFX2-CFY-5 AFX-CFY-4 AFX2-CFY-4 AG1-CFY-1 AG1-CFY-2 AG1-CFY-3 AG1-CFY-6 AG1-CFY-4 AG1-CFY-5 GEH-CFY-1 GEH-CFY-2 GEH-CFY-3 GEH-CFY-6 GEH-CFY-4 GEH-CFY-5 ABI-AA-1 ABI-AA-2 ABI-AA-3 ABI-AA-4 ABI-AA-6 ABI-AA-5 AFX-AA-1 AFX-AA-3 AFX-AA-2 AFX-AA-4 AFX2-AA-4 AFX2-AA-1 AFX2-AA-2 AFX2-AA-3 AFX2-AA-6 AFX-AA-6 AG1-AA-1 AG1-AA-3 AG1-AA-4 GEH-AA-1 GEH-AA-2 GEH-AA-4 GEH-AA-3 AFX-AA-5 AFX2-AA-5 AG1-AA-5 GEH-AA-5 AG1-AA-6 AG1-AA-2 GEH-AA-6 ABI-RDL-1 ABI-RDL-3 ABI-RDL-6 ABI-RDL-2 ABI-RDL-4 ABI-RDL-5 AFX-RDL-1 AFX2-RDL-1 AFX-RDL-2 AFX2-RDL-2 AFX2-RDL-4 AFX-RDL-4 AFX-RDL-3 AFX-RDL-6 AFX2-RDL-3 AFX-RDL-5 AFX2-RDL-5 AFX2-RDL-6 AG1-RDL-2 AG1-RDL-3 AG1-RDL-6 AG1-RDL-1 AG1-RDL-5 GEH-RDL-1 GEH-RDL-2 GEH-RDL-4 GEH-RDL-3 GEH-RDL-5 GEH-RDL-6 AG1-RDL-4

1164

unpublished data), suggesting that the consistent failure of this sample to cluster with its treatment/tissue group may be biologically based. It was also determined that aristolochic acid–treated liver samples showed a relatively small difference in expression profiles when compared to their tissue-matched control group. This result was reproduced across all platforms and is consistent with previous observations that kidney, not liver, is the target organ of aristolochic acid–mediated carcinogenesis7. The reproducibility of the microarray data was further explored by calculating the Pearson correlation coefficients of the log2 intensity data for all pair-wise sample comparisons within a treatment/tissue group for each platform. Table 1 shows the average correlation of biological replicates within each treatment/tissue group for each platform and further demonstrates the high degree of similarity of these data. Because of the presence of an animal that had a diminished treatment response, the aristolochic acid–treated kidney group had a significantly lower correlation, as expected, compared to other groups (e.g., P ¼ 0.0024, two-sided, paired t-test compared to the control kidney group). Removal of sample no. 4 from the aristolochic acid–treated kidney group resulted in a less significant difference (P ¼ 0.085, two-sided, paired t-test compared to the control kidney group). These data coupled with the DNA adduct data consistently indicate that sample no. 4 from the aristolochic acid–treated kidney group has a different response relative to the other group members. Therefore, for the assessment of cross-platform data consistency, the data from this sample have been excluded. Overlap of differentially expressed gene lists across sites One of the fundamental goals of a gene expression profiling experiment is to identify those genes that are differentially expressed within the system being studied. There are a large number of methods for selecting such genes, and ultimately, the genes that are identified have a fundamental impact on the biological interpretation of the data. Therefore, this toxicogenomics study was used to validate the findings in regard to gene selection methods by employing different selection criteria and determining the percentage of overlap between different laboratories or platforms1,2. The overlap across the two sites that generated data using the Affymetrix platform is high (85–90%) when the genes (from a few up to B2,000) are selected by rank ordering the genes based on fold change (Supplementary Fig. 2 online). As more genes are considered differentially expressed (that is, moving to the right on the x-axis) the percentage of overlap begins to decline because of the inclusion of more genes demonstrating smaller fold changes, which are less likely to be reproducible across sites. There is a small decrease in the overlap when a P-value cutoff of 0.01 or 0.05 is applied to the fold change–based,

Figure 2 Hierarchical clustering of all individual sample data from all microarray platforms separated by tissue and treatment. Within each platform/site, a fold change was calculated and log2 transformed for all 5,112 common genes that did not have any missing values (n ¼ 4,609) for each of the 24-treated individual samples compared to a tissue-match control. These values were then hierarchically clustered using Euclidean distance metric and average linkage. Each row represents the results from an individual treated animal assayed on a particular platform. Each row is labeled with a platform designation first, followed by the organ assayed for kidney samples, and then the treatment and unique animal identifier (1–6). ABI, Applied Biosystems platform; AFX, Affymetrix site 1; AFX2, Affymetrix site 2; AG1, Agilent; and GEH, GE Healthcare. K, kidney; AA, aristolochic acid; RDL, riddelliine; and CFY, comfrey. The yellow boxes highlight areas in which replicates of the same sample across all multiple platform and/or sites have clustered together.

VOLUME 24

NUMBER 9

SEPTEMBER 2006


high (490%). Furthermore, global scaling methods do not alter the rank order of genes based on fold change (hence the gene lists); therefore, the overlap between raw, mean-, or median-scaled data is 100% when using the fold change for ranking and selecting genes. However, these scaling factors can affect the magnitude of the fold changes and the P values and thus will only affect the gene list when a P-value criterion is involved in gene selection. Our results are consistent with those reported elsewhere8. In addition to the standard t-test, numerous different statistical tests have been used for the identification of differentially expressed genes9. One commonly used method is Significance Analysis of Microarrays (SAM)10. Supplementary Figure 4 online illustrates the intersite concordance results of differentially expressed genes selected based on fold-change ranking, SAM, t-test and random selection when the data from the comfrey-treated liver samples are compared to their corresponding controls. The site-site concordance based on SAM was clearly improved over that based on a simple t-test, but did not achieve the same level of concordance as that reached based on foldchange ranking. Similar results were obtained when other sample pairs or cross-platform data were analyzed in the same manner (data not shown). Cumulatively, these results illustrate that fold change–based

gene-selection methods. This results from the P-value threshold altering the composition of the total list of genes such that each test site has a different list of genes to begin with in the gene selection process, thereby increasing the intersite inconsistency. Supplementary Figure 2 also illustrates the overlap when genes are selected based on P-value rank ordering alone or with a fold-change criterion of 2.0 or 1.4. For P value–based gene-selection methods, the overlap gradually increases as the number of differentially expressed genes increases. An increase in the overlap is also observed when a fold-change cutoff of 1.4 or 2.0 is applied in conjunction with the P-value criterion. This is understandable since the larger fold changes are more easily reproduced than smaller ones. The impact of different normalization methods on the overlap of gene lists was also assessed by comparing the overlap of gene lists derived from two normalization methods using the same gene selection method on the same sample pair comparison from data generated at the same test site (Supplementary Fig. 3 online). When P value is used as the criterion for gene selection, the overlap from different normalization methods is relatively low. However, when genes are ranked and selected based on fold change with or without a P-value cutoff, the overlap between different normalization methods is very

Intralaboratory concordance (%)

a

100

ABI

b 100

AFX

c 100

90

90

90

80

80

80

70

70

70

60

60

60

50

50

50

40

40

40

30

30

30

20

20

20

10

10

10

0

0 1

10

100

1,000

d 100

AG1

100

e 100 90

80

80

70

70

60

60

50

50

40

40

30

30

20

20

10

10

0

10

1,000

10,000

Number of genes selected as differentially expressed

90

AFX2

0 1

10,000


Intralaboratory concordance (%)


ARTICLES

1

10

100

1,000

10,000


GEH

Fold-change rank ordering Fold-change rank ordering/P-value 1.4 P -value rank ordering

0 1

10 100 1,000 10,000 Number of genes selected as differentially expressed

1

10 100 1,000 10,000 Number of genes selected as differentially expressed

Figure 3 Intralaboratory overlap of differentially expressed gene lists generated using different selection criteria. For each platform, the liver control and comfrey treatment groups were equally and randomly divided into two experiments and the differentially expressed genes were identified independently from the two experiments using different gene selection criteria. Differentially expressed genes were selected from a subset of genes that are detectable by both experiments. The x-axis represents the number of genes selected as differentially expressed, and the y-axis represents the overlap (%) of two gene lists for a given number of differentially expressed genes. Each line on the graph represents the intralaboratory overlap of differentially expressed gene lists based on one of six different gene ranking/selection methods. Red, fold-change rank ordering only; orange, P-value rank ordering only; light green, fold-change rank ordering and P o 0.01; blue, fold-change rank ordering and P o 0.05; teal, P-value rank ordering and fold change 41.4; and purple, P-value rank ordering and fold change 42.0. (a) Applied Biosystems (ABI). (b) Affymetrix site 1 (AFX). (c) Affymetrix site 2 (AFX2). (d) Agilent (AG1). (e) GE Healthcare (GEH).


VOLUME 24

NUMBER 9

SEPTEMBER 2006

1165

20

40 60 80 100 120 140 Number of GO terms

100 90 80 70 60 50 40 30 20 10 0

c

AFX_FC (P < 0.05) AFX_P

0

20


100 90 80 70 60 50 40 30 20 10 0

AFX2_FC (P < 0.05) AFX2_P

0

20

d

Figure 4 Intralaboratory overlap of enriched GO terms. The control and treatment groups were equally and randomly divided into two experiments. From each experiment, the top 200 genes based on either a fold change (blue line) or P value (pink line) ranking were selected. The GO terms associated with these genes were then rank ordered and the overlap between the two experiments was identified and graphed to compare the percentage of overlap (y-axis) against the total number of GO terms present in both experiments. The results depicted are derived from the comfrey-treated comparisons for each platform, but similar results were generated with the other treatment comparisons. (a) Applied Biosystems (ABI). (b) Affymetrix site 1 (AFX). (c) Affymetrix site 2 (AFX2). (d) Agilent (AG1). (e) GE Healthcare (GEH).

Overlap of differentially expressed gene lists across platforms To assess the reproducibility of data across multiple microarray platforms, we identified the list of genes that was measured by all four of the microarray platforms using the March 2006 version of the RefSeq database and the methods described by the MAQC project1. This resulted in the identification of 5,112 common genes, which were used in all subsequent cross-platform comparisons. Consistent with results from intersite comparisons (Supplementary Fig. 2 online), the crossplatform data comparisons reveal the same trends. Specifically, the percentage of overlap for differentially expressed gene lists is highest when fold change–based gene selection methods are used (Supplementary Fig. 5a online). Not surprisingly, the cross-platform overlap is higher (B80%) in all instances when genes that are not reproducibly detected on the microarrays are omitted (e.g., those probes that are flagged as ‘not present’) (Supplementary Fig. 5b online). These results combined with intersite results further corroborate the findings of the MAQC project that fold change–based selection criteria for differentially expressed genes generate more reproducible results1,2. No measure of sensitivity or specificity of the approach was included in the analysis. Within each platform/site, the fold change was calculated for all 5,112 common genes that did not have any missing values (n ¼ 4,609) for each of the 24-treated individual samples compared to a tissue-match control and these values were then hierarchically clustered (Fig. 2). The resulting dendrogram illustrates that the samples are separated by tissue and then by treatment. Each of the four major branches of the dendrogram contain all of the biological replicate data for a given treatment/tissue group regardless of the site or platform that was used to generate the data. Within each of these branches, the platform as opposed to the biological replicate is the next major division. There are a few notable exceptions to this observation. When the same platform is performed at different test sites, the replicates of the same sample assayed at different sites cluster more closely together. In a few instances, the results from multiple different platforms for the same biological sample cluster together (e.g., aristolochic acid–treated liver sample no. 5). Because no gene selection criteria were used to generate this visualization, these results further indicate that interlaboratory and cross-platform data are highly reproducible.

1166

AG1_FC (P < 0.05) AG1_P

0

e

selection methods usually offer a higher level of consistency of lists of differentially expressed genes.

100 90 80 70 60 50 40 30 20 10 0


Common GO terms (%)


0

b

Common GO terms (%)

ABI_FC (P < 0.05) ABI_P

Common GO terms (%)

100 90 80 70 60 50 40 30 20 10 0

Common GO terms (%)

a

Common GO terms (%)

ARTICLES

20


100 90 80 70 60 50 40 30 20 10 0

GEH_FC (P < 0.05) GEH_P

0

20

40

60

80 100 120 140

Number of GO terms

Agreement of biological interpretation with GO and pathways Typically, a microarray-based experiment is performed in a single laboratory using a single platform. Furthermore, it is relatively common to use three biological replicates in a toxicogenomic study when multiple groups of samples are involved. To explore whether or not a similar biological response was obtained when comparing results within a given laboratory, we generated data from six biological replicates. The control and treatment groups were then equally and randomly divided into two artificial experiments. Consistent with the interlaboratory and cross-platform results, the overlap of differentially expressed genes using different gene selection criteria from the intra-laboratory results revealed the same trend, namely that fold change–based selection criteria generate more reproducible results (Fig. 3). For each of the ABI, AFX and AFX2 intralaboratory comparisons, the overlap of gene lists was almost identical with or without a P cutoff (o0.05) for up to B1,000 genes selected as differentially expressed; for AG1 and GEH, the use of a P cutoff (o0.05) slightly increased the overlap of gene lists. However, the use of a more stringent P cutoff (o0.01) decreased the overlap of gene lists. These intralaboratory comparison results are consistent with those of interlaboratory comparisons (Supplementary Fig. 2 online). Therefore, a modest P cutoff (o0.05) appeared to be reasonable for data sets of this small sample size (3). Furthermore, the use of a fold-change threshold increased the overlap of gene lists derived from P-value ranking; a more stringent fold-change threshold leads to higher overlap of gene lists (Fig. 3 and Supplementary Fig. 2 online). The differences in overlap of gene lists based on selection criteria were further investigated by assessing the impact on the associated GO terms. From each artificial experiment, the top 200 genes based on either a fold-change (with P o 0.05 cutoff) or P-value ranking were selected. The P value from the Fisher’s exact test was calculated for each GO term associated with these genes. For each artificial experiment, the GO terms were then rank-ordered based on the P value. The overlap between the two artificial experiments was determined by dividing the number of GO terms commonly meeting a P-value ranking criterion in both of the artificial experiments by the total number of GO terms meeting the P-value criterion for either experiment. Figure 4 illustrates the percentage of overlapping GO terms plotted against a defined number of the highest ranking GO terms from both experiments. Clearly, the overlap of GO terms was much higher when genes are selected by fold change compared

VOLUME 24

NUMBER 9

SEPTEMBER 2006



0

5

10

15

20

25

30

35

Number of KEGG pathways

0

5

10

15

20

25

30

c

35


100 90 80 70 60 50 40 30 20 10 0

AFX2_FC (P < 0.05) AFX2_P

0

5

10

15

20

25

30

d

35

Figure 5 Intralaboratory overlap of differentially enriched KEGG pathways. The control and treatment groups were equally and randomly divided into two experiments. From each experiment, the top 200 genes based on either a fold change (blue line) or P-value (pink line) ranking were selected. The KEGG pathways associated with these genes were then rank ordered and the overlap between the two experiments was identified and graphed to compare the percentage of overlap (y-axis) against the total number of KEGG pathways present in both experiments. The results depicted are derived from the comfrey-treated comparisons for each platform, but similar results were generated with the other treatment comparisons. (a) Applied Biosystems (ABI). (b) Affymetrix site 1 (AFX). (c) Affymetrix site 2 (AFX2). (d) Agilent (AG1). (e) GE Healthcare (GEH).

Agreement of biological response To further explore the agreement of biological response across the microarray platforms, we combined data from the cross-platform common gene list (5,112 genes) and from the six comfrey-treated liver samples and compared them to the data from the six control liver samples for each platform. A t-test was performed and genes with P o 0.05 were identified. This filtered gene set was then rank ordered by fold change and for each platform the top 250 up- and downregulated genes were selected, generating a list of the top 500 differentially expressed genes for each of the five platform/site combinations (the overlap in genes between the gene lists for any two platforms is 470%). A GO enrichment analysis was performed for each platform by comparing the content of the top 500 differentially expressed genes to the content of the 5,112 common gene list using a Fisher’s Exact Test in GoMiner11,12, resulting in an enrichment P for each GO term. A comparison of P values across platforms identified 101 nodes that were significantly over- or underenriched (P o 0.05) in at least four of five platforms, with nearly 60% of these terms being significant in all five platforms. Inspection of these enriched categories confirmed that the different microarray platforms were reporting the same biological responses in these samples, and also provided novel insight into the effects of comfrey exposure. Comfrey is a perennial plant that has been widely used for over 2,000 years as an herbal medicine for a wide variety of ailments. However, comfrey has been shown to be both genotoxic and hepatotoxic13. The exact molecular mechanism underlying these toxicities is not fully understood, but is known to be associated with the pyrrolizidine alkaloids present in comfrey, which can be metabolically activated and bind to DNA6,14. Considering that there are 4350 different pyrrolizidine alkaloids found in over 6,000 different species15, it has been suggested that pyrrolizidine alkaloids are ‘‘probably the


VOLUME 24

NUMBER 9

SEPTEMBER 2006

100 90 80 70 60 50 40 30 20 10 0

AG1_FC (P < 0.05) AG1_P

0


e

to those selected by P value. Similar results were obtained when the gene lists are mapped to KEGG pathways (Fig. 5) or other pathway databases (e.g., Ingenuity) (data not shown). These results clearly show that common biological responses are evident when genes are selected by criteria that lead to reproducible gene lists. Nonoverlapping lists of differentially expressed genes generally lead to inconsistent biological interpretation of microarray results in terms of GO terms and pathways.

Common KEGG pathways (%)

AFX_FC (P < 0.05) AFX_P


ABI_FC (P < 0.05) ABI_P

100 90 80 70 60 50 40 30 20 10 0


b

100 90 80 70 60 50 40 30 20 10 0


a


ARTICLES

5 10 15 20 25 30 Number of KEGG pathways

100 90 80 70 60 50 40 30 20 10 0

35

GEH_FC (P < 0.05) GEH_P

0

5

10

15

20

25

30

35


most common poisonous plant constituents that poison livestock, wildlife, and humans, worldwide14.’’ Examination of the 101 significant GO terms revealed at least two that were noteworthy: copper ion homeostasis (GO:0006878) and vitamin A metabolism (GO:0006776). Dietary or medicinal exposure to several pyrrolizidine alkaloid–containing plants has been shown to result in decreased levels of vitamin A in the liver and increased liver levels of copper16–18, but there is no indication that these effects have been observed in response to comfrey exposure. These results suggest that comfrey influences copper and vitamin A levels similar to other pyrrolizidine alkaloid–containing plants. Furthermore, these data are the first indication that changes in liver vitamin A and copper levels in response to pyrrolizidine alkaloid– exposure are transcriptionally regulated. Interestingly, only four genes associated with copper ion homeostasis are present in the common gene list and in all instances each platform identified two of these genes as significantly upregulated: amyloid beta (A4) precursor protein (APP) and prion protein (PRNP). Previously, both of these genes were shown to bind copper and were shown to be upregulated in response to chronic copper exposure19–21. Cumulatively, these findings indicate that comfrey, like several other pyrrolizidine alkaloid–containing plants, may affect liver levels of vitamin A and copper. Importantly, these data demonstrate that different microarray platforms can consistently report novel biological findings at the level of biological processes and of individual genes. DISCUSSION In this study, a data set was created that could validate and extend the findings of the MAQC project by focusing on a biologically relevant set of samples. Specifically, a large toxicogenomics data set was generated using 36 RNA samples from rats treated with three chemicals and four commercial microarray platforms to investigate the agreement of intersite and cross-platform gene lists. When a few or up to 2,000 genes are selected as differentially expressed from different sites using the same microarray platform, the percentage of overlap is B85% based on a fold-change criterion for gene ranking and selection (Supplementary Fig. 2 online). A lower percentage of overlap is observed using P as the criterion for gene ranking and selection, in particular when fewer genes are selected as differentially expressed. This same trend was also observed when gene selection methods were compared across platforms using the subset of 5,112 common genes (Supplementary Fig. 5 online). In addition,

1167


ARTICLES concordance offered by the widely used SAM approach did not achieve the same high level of concordance generated by fold-change ranking (Supplementary Fig. 4 online). These results are also consistent with those based on MAQC human samples and highlight the problems with commonly used gene selection methods that are solely based on t-test P values1,2. As expected, the degree of overlap of gene lists directly affects the ability to consistently identify the same biological response in regard to GO terms (Fig. 4) and KEGG pathways (Fig. 5). Therefore, to ensure reproducible biological interpretation of microarray results, it is important that criteria for generating lists of differentially expressed genes are selected properly. The lack of overlap of lists of differentially expressed genes selected using a P-value criterion may be explained by the fact that fold change is calculated by comparing signal intensity for a given gene as directly measured using a microarray, whereas the P-value calculation incorporates the signal-to-noise ratio. Therefore, if the signal intensity for the gene is more reproducible across laboratories or platforms than the associated noise level, this would result in the finding that fold change–based, gene-selection methods are more reproducible. However, the impact of the proposed analysis method on two other parameters, sensitivity and specificity, will also have to be assessed before any final conclusions can be drawn regarding the generalizability of this approach. Sample size is another important factor that impacts concordance of lists of differentially expressed genes. It is interesting to compare the results of Figure 3 (AFX and AFX2) with those of Supplementary Figure 2b online in which for the same microarray platform, one can observe an overall increased level in the overlap of differentially expressed genes when six replicates from different laboratories are compared as opposed to the three replicates from within the same laboratory. This increase is observed despite the potential for interlaboratory variation, which would affect the six-replicate comparisons but not comparisons of three replicates within a laboratory. This demonstrates the relationship between increases in statistical power and the resulting gain in reliable detection of differential expression that occurs with increased sample sizes. It is worth noting that differences between individual biological replicates also contribute to the relatively lower overlap observed in Figure 3. To illustrate the importance of using gene selection criteria that maximize overlap of gene lists, we first filtered the data (comfrey compared to control) using a relatively nonstringent P cutoff (o0.05) and then the remaining genes were rank ordered using fold change. By selecting the top 250 up- and downregulated genes from each platform and performing a GO enrichment analysis, not only was the cross-platform reproducibility of GO terms demonstrated, but a novel biological finding was also revealed on all platforms and at all sites. Specifically, comfrey, like several other pyrrolizidine alkaloid–containing plants, affects liver levels of vitamin A and copper; furthermore, these changes are, at least in part, transcriptionally regulated. Microarray technology has had a profound impact on biological research partially from its ability to identify differentially expressed genes that may be used to develop potential biomarkers, elucidate molecular mechanisms and group similar samples based on gene signatures. Therefore, the reproducibility and reliability of the data from a study and the choice of methods that lead to the identification of concordant lists of differentially expressed genes are critical for biological interpretation. Concerns have been raised regarding the reliability of microarray results due to the apparent lack of overlap of the lists of differentially expressed genes22–28. The results from this study suggest that the disappointingly low concordance reported in

1168

some earlier publications can be attributed in large part to the practice of deriving differentially expressed gene lists based on the ranking of genes solely by a statistical significance measure. Furthermore, these results demonstrate that microarray data generated from different platforms can not only result in a similar biological interpretation, but also reveal novel findings. METHODS Microarray processing. Details on the description of the in vivo portion of this study has been described3–6. Briefly, groups of six 6-week-old Big Blue rats were gavaged with riddelliine (1 mg/kg body weight) or aristolochic acid (10 mg/kg body weight) five times a week for 12 weeks or Big Blue rats were fed a diet of 8% comfrey roots for 12 weeks. The animals were sacrificed after 12 weeks of treatment, and the tissues were isolated, frozen quickly in liquid nitrogen and stored at –80 1C. RNA was isolated from tissues of rats that had been exposed to aristolochic acid (liver and kidney), riddelliine (liver), comfrey (liver) or a control group (liver and kidney). There were six biological replicates for each treatment/tissue group for a total of 36 samples. The samples were randomly labeled and each test site was provided an aliquot of each sample. To avoid potential confounding factors in experimental implementation, the identity of the RNA samples was kept unknown to the test sites before data were submitted to FDA/NCTR. The sample ID, RNA Integrity Number, OD ratio, microarray ID and data file names are provided in the Supplementary Table 1 online. Each of the RNA samples was labeled and hybridized to a microarray from one of four commercial platforms: Affymetrix (Rat Genome 230 2.0), Agilent (Whole Rat Genome Oligo Microarray, G4131A), Applied Biosystems (Rat Genome Survey Microarray) and GE Healthcare (Rat Whole Genome Bioarray, 300031). Except for Affymetrix, which was performed at two independent test sites, each platform was used at one single test site with 36 microarrays using biological replicate RNA samples. The labeling and hybridizations were performed according to the manufacturer’s recommendation using methods detailed in the MAQC project1. Data analysis. Unless otherwise stated, the manufacturer’s recommended normalization methods were used: quantile normalization for Applied Biosystems, PLIER with an offset value of 16 for Affymetrix and median-scaling for both Agilent and GE Healthcare1. To assess the impact of normalization methods on microarray results, we compared a limited number of commonly used normalization methods: raw, mean, median and quantile (Supplementary Fig. 3 online). The toxicogenomics data set generated in this study has also been used for the evaluation of microarray assay performance based on external RNA controls29. Six different gene selection methods were used: (i) fold-change rank ordering only, (ii) fold-change rank ordering and P o 0.01, (iii) fold-change rank ordering and P-value cutoff o 0.05, (iv) t-test P value (assuming equal variance) rank ordering only, (v) P-value rank ordering and fold change 41.4, (vi) P-value rank ordering and fold change 42.0. The percentage of overlapping genes from these differentially expressed gene lists was then calculated in the same way as was described elsewhere1. ArrayTrack30 was used for GO and KEGG pathway mapping, whereas GO enrichment analyses were performed using High Throughput GoMiner11,12. Cross-platform sequence mapping to RefSeq. Probe sequences from each microarray platform were mapped onto the NCBI-curated rat RefSeq database from March 2006. The same mapping criteria as reported for the main MAQC study was used1. The primary mapping criterion is a perfect match between a probe sequence and the target transcript sequence: a probe perfectly matches a transcript provided that a completely homologous sequence of length equal to the probe length is found anywhere on the transcript. The only exception to this rule is from the Affymetrix platform in which a ProbeSet is considered a perfect match to a transcript as long as 80% of probes within the ProbeSet (usually nine out of 11) perfectly match the same transcript. To simplify the cross-platform data analysis, a mapping table was generated with one probe per gene. Consistent with the MAQC main study1, if more than one probe from a platform perfectly matches the same gene, the probe closest to the 3¢ UTR was considered, resulting in 5,204 common non-model RefSeq mRNAs (NMs) mapped across 5,112 common genes (Supplementary Table 2 online).

VOLUME 24

NUMBER 9

SEPTEMBER 2006


ARTICLES Accession numbers. All data are available through GEO (series accession number: GSE5350), ArrayExpress (accession number: E-TABM-132), ArrayTrack (http://www.fda.gov/nctr/science/centers/toxicoinformatics/ArrayTrack/), and the MAQC web site (http://www.fda.gov/nctr/science/centers/ toxicoinformatics/maqc/).


Note: Supplementary information is available on the Nature Biotechnology website.

ACKNOWLEDGMENTS E.K.L., K.L.P. and P.H. acknowledge Agilent Technologies, Inc. and Affymetrix, Inc. for their material contributions to this work, thank John Pufky, Stephen Burgin and Jennifer Troehler for their outstanding technical assistance, and gratefully acknowledge the Advanced Technology Program of the National Institute of Standards and Technology, whose generous support provided partial funding of this research (70NANB2H3009). C.W. acknowledges Affymetrix, Inc. for material contributions to this work. R.S. acknowledges technical support of Alan Brunner for generating GE Healthcare microarray data. L.G. and L.S. thank X. Megan Cao, Stacey Dial, Carrie Moland and Feng Qian for their superb technical assistance. DISCLAIMER This work includes contributions from, and was reviewed by, the FDA and the NIH. Certain commercial materials and equipment are identified in order to adequately specify experimental procedures. In no case does such identification imply recommendation or endorsement by the FDA or the NIH, nor does it imply that the items identified are necessarily the best available for the purpose. COMPETING INTERESTS STATEMENT The authors declare competing financial interests (see the Nature Biotechnology website for details). Published online at http://www.nature.com/naturebiotechnology/ Reprints and permissions information is available online at http://npg.nature.com/ reprintsandpermissions/ 1. MAQC Consortium. The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nat. Biotechnol. 24, 1151–1161 (2006). 2. Shi, L. et al. Cross-platform comparability of microarray technology: intra-platform consistency and appropriate data analysis procedures are essential. BMC Bioinformatics 6 (Suppl 2), S12 (2005). 3. Chen, L., Mei, N., Yao, L. & Chen, T. Mutations induced by carcinogenic doses of aristolochic acid in kidney of Big Blue transgenic rats. Toxicol. Lett. 165, 250–256 (2006). 4. Mei, N., Chou, M.W., Fu, P.P., Heflich, R.H. & Chen, T. Differential mutagenicity of riddelliine in liver endothelial and parenchymal cells of transgenic Big Blue rats. Cancer Lett. 215, 151–158 (2004). 5. Mei, N., Heflich, R.H., Chou, M.W. & Chen, T. Mutations induced by the carcinogenic pyrrolizidine alkaloid riddelliine in the liver cII gene of transgenic Big Blue rats. Chem. Res. Toxicol. 17, 814–818 (2004).


VOLUME 24

NUMBER 9

SEPTEMBER 2006

6. Mei, N., Guo, L., Fu, P.P., Heflich, R.H. & Chen, T. Mutagenicity of comfrey (Symphytum Officinale) in rat liver. Br. J. Cancer 92, 873–875 (2005). 7. Arlt, V.M., Stiborova, M. & Schmeiser, H.H. Aristolochic acid as a probable human cancer hazard in herbal remedies: a review. Mutagenesis 17, 265–277 (2002). 8. Patterson, T.A. et al. Performance comparison of one-color and two-color platforms within the MicroArray Quality Control (MAQC) project. Nat. Biotechnol. 24, 1140– 1150 (2006). 9. Allison, D.B., Cui, X., Page, G.P. & Sabripour, M. Microarray data analysis: from disarray to consolidation and consensus. Nat. Rev. Genet. 7, 55–65 (2006). 10. Tusher, V.G., Tibshirani, R. & Chu, G. Significance analysis of microarrays applied to the ionizing radiation response. Proc. Natl. Acad. Sci. USA 98, 5116–5121 (2001). 11. Zeeberg, B.R. et al. GoMiner: a resource for biological interpretation of genomic and proteomic data. Genome Biol. 4, R28 (2003). 12. Zeeberg, B.R. et al. High-Throughput GoMiner, an ‘industrial-strength’ integrative gene ontology tool for interpretation of multiple-microarray experiments, with application to studies of Common Variable Immune Deficiency (CVID). BMC Bioinformatics 6, 168 (2005). 13. Stickel, F. & Seitz, H.K. The efficacy and safety of comfrey. Public Health Nutr. 3, 501–508 (2000). 14. Fu, P.P., Xia, Q., Lin, G. & Chou, M.W. Pyrrolizidine alkaloids–genotoxicity, metabolism enzymes, metabolic activation, and mechanisms. Drug Metab. Rev. 36, 1–55 (2004). 15. Betz, J.M., Eppley, R.M., Taylor, W.C. & Andrzejewski, D. Determination of pyrrolizidine alkaloids in commercial comfrey products (Symphytum sp.). J. Pharm. Sci. 83, 649–653 (1994). 16. Cheeke, P.R. Toxicity and metabolism of pyrrolizidine alkaloids. J. Anim. Sci. 66, 2343–2350 (1988). 17. Huan, J. et al. Dietary pyrrolizidine (Senecio) alkaloids and tissue distribution of copper and vitamin A in broiler chickens. Toxicol. Lett. 62, 139–153 (1992). 18. Moghaddam, M.F. & Cheeke, P.R. Effects of dietary pyrrolizidine (Senecio) alkaloids on vitamin A metabolism in rats. Toxicol. Lett. 45, 149–156 (1989). 19. Armendariz, A.D., Gonzalez, M., Loguinov, A.V. & Vulpe, C.D. Gene expression profiling in chronic copper overload reveals upregulation of Prnp and App. Physiol. Genomics 20, 45–54 (2004). 20. Hesse, L., Beher, D., Masters, C.L. & Multhaup, G. The beta A4 amyloid precursor protein binding to copper. FEBS Lett. 349, 109–116 (1994). 21. Varela-Nallar, L., Toledo, E.M., Chacon, M.A. & Inestrosa, N.C. The functional links between prion protein and copper. Biol. Res. 39, 39–44 (2006). 22. Tan, P.K. et al. Evaluation of gene expression measurements from commercial microarray platforms. Nucleic Acids Res. 31, 5676–5684 (2003). 23. Ramalho-Santos, M., Yoon, S., Matsuzaki, Y., Mulligan, R.C. & Melton, D.A. ‘‘Stemness’’: transcriptional profiling of embryonic and adult stem cells. Science 298, 597– 600 (2002). 24. Ivanova, N.B. et al. A stem cell molecular signature. Science 298, 601–604 (2002). 25. Fortunel, N.O. et al. Comment on ’’ ‘Stemness’: transcriptional profiling of embryonic and adult stem cells’’ and ‘‘a stem cell molecular signature’’. Science 302, 393 author reply 393 (2003). 26. Marshall, E. Getting the noise out of gene arrays. Science 306, 630–631 (2004). 27. Miller, R.M. et al. Dysregulation of gene expression in the 1-methyl-4-phenyl-1,2,3,6tetrahydropyridine-lesioned mouse substantia nigra. J. Neurosci. 24, 7445–7454 (2004). 28. Frantz, S. An array of problems. Nat. Rev. Drug Discov. 4, 362–363 (2005). 29. Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006). 30. Tong, W. et al. ArrayTrack–supporting toxicogenomic research at the US Food and Drug Administration National Center for Toxicological Research. Environ. Health Perspect. 111, 1819–1826 (2003).

1169

E R R ATA , C O R R I G E N D A A N D R E T R A C T I O N

Erratum: Alfimeprase to succeed Genentech’s alteplase? Brian Vastag Nat. Biotechnol. 24, 875–876 (2006)


In the 6th paragraph, the statement “in 1996, Genentech launched Alteplase” is incorrect. Genentech received FDA approval of Alteplase for heart attack in 1987. In 1996, it received approval for Alteplase’s second indication, stroke.

Erratum: Diversifying chemical arrays Laura DeFrancesco Nat. Biotechnol. 24, 799 (2006) In the print version of the article, the author of the featured article is incorrectly identified as Brandord et al. The author’s name is Bradner.

Corrigendum: All in the RNA family Beverly L. Davidson Nat. Biotechnol. 24, 951–952 (2006) In the fifth paragraph, the abbreviation for prostate-specific membrane antigen (PSMA) was mistakenly written several times as PMSA. This error also appears in Figure 1.

Corrigendum: Engineering and characterization of a superfolder green fluorescent protein Jean-Denis Pédelacq, Stéphanie Cabantous, Timothy Tran, Thomas C Terwilliger & Geoffrey S Waldo Nat. Biotechnol. 24, 79–88 (2005) In the legend for Figure 4b and in the last line of paragraph 6 in Methods, “number of moles” should be moles. Also in Methods, paragraph 3, “superfolder GFP (27.747 kDa/mole)...” should read “superfolder GFP (27747 g/mole)” and “folding reporter GFP (27.742 kDa/mole)...” should read “...folding reporter GFP (27,742 g/mole).” The error has been corrected in the PDF version of the article.

Retraction: Identification of genes that function in the TNF-α-mediated apoptotic pathway using randomized hybrid ribozyme libraries Hiroaki Kawasaki, Reiko Onuki, Eigo Suyama & Kazunari Taira Nat. Biotechnol. 20, 376–380 (2002) Although the gene discovery technology described in this paper has been demonstrated to have practical utility by several independent researchers, the first author of the paper failed to maintain a proper data notebook to support the results presented. As this constitutes nonadherence to the ethical standards in scientific research, and in accordance with the recommendations from the National Institute of Advanced Industrial Science & Technology (AIST), R. Onuki, E. Suyami and K. Taira respectfully retract this paper. H. Kawasaki declines to associate himself with this retraction and maintains that all the data contained in the paper are valid.

1170



CAREERS AND RECRUITMENT

Five attributes of a successful manager in a research organization Grace H W Wong What does it take to make the transition from scientist to manager?

L

ittle in the education, training or background of scientists prepares them for management. Good managers tend to be good with people; they look at the larger picture, are good at motivating their team, adapt to unanticipated business events, are comfortable working to budgets and are able to assess and respond to risk. In contrast, researchers at the bench spend most of their time focusing on narrow scientific questions, designing experiments and budgeting resources needed for those experiments. One would think that these two professions—business management and scientific research—were mutually exclusive, and in the vast majority of cases, one would be right. But a talented few have been successful in making the transition from the bench to the boardroom. I asked several of these individuals to identify the key attributes to their success and the factors that influenced their transition from academia into management (Box 1).

The industry research manager It’s difficult enough managing a team of people in any business. One must manage budgets, prioritize time, delegate tasks, motivate a team and provide clear leadership. But managers in a research-intensive organization, such as a biotech or pharmaceutical company, also have to contend with several additional challenges. First and foremost, a research manager’s team (that is, scientists) comprises probably one of the least manageable groups of people on the planet. As Bob Ruffalo, president of R&D at Wyeth (Madison, NJ, USA), succinctly puts it: “Although I enjoy heading a group of scientists, they are, by their nature, very difficult to

Grace H. W. Wong is chief scientific officer at ActoKine Therapeutics and president of Student Vision. e-mail: [email protected]

manage, and they are not always comfortable with change.” Second, the team’s business goal—discovering (if not developing and marketing) drugs—is an endeavor with one of the highest rates of Wyeth’s Bob Ruffalo: failure and attrition “Scientists are, by of any industry. This their nature, very means that a manager difficult to manage.” is often faced with the decision to close or shelve projects—projects that their teams often are invested in intellectually—and the highs do not necessarily outweigh the disappointments. Third, the drug sector is so incredibly diverse that expertise may not be transferable. For example, a manager at a small startup venture faces different challenges from one heading a large team at a multinational pharma company. Business and management skills acquired in a small-to-medium enterprise (SME) environment, where money and resources are at a premium, may be less relevant to teams in big pharma, and vice versa. The jobs themselves are intensive and balance many different skill sets. Although most research managers remain married to the science—devoting as much time as possible to planning experiments, carrying out secondary analysis of data and reviewing key and pertinent literature—they spend an equal amount of their time performing managerial activities (personnel and site-wide meetings), prioritizing workloads for a particular day and attending various conferences. Many of the managers interviewed emphasized the importance of remaining close to their teams, and visiting the laboratories under their supervision to


demonstrate interest and support and to answer any questions. Five key attributes Given the demands and responsibilities of a manager in a drug research group—whatever its size or focus—it’s no surprise that it takes a talented person to succeed. Exceptional individuals may each have a unique way of tackling their jobs, but on the basis of feedback from our respondents, most successful R&D managers share several common attributes. Determination. In a sector where progress often appears to take the form of two steps forward and one step back, several executives regard staying power, quiet determination and persistence in the face of adversity as a key characteristic. Lex Van der Ploeg, site head for Merck Research Laboratories in Boston, Massachussetts, believes that research managers need “motivation and drive” and a resolve not to “get discouraged by failures.” Wyeth’s Ruffalo agrees, saying that you need to “learn how to cope with disappointment. The thing that frustrates me most is the enormous risk that we face in drug discovery and development. Most people do not understand the kinds of risk. . . the pharmaceutical industry is an extremely hard place to work.” Drive and diligence. Long hours and hard work are a given in biotech and pharma research groups. A typical day for Van der Ploeg starts between 4 and 5 a.m. “I get up early, walk the dog and then head to work. Because the days are generally packed with meeting and events, I make sure to reserve space and time for reflection. Evenings are spent with my family or with late-day work events. After 9 p.m., I do a bit more work. This daily cycle runs six days per week.”

1171


CAREERS AND RECRUITMENT Robert Lewis, former senior vice president at Aventis and former chief scientific officer at Seattle-based Cell Therapeutics, also puts in long hours. At Aventis his daily routine included “at least two group scientific meetings on basic or early development projects (1–2 1/2 hours each), a group site management meeting (human resources, budget, site services, etc.) (1/2–1 1/2 hours), up to three one-on-one meetings with colleagues/scientists from the organization (1/2–1 hour each), time for reading enclosed materials on e-mail and answering/initiating correspondence (1 1/2–2 hours),

with the remainder of the time spent reading scientific journal articles. This, without lunch, amounted to a 11- to 12-hour day.” William Shek, senior scientific director at Charles River Laboratories (Wilmington, MA, USA), spends the majority of his waking hours at his company. His working day largely involves resolving “technical and management problems.” But once the laboratory has emptied in the late afternoon to early evening, he uses the time to “concentrate on writing and computer programming, which has become an avocation and occupation of mine because

of the critical role of information management in the laboratory.” He usually goes home “around midnight.” Because of these long and intense workdays, Lewis is keenly aware of the need for managers to develop time management skills.

Long days are the norm, according to Aventis’ Robert Lewis.

Box 1 In their own words William R. Shek, senior scientific director, research animal diagnostic services, Charles River Laboratories, Wilmington, MA, USA. “I became a veterinarian when I was just 13. Consequently, I went to a high school with a special program in agricultural and spent my summers working on dairy farms in upstate New York. At that time, farm experience was a requirement for entry into vet school. I graduated from high school and went on to attend the College of Agriculture at Cornell University where I majored in biology. After three years as a undergraduate, including a semester at Tel Aviv University, I was accepted to the Cornell New York State College of Veterinary Medicine where I matriculated in the fall of 1974. During the summer of 1975, I started graduate research in microbiology at the veterinary school. I graduated from there in 1977, and went on to complete MSc and PhD degrees in 1979 and 1982, respectively. Although I had been offered an a position as assistant professor at the Cornell’s State Veterinary Diagnostic Laboratory, I decided after 12 years it was time to move on. And so I accepted a job as director of microbiology and immunology at Charles River Laboratories, where I began work in the spring of 1982 and have been employed ever since.” Gary Peltz, head of genetics and genomics, Roche Palo Alto, Palo Alto, CA, USA. “I was an MD/PhD student at Stanford University who did a residency in internal medicine and a fellowship in rheumatology at the University of California, San Francisco. Although I had planned to go into academic medicine, I changed course when I looked at several academic positions in the early 1990s. The very low level of research funding was very discouraging and was coupled with very demanding clinical obligations placed on junior faculty. This made it very difficult to engage in the type of cutting-edge research that I wanted to pursue. Therefore, my first job was at Syntex Research, which subsequently became part of Roche.” Scott Wadsworth, research fellow, medical devices group, Center for Biomaterials & Advanced Technologies, Johnson & Johnson, Somerville, NJ, USA. “I had an MSc in agricultural biochemistry/ marine sciences and wanted to be a marine biologist. When I realized only a few jobs were available, I rethought my option and spent two years in a rheumatology laboratory at Children’s Hospital of Philadelphia, which inspired me to obtain a PhD in immunology at the University of Pennsylvania. Between 1985 and 1989, I held postdoctoral/staff fellow positions at the National Institute of Allergy and Infectious Diseases, studying the role of integrins in T cell development and function. And from there I joined J&J as a

1172

senior scientist. From 1995 I have been biology leader for J&J’s p38 kinase inhibitor program, anti-inflammatory drug discovery, putting four compounds into preclinical development. Since 2002, I have worked on various drug-device combination products, resulting in four prototypes handed off to operating companies for preclinical development. Currently at the Center for Biomaterials & Advanced Technologies I am continuing work on discovery/development of novel drug/biologic-device combination products for indications in orthopedics, postsurgical adhesion, postoperative ileus and drugeluting stents.” Martin Wasserman, former Pfizer, GlaxoSmithKline, Bristol-Myers Squibb, Roche, Aventis and AtheroGenics (Alpharetta, GA, USA) executive. “I began my career with an undergraduate degree in pharmacy and spent five years as a registered pharmacist in a drugstore. I decided to matriculate to The University of Texas Medical Branch in Galveston to pursue a PhD degree in pharmacology and toxicology, which I received in 1972. I was immediately recruited by The Upjohn Company in Kalamazoo, Michigan (now Pfizer), where I spent over nine years as a bench researcher in the hypersensitivity diseases research department. I was then recruited by SmithKline & French (now GlaxoSmithKline) to head the pharmacology department. When SK&F merged with Beecham in the late 1980s, my position was eliminated and I sought a position with Bristol-Myers Squibb as their first director of human pharmacology (a newly created position in clinical research) where my group performed creative phase 1 studies. After spending over three years at BMS, I was sought and hired by Hoffmann-La Roche as director of bronchopulmonary pharmacology in research, from which four years later I was recruited by Marion Merrell-Dow to become the group director of three departments (immunology, metabolic diseases and respiratory research). Soon after, MMD became Hoechst Marion Roussel and later, Aventis, and then Sanofi-Aventis, where my title was vice president and senior distinguished scientist in the respiratory and rheumatoid arthritis disease group and the acting head of oncology. After seven years, an interesting opportunity arose at a small startup biotech company, AtheroGenics. I became senior vice president of discovery research and chief scientific officer. After four and a half years, I chose to officially retire after 35 years in the drug industry and relocate to be closer to my children in California. Now settled there, I am commencing a campaign to explore opportunities to consult with the industry, academia or institutions.


CAREERS AND RECRUITMENT


“A manager should ask him/herself how able he/she is to thoughtfully delegate and monitor tasks without micromanaging, on one hand, or being blindly dependent upon others, on the other hand,” he says. “It is important for a scientific manager to be able to modulate the pace of his/her day and not to be constantly overworked. There is no reward for burnout!” Passion. Given the workload and day-today frustrations associated with working in the drug industry, a common motivation among the interviewees was altruism—to help reduce human suffering through the discovery of new medicines. Franz Hefti, formerly at Genentech Rinat’s Franz Hefti: and Merck, and Helping people is the now executive vice dominant motivational president at Rinat force. Neuroscience (S. San Francisco, CA, USA), says: “It has always been my dream and goal to bring better medication to people who suffer from diseases of the nervous system. The ability to help [do] this is [my] dominant motivational force.” Van der Ploeg is also upbeat about the drug discovery endeavor: “I am in this business because of great science, stimulating and excellent colleagues, and motivated teams.” At Roche Palo Alto in California, head of genetics and genomics Gary Peltz emphasizes the research challenge: “I enjoy solving scientific problems that can impact human health. I am particularly fortunate to work with a motivated and talented multidisciplinary team of scientists (in genetics, statistics, computation, genomics and biology) that can undertake high risk/high reward projects.” Several other managers also emphasize the rewards of participating in the research endeavor. “I enjoy the collegial nature of scientific/biomedical pursuit, the people-to-people interactions and achieving global recognition for my work,” says Martin Wasserman, a former manager at five pharma companies who recently retired from his post as chief scientific officer at AtheroGenics (Alpharetta, GA, USA). Elsewhere, Aventis’ Lewis praises Martin Wasserman advises scientists interested in management to network.

Box 2 Starting out The research managers interviewed for this article had several pieces of advice for those thinking of moving from the bench into research management at a company. Wyeth’s Bob Ruffalo exorts fledgling managers to “work very hard, publish extensively and remember that discovering and developing new drugs is one of the most noble professions, which patients depend upon us to do.” But what practical steps can you take to increase your chances of making the transition? Roche’s Gary Peltz says that when he visits universities and meets with graduate students and postdocs, he was initially quite surprised to find one universally asked question: “What was it like in industry?” “They were more concerned with my answer to that question than discussing their science,” he says. “It was clear that virtually all academic programs offer very little career counseling or direction for trainees, which is a major deficiency.” Rinat’s Franz Hefti agrees: “It’s important to understand the differences between academic and industrial research. The goal of academic research is to understand nature; the goal of biopharmaceutical research is to find effective treatment for human diseases. Academic research favors an individualistic approach that emphasizes the contribution of an individual; industrial research favors teamwork and emphasizes the common goal.” Peltz’s pragmatic suggestions for students: “First, inquire about and explore a number of options before choosing a career path. Second, realize that there is a wide range of options within industry. Just as the experience at Stanford is very different from that at a local community college, the cultures and experiences in small startup companies differs from that in large pharma companies. Lastly, I strongly suggest that students read Tom Friedman’s book, The World Is Flat. Things are changing within the pharma industry; the pace of change is going to accelerate, and you’d better be prepared for it.” Martin Wasserman advises those interested in research management careers to “consider an undergraduate degree in pharmacy, which permits exposure to most of the biomedical disciplines, unlike a pre-med degree,” adding, “Where possible, take courses in biotechnology.” Charles River’s William Shek also notes, “some of my colleagues have gotten MBA degrees and gone on to senior management.” Wasserman also stresses the importance of attending job fairs and local and national meetings for exposure and appointments. “Try to network with recruiting firms,” he says, “and consider investing in society memberships; invest in the FASEB [the Federation of American Societies for Experimental Biology] directory of members.” Finally, he advises “learn who the executives are and try to set up appointments with them.”

“the people, individually, the science (as a continuous learning experience) and the opportunity to drive many new and potentially productive ideas into actual experiments that challenge hypotheses.” Reinhard Ebner, principal scientist at Avalon Pharmaceuticals (Germantown, MD, USA), feels that a manager’s capacity to play an instrumental, or even leading, role in a team that finds the answer to a complex, long sought-after problem can be an “indescribably rewarding experience, second only to the involvement in an initiative that succeeds in making a concrete contribution to the development of a solution for a previously unmet medical need.” And it’s not only the altruistic side of the drug discovery business that galvanizes people. For Scott Wadsworth, research fellow at Johnson & Johnson (New Brunswick, NJ, USA), it’s “the independence and entrepreneurial spirit that exists, despite being part of a huge corporation. I like the opportunity to have an impact in a large corporation.”


Broad experience. Given the diverse responsibilities and skill required in a research manager position, it helps to be well read and to develop as broad a scientific and business knowledge as possible. Wadsworth says it is important to “diversify your experience,” adding, “Make sure you have demonstrated Reinhard Ebner significant, quantifi- of Avalon says the able, reproducible smaller the company, successes in your the more demanding and wide-ranging early career. Network the management as much as possible problems. within your company and outside. Gain as much exposure outside your company as possible, via speaking engagements, chairing meetings, publishing, etc. Gain as much

1173


CAREERS AND RECRUITMENT management experience as possible, by leading project teams, mentoring postdocs, hosting interns, etc. Do all that and management positions will come naturally.” Charles River’s Shek also emphasizes the importance of broad horizons. “I have found it to be particularly important to acquire knowledge and skills beyond my field of scientific specialization in the areas of quality control, project management and bioinformatics,” he says. He adds, “I have had the resources to do many interesting things, to expand my knowledge and skills and to collaborate with highly intelligent and talented colleagues at and/or outside [the company]. As a member of the research models and services division of Charles River, I have participated in a wide array of projects involving diverse disciplines including genetics, diagnostics, engineering, bioinformatics and so forth.” Flexibility, inspiration and leadership. There is no doubt that the pharma industry is currently undergoing a difficult period in terms of sustaining growth, meeting investor expectations and public perception. Working to improve the poor productivity and high attrition of drug pipelines is a key goal for many research managers. Roche’s Peltz puts the problem like this: “My major challenge is maintaining momentum and progress within a constantly changing environment that has an increasingly near-term outlook. This makes it more difficult to maintain cohesion among the large number of individuals performing

1174

the work, and with the stakeholders concerned about the outcome. Discovery science is a lot like cooking; if you open the oven door too often, the cake will not rise.” There is a burgeoning demand for experienced research managers in biotech companies; many of these are being recruited from pharma. But as Avalon’s Ebner points out, having consecutively worked for established large, growing medium-sized, and entirely new startup biotech companies, the decision paths in small and large enterprises are very different. “The younger and more unfinished an institution, the more demanding, wide-ranging and intensive the management problems. This is most acute in the startup setting, which is a bit like starting a family restaurant, where everyone has to help out on every front.” The increasingly cross-disciplinary nature of research and the need to collaborate intramurally and extramurally also creates management headaches. “Some of the bigger and most difficult, yet most important, questions can only be answered by the coordinated studies of many investigators, often from different institutions and countries. This has been true for many fields of discovery for a while, but is now increasingly apparent in the biological sciences. Making the best use of combined efforts almost always requires a great deal of organizational, communicative, planning and even diplomatic skills,” says Ebner. A research manager’s job also offers the opportunity to mentor and reward excellence

and achievement within a team. According to Lewis, with senior management positions, he really enjoyed the chance to do “things that seriously affect the quality-of-life for employees in a positive way; this means that the ‘power’ of a senior job is most useful when it is used to (appropriately) enrich the lives of the most junior colleagues.” Conclusions There are several keys to success in making the transition from the bench to the boardroom (Box 2). Research managers need determination, diligence and passion to do work that matters and that makes a difference. They must also possess the experience to lead and ability to inspire their team. The most effective managers are patient, have a sense of humor, respect their colleagues and are willing to subordinate their ego for the benefit of the organization. If you love being a scientist but crave the financial and professional benefits of management, heading a research group as a department or division leader at a company offers several opportunities. In this role, you will have greater supervisory and budget management responsibilities, and the compensation that comes with them. A person who is a great bench scientist will never be happy being a mediocre manager, but a great scientist who has the ability and desire to move into management has a whole new set of opportunities to achieve important and satisfying results.



PEOPLE

Archemix (Cambridge, MA, USA) has announced the appointment of Robert Schaub (left) as vice president of preclinical discovery. Dr. Schaub comes to Archemix after 16 years with Genetics Institute and Wyeth Pharmaceuticals, most recently as the assistant vice president for cardiovascular and metabolic diseases. “Archemix’s aptamer technology has the potential to bring forth an entirely new class of therapeutics for the treatment of acute and chronic diseases,” says Dr. Schaub. “I look forward to leveraging my experience with both biotherapeutics and small-molecule drug candidates to this new class of therapeutics.” In addition, Archemix has elevated Page Bouchard (above) to lead its research and preclinical development group. Dr. Bouchard joined Archemix in November 2004 as senior vice president of preclinical drug discovery and development. “Page possesses the unique combination of scientific experience and leadership skills necessary to guide our rapidly expanding pipeline of therapeutic aptamers through research and preclinical development,” says Archemix president and CEO Errol De Souza. “We are privileged to have a leader of his caliber directing our R&D efforts.”

Genomic Vision (Paris) has appointed founder Aaron Bensimon as president and CEO. He will be assisted by Daniel Nerson, who has been named chief operating officer. Dr. Bensimon has been head of the genome stability unit at the Institut Pasteur since 1994, where he developed molecular combing technology and its use in the precise study of genomes. The technology has resulted in 13 patents granted to the Institut Pasteur, for which Genomic Vision has an exclusive license. Novavax (Malvern, PA, USA) has announced the appointment of Jeffrey Church as vice president, chief financial officer and treasurer. He joins the company from GenVec, where he served as CFO, treasurer and corporate secretary since 1988. Bernhard R. M. Ehmer has been appointed to the supervisory board of Hybrigenics (Paris) as a non-executive independent director. Dr. Ehmer is currently CEO of BioPheresis Technologies, and previously served at Merck KGaA in several capacities, most recently as vice president for corporate strategic planning and alliance management. Sylvie Gregoire has been appointed executive chair of the board of directors at IDM Pharma (Irvine, CA, USA). She has been a board member since August 2005. Dr. Gregoire previously served

1176

as president and CEO of GlycoFi, and currently serves on the boards of Cubist Pharmaceuticals and Caprion Pharmaceuticals. Algeta (Oslo, Norway) has appointed Johan Harmenberg as chief medical officer and Michael Dornish as chief scientific officer. Before joining Algeta, Dr. Harmenberg spent nine years at Medivir as vice president of development. Dr. Dornish has nearly 25 years’ research experience in the life sciences industry, most recently as vice president, R&D at FMC Biopolymer. Dr. Dornish replaces Algeta cofounder Roy Larsen, who has decided to pursue other interests and opportunities but will continue as a consultant to the company. Peter Hnik has been named chief medical officer at iCo Therapeutics (Vancouver, BC, Canada). Dr. Hnik most recently served as associate director of clinical research with QLT, playing a critical role in designing and directing Visudyne clinical trials in AMD and diabetic retinopathy. Celera Genomics Group (Rockville, MD, USA) has named Joel Jung vice president of finance. Mr. Jung has held several executive and senior positions with Chiron, including most recently vice president and treasurer. Rosemary Mazanet has joined the board of directors of Cellumen (Pittsburgh, PA, USA).

Dr. Mazanet is presently CEO of Breakthrough Therapeutics and acting CEO of Access Pharmaceuticals. Previously, she has served as the CSO and general partner of Oracle Partners, and before that was director of clinical research at Amgen. James A. Ratigan has joined Nitric BioTherapeutics (Philadelphia, PA, USA), formerly known as Theranox, as CFO. He previously served as executive vice president and CFO of Orapharma, where he raised private capital for the startup, directed its IPO in 2000 and helped orchestrate its sale to Johnson & Johnson. ArQule (Woburn, MA, USA) has named Nigel J. Rulewski as chief medical officer. Dr. Rulewski brings to ArQule more than two decades of experience in R&D, regulatory affairs and commercialization, having previously served as senior vice president of BioAccelerate and vice president, medical affairs and chief medical officer at Astra USA. ADVENTRX Pharmaceuticals (San Diego, CA, USA) has announced that Joachim P. H. Schupp has been appointed to the newly created position of vice president of medical affairs. Dr. Schupp served most recently as vice president of clinical business solutions and clinical data services at ProSanos. Steve Toon has joined the board of Simcyp (Sheffield, UK), which offers in silico simulation and prediction of pharmacokinetics and drugdrug interactions in virtual patient populations. Dr. Toon has over 20 years’ experience in the pharmaceutical industry, previously serving as CEO of Medeval. AAIPharma (Wilmington, NC, USA) has named Martin Tyson to the position of senior vice president, information systems and technology. Mr. Tyson was most recently senior vice president and chief information officer for Quintiles Transnational. The company also announced the appointment of Ninad Deshpanday to the newly created position of vice president of pharmaceutical business development. Previously, Dr. Deshpanday was vice president of drug product development for Synta Pharmaceuticals.


No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

Recommend Documents