Bioinformatics Software Engineering: Delivering Effective Applications

Bioinformatics Software Engineering

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

Bioinformatics Software Engineering Delivering Effective Applications

Paul Weston Woodcock Stewart Ltd

Copyright © 2004 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777 Email (for orders and customer service enquiries): [email protected] Visit our Home Page on www.wileyeurope.com or www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to [email protected], or faxed to (+44) 1243 770620. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The Publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Weston, Paul, HGMP. Bioinformatics software engineering : delivering effective applications / Paul Weston p. ; cm. Includes index. ISBN 0-470-85772-2 (alk. paper) 1. Bioinformatics. 2. Software engineering. I. Title. [DNLM: 1. Computational Biology. 2. Software Design. QU 26.5 W536b 2004] QH324.2.W47 2004 572.8¢0285 – dc22 2004016911 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0 470 85772 2 Typeset in 11/131/2pt Sabon by SNP Best-set Typesetter Ltd., Hong Kong Printed and bound in Great Britain by TJ International Ltd., Padstow, Cornwall This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.

Contents

Preface

ix

List of figures

xi

Acknowledgements

Part 1 Introduction 1 What You Need to Know

2

1 3

The software development process: what is it you do all day? The application lifecycle: from concept to construct How to deliver effective bioinformatics applications

3 4 6

What Is Software Engineering?

9

Software engineering: applying formal rules to creativity The benefits of discipline: reproducibility, stability, solidity, reliability A limitation of engineering On the edge: agile computing and evolutionary development

9

Part 2 Before Beginning 3

xiii

11 11 12

15

Project Definition

17

What you have got to do, in one sentence? When have you got to deliver? How will success be measured? Who will measure success? Overall vision? Expected results? Required functionality? You need them all Work with what you have

17 18 18 19 19 20

vi

CONTENTS

4

5

Requirements Capture

29

Separating Function, Interface and Implementation

33

User stories will tell you most of the functionality that is required From screen drawings to screens Your implementation has to create and connect the interface and the functions

6

7

21 22 24 24

30 30 31

33 37 37

Implementation Considerations

39

Languages Platforms Operating systems Prototyping tools Debugging tools

39 40 40 42 42

Proof of Concept, Prototyping and Buy-in

45

Explaining the development process with prototypes and proofs of concept ‘Too much information! Too much information!’ Markets: how to sell yourself and your work, and why you need to HyperCard, genetic algorithms, evolutionary development and ‘doing science’ Using adaptive development to encourage customer buy-in

Part 3 Getting it Done 8

21

When your customers are talking to you about what they want, stay focused User stories Avoid ‘blue-skying’ in meetings – summarize, don’t improvise Pencil > keyboard: screen drawings You will need to know what your sample datasets and expected results will be Yes, now is the time to start thinking about how to test for success Documents describing staged deliverables need to be agreed to, if not formally signed off Get requirements capture right at this stage, so that toppriority requirements can go into the functional specification

45 46 47 48 50

53

Data in, Data out and Data Transformation

55

Begin, process, end: process flow diagrams Boxes in boxes: data structure diagrams

55 59

vii

CONTENTS

9

10

Where to Start?

63

You will need to get in data You will need to cope with errors, because they will happen You will need to report results Known and unknown coding From designs to pseudocode From pseudocode to code

64 64 65 67 67 71

Functional, then Optimized

73

Get it out of the door, and in front of your customers, as soon and as often as safely possible Planning shows When not to display work in progress – getting them used to having it Customer retention and repeat buyers Optimization: benchmarking, assessment, refinement, hardware Focus on essentials, because your customers will

11

73 74 75 76 77 78

Coding Style

79

Don’t write it if you don’t have to KISS – Keep It Simple, Stupid When you look at your code, does it make your eyes hurt, or does it look like poetry? Naming of parts Going gracefully into the darkness Prevention is better than cure Flexibility is vital Keeping track of what you are doing

79 80 80 81 82 83 83 84

Part 4 For Some Values of Done . . .

87

12

13

Writing the Friendly Manual

89

Start as you mean to go on: make your code clean and clear Using what you have done so far: it’s half-written already Writing documentation from functional specifications, and vice versa Ask yourself, and others, who still needs to know what?

89 90 91 91

Testing – What and When

93

Test plans Writing a test plan Expect the unexpected Why developers shouldn’t be their own testers Involving users in the testing process Some things to watch out for When to stop testing

93 94 95 95 96 97 99

viii

CONTENTS

14

Rollout and Delivery You will need separate development, test and production environments Delivery notes Verbal handovers Building the installation package Instant gratification mode The useful/stable balance

15

Support and Feedback When do you start? Pre-emptive support What are your local customs? Development costs v. support costs Watch out for unanticipated feedback requiring urgent priority readjustment

16

Planned and Unplanned Enhancements Good applications are never finally finished Things not to say in meetings, No. 94: ‘Oh, you found that one . . .’ ‘It’s not a bug, it’s an opportunity to further enhance the user experience’ Big shoes: managing change in the workplace Priority 2: It’ll be along real soon now, when we’re all less busy Slippage

17

Project Signoff Dealing with bad stuff: focusing on the successfully delivered objectives Identifying potential endpoints and agreeing an exit strategy Where now?

Index

101 102 104 105 105 106 107

109 109 109 110 110 111

113 113 114 114 114 115 115

117 118 119 119

121

Preface

So, what is this book all about, then?

Real bioinformatics – building adaptable tools to process, manage, analyze and display biological information – requires not just an appreciation of the underlying science, but also the ability to write efficient computer programs. Bioinformatics Software Engineering aims to build on these basic capabilities. It explains the principles and techniques that will help you to give your customers the software they need. Bioinformatics Software Engineering shows you how you can save time and trouble by doing the right thing, at the right time, in the right way. It provides practical, helpful information that you can apply to your own work. This book goes beyond coding to examine the whole project lifecycle. It guides you through the application development process from start to finish, without tying you to a particular methodology, paradigm, or platform. I hope it will give you the confidence to take on new challenges. Bioinformatics Software Engineering covers: • Agreeing requirements • Design tools and techniques • Defining appropriate documentation • Hints and tips on development and coding • Test planning and execution • Rollout and delivery

x

PREFACE

• Support, maintenance and enhancement • Project signoff If some of these terms seem unfamiliar, don’t worry. They are explained at the start of the next chapter, and are discussed in more detail throughout the book. Who is the book written for?

Both bioinformaticians looking to enhance their development skills, and scientists seeking to understand more about how the applications they use are created, should benefit from this book. How do I find what I need to know?

Bioinformatics Software Engineering is arranged according to the stages in the development lifecycle, so you can read it straight through, or go to the section that covers your problem area. Why did you write this book? And who are you, anyway?

I have tried to write the book that I didn’t realize I needed when I began my computing career. I hope that it helps you avoid some of the mistakes I made. I started developing software tools for scientists at the UK’s Human Genome Mapping Project Resource Centre in 1996. I have also projectmanaged the development of an online game, and then been European Technical Consultant for a San Diego-based distributed computing company. As well as building tools, I now provide consultancy and training in application development and software engineering, and I have a particular interest in genetic algorithms. Outside work, I have been a rhythm guitarist and songwriter for twenty years, I enjoy gardening, and I have followed Liverpool Football Club since 1987. And as far as George and Lillie are concerned, I’m just Dad. Paul Weston

List of Figures

Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure

4.1 4.2 4.3 4.4 4.5 4.6 8.1 8.2 8.3 8.4 8.5 8.6

Screen drawing: desktop after program installation Screen drawing: program start-up screen Screen drawing: simple dialogue box Screen drawing: selecting options Screen drawing: confirming the choice Screen drawing: program finish screen Process flow diagram: linear process Process flow diagram: repeated process Process flow diagram: selection process Process flow diagram: complex process Data structure diagram: simple assembly structure Data structure diagram: complex assembly structure Figure 9.1 Process flow diagram: looping until success or exit Figure 9.2 Process flow diagram: report construction

26 26 27 27 28 28 56 57 57 58 61 62 66 68

Acknowledgements

Everyone that I have worked with has helped me write this book. Thank you all. Particular thanks are due to: • Lee Cave-Berry, for getting me interested in what you can do with computers; • Jo Wixon, for the continuous encouragement that turned my ideas into a book; • Damien Counsell and Phil North, for reviewing the manuscript in draft; and • Phil Gardner, the best boss I have ever had.

Part 1 Introduction

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

1 What You Need to Know

This chapter introduces some basic software engineering concepts that the book will use, and provides a brief tour of what is covered and why.

The software development process: what is it you do all day? Some people think that application development just means cranking out code. They are nearly right, and almost completely wrong. If you just crank out code, you will never deliver an effective application. Writing software properly involves talking to people – often lots of people – and plenty of non-coding work on your part. It requires the ability to dream up new solutions to problems so complicated that they are hard to describe. You have to be a detective, following up clues and examining evidence to discover what has gone wrong and why. And you have to be a politician, understanding what people want, both in public and in private, and how this is likely to affect what you are trying to do. This book cannot teach you to do all of that, but it can help. The actual coding is a small part of application development. This process starts long before you sit down at your computer and ends long after you have switched it off. For example, what is the point of writing code if you don’t know what your users want it to do? You might think Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

4

WHAT YOU NEED TO KNOW

you know what they want. But even if you are right, can you be sure that is all that they will ever want it to do? And can you prove this to them?

The application lifecycle: from concept to construct Requirements capture (finding out what your customers want), testing (making sure that your product does what they specify), and demonstrating your application to users (showing them that they have got what they asked for) are three parts of the application lifecycle. A program begins as an idea, and becomes a tool for others to use. That’s the theory, anyway; in practice, this journey from one mind to many is often very convoluted. You can make it easier for everyone involved if you can learn what sort of situations to expect along the way and are suitably prepared for them. In the preface to this book, I mentioned a number of aspects of what is known as ‘software engineering’. Let’s take a look at each and explain what we mean by them.

Agreeing requirements

Agreeing requirements means more than finding out what your customers want. It means coming up with a definition that tells you what your program has to do and tells them what they can expect to receive, and which all parties are happy with. The work you do at this stage should also generate enough information for you to start thinking about how to plan tests and specify the criteria by which the success of the program will be measured. Later on, we will explore a couple of techniques that I have found useful in this situation.

Design tools and techniques

Next, you need to think about how you are going to start the design of your solution to your customer’s problem. One of the first things that you will need to do is decide which tools and techniques are appropriate. There are many ways that you can write software; you need to identify the best for your situation.

THE APPLICATION LIFECYCLE: FROM CONCEPT TO CONSTRUCT

These decisions need to be made once you know what the problem you have to solve is, as the choices that you make at this point will determine what options are available to you later on.

Defining appropriate documentation

You will need to consider the documentation that will be required. The volume and type of documentation that you need to produce will depend on your organization’s culture and way of working, as well as on the tools and methodology that you decide to use. Its nature will also be determined by what sort of customers you have, and what environment your program will be used in.

Hints and tips on development and coding

At some point you have to stop designing and documenting, and start developing. Whether you are working from a complete formal specification, or creating an initial prototype, this can be a daunting step. We will look at ways to make this easier.

Test planning and execution

As soon as you begin coding, you begin testing. It can be helpful to plan not only what you are going to test, but also how and when. This is an area we will examine in more detail.

Rollout and delivery

Your customers will want to try what you are building for them as soon as possible. We will discuss ways of managing this process of rollout and delivery to make it as painless as possible for all concerned.

Support, maintenance and enhancement

Once you have delivered what was promised, you move into the support, maintenance and enhancement period. ‘Support’ means

5

6

WHAT YOU NEED TO KNOW

looking after your program’s users as they get to grips with its capabilities. ‘Maintenance’ means making sure it keeps on working for them, and dealing with unexpected occurrences. ‘Enhancement’ involves adding new functionality; often the need for this is identified as your program is used. Project signoff

Finally, we will look at how to end each project gracefully, signing off and moving on. This is a time to look at lessons learned and at things that could be done better next time. It is also a time to celebrate your success. Each chapter from now on will cover one stage in the journey of an application from inside your head to the hands of your customers, and will discuss what is involved and what to watch out for.

Always keep your customers in mind

Even if you are in a not-for-profit environment, it is useful to adopt a customer-focused approach. All software development is funded by somebody for some reason, even if you are writing it in your free time simply because you enjoy it so much. If you are your own customer, you will know better than anyone else what you want. If your customers are colleagues, treat them as you would like to be treated if they were building something for you.

How to deliver effective bioinformatics applications The most important thing that you can do in order to ensure that the applications you deliver are effective is to listen. • Listen when people are talking about what they want. • Listen to what they aren’t saying – what is being assumed? • Listen to what they say or don’t say when you show them what you have done – what were they expecting to see?

HOW TO DELIVER EFFECTIVE BIOINFORMATICS APPLICATIONS

• Listen to what other developers say, and learn from their experience – especially their mistakes. • Listen to senior colleagues and learn about your organization from what they imply or allude to. • Listen to more general discussions and get a feel for the trends in your field. • Listen to yourself – do you feel that you can be proud of your work?

7

2 What Is Software Engineering?

We have looked briefly in the last chapter at some aspects of what is called ‘software engineering’. Before we go on to examine it in more detail, let’s first think about what software engineering is, why it is useful and why it is sometimes not enough.

Software engineering: applying formal rules to creativity There is an interesting tension between the left-brain and right-brain aspects of software development. On one side is a creative process; on the other, the need to produce accurate results, every time. Many attempts have been made to formally define a successful approach to this situation. Some have proved more popular than others. I do not propose to review any; several other works already do this better than I could. What I am going to try to do is take the best bits of all of them, and show you how you can use these ideas in your work. In my experience, any way of applying engineering to software development is probably better than no engineering at all, simply because we can learn a lot about how to build structures of the imagination from structures of the physical world. I was standing on top of a C-130 Hercules transport aircraft, watching the bustle in the hangar around it, when I started to realize what Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

10

WHAT IS SOFTWARE ENGINEERING?

‘engineering’ meant. I had come across the road from the Technical Publications department to find out how the fitters had put in the latest modification, so that I could write the installation instructions that would be sent with it to airfields around the world. I could have asked the designers – after all, they had already explained to the fitters what was needed – but there were problems with this. By now, the designers were already deeply involved in new projects and would not relish having to turn their attention back to something that they had moved on from. Moreover, the fitters were used to working from rough sketches and their experiences when trying to install the modification often led to changes in its design. What I had to create was a set of instructions that covered exactly what a fitter needed to do at each stage. Therefore, I spoke to those who had thought up ways of how to make tab A actually fit into slot B, and asked them how they did it, then wrote that down, and then got them to check that I had described it properly. Without the creativity of the designers, able to invent solutions, and the ability of the fitters to think up the best way to implement them in their often cramped, crowded and inaccessible locations, the improvement would not have happened. But without someone writing down, step by step, the best way to fit the modification, in aircraft all around the world, engineers would have to work out their own ways of doing it – and chances are it would not be as the designers planned it. But the designers would not just have had the idea and then gone off to celebrate. And the fitters hadn’t just slotted the modification in first time. ‘Trial and error’ means what it says. You try a possible solution and it’s wrong. So you try another. The fitters and designers who stay employed are those who are good at quickly identifying solutions that work. Both the fitters and the designers made progress steadily; creativity had been accompanied by analysis, contemplation, experimentation. There had been setbacks, but these had been overcome. Everybody worked together as a team to achieve a common objective – everybody played their part. My writing the installation instructions was itself a step in the process of engineering a solution to a problem. In the same way as a designer or a fitter, I had to determine what was wanted and deliver it as efficiently as possible. When you write software, you invent solutions and find the best way to implement them in your environment. Adding engineering to this

A LIMITATION OF ENGINEERING

basically means that you use what you have learnt before and do things in a logical fashion. Using software engineering principles and techniques doesn’t mean that you have stopped being creative; it means that you are taking your creativity seriously.

The benefits of discipline: reproducibility, stability, solidity, reliability Let’s take a look at the advantages of supporting our programming creativity with the disciplined, methodical way of working found in traditional engineering. One thing that we need to do is produce reproducible results; if you run your program ten times on the same sample data with the same program settings and you don’t get the same output each time, you are going to be concerned. You want to get the same results from that dataset with those parameters, whoever runs the program and whatever external conditions it runs under. What’s more, you don’t want the program to crash halfway through the seventh run – it needs to be designed to fail gracefully, if at all. Once you have your error-handling routines in place, you will then need to minimize the number of times that this code is called, by fixing all the bugs and pre-empting possible problems. Eventually, the only unexpected program terminations should be those caused by factors outside your control, such as hardware failures or operating system bugs. The formality of an engineering approach will ensure that the application is seen as solid to its users and they will come to rely on your software – and on you – if what you build has been well constructed. This is much more important than fancy bells and whistles. A car that looks flash but rusts easily and won’t start on cold wet days doesn’t have a long-term future. If you don’t write reliable software that does what its users want it to, what will your future be like?

A limitation of engineering One limitation of the engineering approach is at first glance an advantage: it copes well with fixed and clearly defined requirements.

11

12

WHAT IS SOFTWARE ENGINEERING?

The converse is also true: this way of working is difficult to adapt to more fluid circumstances. Let’s consider a situation where your job is to build a bridge across a valley. You are not going to start unless you have done all the calculations which prove that your design is going to stand up to the stresses and strains that will be placed upon it throughout its planned life. Equally, if your job is to summarize for management the most valuable transactions that took place last week, you have got a fixed and clearly defined requirement, which allows you to design a solution, plan testing, and consider rollout and support issues before you start to code, just as the classical software engineering approaches say that you should. However, in bioinformatics (and many other fields), it often doesn’t work like that. Traditional techniques – especially those that emphasize the importance of documentation before development – are often completely inappropriate, simply because they are too inflexible. We work in a field where people try things just to see if they are possible, and then look to see what they might do with them. Sometimes you have to start building the bridge before you can even see the other side of the valley. Scientists hypothesize and experiment and would be the first to admit that things often don’t go according to plan. Sometimes a completely new approach is needed halfway through a project, and this may require extensive reworking of software tools. Not many industrialstrength methodologies include a phase where you have to throw away much of what you have done so far. But what about more modern ideas?

On the edge: agile computing and evolutionary development There has been a lot of interest in more flexible approaches to application development and there are many merits in what they propose. However, in my view none of them is a silver bullet. My own approach is based on my experience of working with rapidly changing requirements and imaginatively experimenting customers. It has been further influenced by my work with genetic algorithms, which use mechanisms like selection, mutation, and crossover of characteristics on a population of candidate solutions, to cope with complex multi-parameter problems.

ON THE EDGE: AGILE COMPUTING AND EVOLUTIONARY DEVELOPMENT

The best way I can describe my approach is to say that it is adaptive. You do what you need to do in order to successfully complete the job that you have been asked to do in the circumstances where you find yourself. If your boss says you need to do a task a certain way, then try it. You might find that whatever novel technique is being proposed has benefits that you hadn’t anticipated and that you will be able to use what you have learnt on future projects. That’s what I have done. I have been taught, and trained, and tripped up. I have enquired, and studied, and observed. I have thought about what it is I do when I do my job. Some approaches work for me in one set of circumstances; other techniques are needed at different times. No one way is right for everyone, even this one; but adaptive programming means that you simply take the bits you think are best out of this book, try them out in your work, and alter them to suit your situation. If it helps you, I’ll be happy – and I would like to know how you get on.

13

Part 2 Before Beginning

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

3 Project Definition

This chapter identifies basic questions – what? when? how? who? – that must be answered before you start development. It goes on to suggest information sources which will help you answer these questions, and how to cope without them.

What have you got to do, in one sentence? Here is where you start thinking about the point or purpose of the application that you are about to build. It’s quite likely at this stage that you couldn’t sum it up in one sentence, but rest assured that you will soon be able to, once you are further down the road. For now, use as many sentences as you need to, and periodically come back and rework this definition, aiming to make it more concise every time. This will help you focus on what is a core element and what is less necessary. For instance, you could try to get your original definition onto a single page, and then aim at the next stage to reduce this to a single paragraph. If you don’t have a project proposal or similar document to get you started, find everything you can that helps even partially to define your project – business plans, funding applications, tender documents, press releases, and so on. Use these to build up a list of key phrases. If you think that your list is shorter than it should be, then use that as a

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

18

PROJECT DEFINITION

starting point in discussions with your customers and, if possible, the authors of the original documents.

When have you got to deliver? This may well be difficult to determine, particularly if you are planning to introduce new functionality gradually. If in doubt, ask your customers; if you cannot get even a rough feel for how long the first stage of work is going to take, you need to clarify what it will involve by talking to them some more about it. If your customers ask how long something will take, go away and work it out as well as you can, then double it, and then add ten per cent, and it will still probably take longer than that. The reason why you won’t have a problem with this is that you have in reserve the programmer’s secret weapon – the 24-hour day. Yes, there are times in software development, as in life, where sleep is not an option. These times are brought about by things known as ‘approaching deadlines’. We will deal later on with how to cope with worst-case scenarios; for now, don’t get pressured by timescales, because slippages are recoverable. If you really have to give a delivery date, give a partial one – when you expect to deliver at least some of the functionality required, preferably the bits that you have already started. Then explain to your customers that you will need to consider their comments on that rollout, and you will then be able to give them a date for the subsequent deliveries. Your project proposal may contain predetermined delivery dates. If so, this will help you plan your work. If not, you may need to plan this yourself. We will look more at what this involves later on – we need to determine what we have to deliver before we can say when it will arrive.

How will success be measured? If you don’t know where the hoops are, how can you jump through them? If your customers cannot tell you how they will define success, you can never succeed. If you and your customers are unable to agree, in writing, before you start, what they will be happy with at what stage, then unless

OVERALL VISION? EXPECTED RESULTS? REQUIRED FUNCTIONALITY?

everyone involved decides to put all their efforts into reaching agreement, you may as well abandon the project right now.

Who will measure success? Your customer – the person who pays for what you do – may not be the only individual who determines the success, or otherwise, of your efforts. Your application may conform to specification in terms of delivering the required results in the right format; however, if your users have to wait too long for those results to arrive, they won’t be happy. Nor will your systems and network team be filled with joy if poor design means your application hogs bandwidth or other resources. Don’t annoy the people who control your computer.

Overall vision? Expected results? Required functionality? You need them all There are many ways of defining a specification for a program. It is important to have as many perspectives on what is envisaged as possible, particularly if different user types with differing requirements are involved in the process. One customer may be concerned with the subtleties of the interface that give the user a ‘feel’ for what the program is like – such as consistency of colour schemes, appropriateness of interaction styles and terminology used. Others will be focused on aspects such as conformance to defined rules of operation – the best hand in the game always wins, for instance. Still more will concern themselves with the time between a user action and the software’s response, and client or server performance under boundary conditions and extreme stress. All of these aspects of a program’s operation need to be addressed; not all of them will influence what actually gets done. If speed of message passing is deemed less important than the quality of the user’s experience of the client software, then you know where to put the most effort. Don’t spend time on functionality that your customers don’t want, and don’t start coding until you know at least some of what they really cannot manage without. We are shortly going to look at how you can

19

20

PROJECT DEFINITION

get this vital information from your customers down on paper in a way that you both can agree on.

Work with what you have Even if it is only one line in a project proposal, you have enough to get started. For instance, given ‘Create an online version of our printed registration form’, you know what it has to look like (the existing form) and what it has to do (allow users to register over the Internet). So, at a minimum you will need to consider the following: • web page design, • data capture, validation and storage, • user response planning, and • program testing. Meanwhile, you can be exploring the grey areas of the specification further. If you don’t have even this much definition, the first thing that you need to do is get the information yourself. Talk to your users. Determine who your customers are and find out exactly what they want. Don’t guess, and don’t assume. Consult with colleagues, if you don’t know which customer to start talking to. Get things down in writing, even if it’s only an email summarizing what was discussed at coffee-time, as long as it helps to clarify what you have to do to succeed.

4 Requirements Capture

Once you have a usable project definition, you can improve your understanding of what is needed by further involving those who need it. We will examine two simple methods – user stories and screen drawings – which will help you and your customers agree a specification of what is to be delivered, and also look at ways of prioritizing requirements.

When your customers are talking to you about what they want, stay focused There are two key points which you should try to bear in mind all through the process of capturing user requirements.

Get the gist, while filtering out the garbage

Very few people only ever say exactly that which is needed. Your customers will be trying to communicate what they think you should know. Some of this will be relevant to the design of your software – ‘I want every dialogue box to have a blue background’. Some of it will affect future interactions with that customer – ‘I want to personally welcome every new user’. Some might affect how your employers deal with the customer’s organization – ‘We cannot commit to Stage II until the business-wide Quality Focus Review is completed’. Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

22

REQUIREMENTS CAPTURE

If you don’t feel confident in your ability to categorize signals like these, check out your local bookstore for something on the psychology of business communication that you like the style of. Or buy a Dilbert book.

Keep asking, ‘Is this core?’

So, you have now identified those things that your customers are saying which actually might have some relevance to the job you have to do, rather than the game last night or the latest rumour. So far, so good, but this is no time to relax. Now is the time for your second-stage filters to kick in. They may be talking about software, but are they talking about your software? If it’s a competing product, are they discussing what features they want, don’t want, or want to see improved? Use your experience and common sense. When you have to code a possible new way of solving a complex problem, form follows functionality. Get the right results with the simplest possible interface. Think ‘Functional, then Optimized’ – you can discuss the graphics for the web pages later, once you can consistently deliver correct results.

User stories User stories are useful when you want your customers to get to the point and describe what they really want. It’s a simple technique which helps to focus on particulars, while still allowing general characteristics to be defined. To generate user stories, you simply need to talk to your customers. Let’s just pause a moment and think in a little more detail about who your customers are. When you start capturing requirements, the first person to define what is needed is probably going to be the one who sold the project to its backers. They are therefore likely to be focused on the big picture and will paint it with a broad brush. This overall vision should be reflected in all aspects of the program. However, as a developer, you need to know in precise detail what all these aspects will actually entail, and that is where you will need to get together with your most senior customer’s colleagues – the ones who have to make it all work. These are the people who will already have thought about how to turn their leader’s vision into a reality, and these are the people you will

USER STORIES

need to talk to in order to develop the user stories which will help to define exactly what it is that this program is supposed to do. A user story is just what it says. You are a user and you want to use this program to achieve a particular result; you write down what you as a user do to reach that destination. • You, the program’s user, sit down at your computer. How do you start the program? • Once you get started, what do you see? • What options are available to you at this stage? • What happens when you choose this option or that one? • Can you save what you have done? • Can you undo what you have just done? • How do you give the program the information it will need in order to do the job that you want it to do? • How does the program deliver its results to you? • If there is a problem, how will you be told about it? • What will you then be able to do? • How can you cancel, go back or quit? • What happens if you want to quit in the middle of something? Ask questions like these of the people who should know and make notes of their answers. Then give them copies of your notes and work together with your customers to turn these notes into something that you can agree on. You and your customers need to write user stories for all possible different users of your program. You may think that you will only have one type of user, but this is rare. • Who is going to install your program? • What about upgrades, when you release new functionality?

23

24

REQUIREMENTS CAPTURE

• Does your software have to integrate with other applications? • Do system administrators have access to special program features? • Do managers need performance and usage statistics? Even if your customers are all members of the same small team, they will probably all have individual ways of working which you should take into consideration when designing their application. Check with your customers that you have covered every situation that all of you collectively can think of. Once the user stories are starting to take shape, get a colleague to listen to them, looking out for gaps, inconsistencies and other problematic areas. Go over the stories looking for where the same set of actions keep occurring and consider how to handle this gracefully.

Avoid ‘blue-skying’ in meetings – summarize, don’t improvise Whatever you mention as potential future functionality, your customers will expect to see in the next delivery. In meetings with customers, therefore, make sure that you just read your notes back and get confirmation or clarification of your understanding of their needs. Don’t panic if the list of things that they want the program to be able to do seems endless. Later on, we will ask the customers to prioritize these requirements; for now, you need to persuade them to agree a cutoff date after which all new requests for features will be deferred to the next version of the program. If your customers do ask you to make a suggestion as to how something could be done, try to avoid explicit commitments to anything, and simply make sure that you clearly understand what needs to be delivered. Make a note of what they want to do, then explain that you will get back to them at the next meeting once you have explored the implications of possible solutions, when you will be able to make a recommendation.

Pencil > keyboard: screen drawings Think about that ‘>’ symbol. In a computing context, it can indicate direction, or movement. And it can also signify that what is to the left

PENCIL > KEYBOARD: SCREEN DRAWINGS

of it is greater than what is to the right of it. So, in both senses, pencil takes precedence over keyboard. You shouldn’t even think about coding at this point. Instead, take the user stories that you have created, sit down with your customers and – together – draw pictures of what will show on the user’s screen at each stage of each story. Not only will these screen drawings help you to improve your understanding of the look and feel that your customers have in mind, they will also clarify what options are available to users under particular conditions and the expected outcome of each action choice. You certainly don’t have to be a great artist to make this a success; in fact, it doesn’t even need to be you who does the drawing. If you can persuade your customers to become physically involved in putting their vision down on paper, you are also getting them to actively buy in to the project – even if they think that they are just showing where on the screen a dialogue box should go. On the next few pages are some screen drawings. They show examples of what a user might see on their monitor at various stages of a program. 1. Figure 4.1 is the user’s desktop, with the program’s icon selected. 2. Figure 4.2 is the splash screen that the users will see as the program is initialized. 3. We need to get some data from the user – Figure 4.3 is a simple dialogue box. 4. Figure 4.4 shows the time to select some configuration options. 5. Confirming the user’s selections, with options to accept or reject them (Figure 4.5). 6. Processing completed (Figure 4.6). There are a few practical points that should be mentioned here to make creating screen drawings easier and more productive. • Use pencil, not pen, so that when something needs changing you can rub it out and redraw it, rather than having to start again. If you find, as I have done in writing this book, that eventually you need proper artwork, keep it as simple as possible.

25

26

REQUIREMENTS CAPTURE

Figure 4.1 Screen drawing: desktop after program installation

Figure 4.2 Screen drawing: program start-up screen

PENCIL > KEYBOARD: SCREEN DRAWINGS

Enter text:

OK

Cancel

Figure 4.3 Screen drawing: simple dialogue box

Select an option:

Option 1: Option 2: Option 3: Option 4:

OK

Cancel

Figure 4.4 Screen drawing: selecting options

27

28

REQUIREMENTS CAPTURE

Please confirm.

You chose Option 2. Please click ‘OK’ to confirm your choice, or ‘Cancel’ to change it.

OK

Cancel

Figure 4.5 Screen drawing: confirming the choice

Processing completed! Thank you for choosing this program! © Your Organization

Figure 4.6 Screen drawing: program finish screen

YOU WILL NEED TO KNOW WHAT YOUR SAMPLE DATASETS

• Use paper to draw on, not a whiteboard – transcription errors are inevitable, unless you have access to an electronic version that can save your sketches safely. Even then, my preference is for the tool with the user interface that we are all familiar with from childhood. • Use at least A3-sized paper, and don’t put more than one story’s screens onto a page. • If possible, distribute copies of the sets of screen drawings that you have all agreed on at the end of each session. The copies will be easier to work with if you shrink them to A4, and that process will itself identify which drawings should be redone to improve their clarity. Once you have sketches of what happens when, make sure that you can trace a path along all the various possible routes a user could travel. Will all users start and end at the same place? Is every route mapped? Which paths through the program are most used and which least used? What does this tell you about the program? Do you need to do anything about it? Look for dead ends – where do you get stuck? Look for loops – where do you end up going round and round? Look for poor signposting – where is it unclear what your options are? Think about what information will be required at each stage, both supplied to and required from the user. Where is this information going to come from and go to? Where will it next be needed? Creating screen drawings with your customers will give you a feel for the shape and structure of the program, and involve them directly in the experience of designing how their application will work.

You will need to know what your sample datasets and expected results will be For instance, your task might be to make sequence data assembled using some third-party software tool viewable in two different sequence-editing programs. This means that your input data – your sample dataset – is going to be the output from the assembly program, and the expected result is that your customers will be able to run the two editor programs side by side and see the same assemblage of

29

30

REQUIREMENTS CAPTURE

sequences. Hey – we’ve already started writing the requirements specification!

Yes, now is the time to start thinking about how to test for success You should already have your primary test – the definition of successful delivery. If you don’t have that, go no further until you have. Once you clearly understand what you have to provide and can demonstrate this to your users, you have an idea of what elements of functionality will be needed. Now you can start to design tests that will prove that the individual functions, and the program overall, are performing to specification. We will look more at testing later, but now is the time to start planning it. Look at your data, in terms of largest value, smallest value, best quality, worst quality, most quantity, least quantity – this will give you a feel for boundary conditions. If your code works at these extremes, it should also operate successfully in more normal circumstances. The reverse is not necessarily true.

Documents describing staged deliverables need to be agreed to, if not formally signed off Not only do you need to agree with your customers what will be provided, you also need to agree when it will actually reach them. It is extremely unlikely that your customers will be happy not to see anything of what their money is going on until the day before the project ends, so you will need to think about when and how you can deliver functionality in stages. For example, your first delivery might consist of some sample screens of the graphical user interface, together with a proof of concept of the core functionality that runs on a small example dataset when called from the command line. Whether this particular solution is appropriate depends on your organization, your customers, and your application. We will look more at proofs of concept and prototypes later on.

GET REQUIREMENTS CAPTURE RIGHT AT THIS STAGE

Get requirements capture right at this stage, so that top-priority requirements can go into the functional specification The functional specification ‘does what it says on the lid’ – defines the things your program will do. This is what you are aiming to produce and to get your customers to agree to. But before you can produce a document that says what you will deliver, you need to be able to describe what is wanted, and that document is a requirements specification. You can write this requirements specification from the screen drawings and user stories and the discussions you have had to clarify them. Your customers should sign it when they agree that it is an accurate and complete definition of what they want. From the top-priority requirements, you can derive a Stage 1 functional specification, which will define what you will deliver first and when. Your customers should determine what is top-priority and what isn’t. However, if your customers are the sort for whom everything is a must-have, find out what they want delivered first. If the answer to that is also ‘Everything!!’, then the first functionality you deliver is the first feature that you can complete. We will look in more detail at this process of developing a requirements specification and, from that, a functional specification, in the next chapter. For now it has to be said that if a requirement is not toppriority, then it is probably no-priority. By the time Version 2 comes around, there will almost certainly be a whole slew more of toppriority requirements – the dreaded ‘feature creep’. So it goes; but if you have a signed-off functional specification, then you can postpone requests for new functionality until the next version. Without it, you either have to take on more work than you have planned for, or you have to say ‘No’ to customers; would you want to be forced to make that decision? So, to sum up: it is worthwhile spending plenty of time capturing requirements accurately at this stage of the development process. Getting it spot-on now costs an awful lot less than trying to fix it later. And you cannot deliver on time, and within budget, without an agreed specification. In the next chapter, we will go on to look at how we get there from here.

31

5 Separating Function, Interface and Implementation This chapter will help you write a functional specification from given requirements. It looks in more detail at the differences between function (what your program has to do), interface (what its users will encounter), and implementation (how you get it all to work). It discusses how you can gradually set what you will deliver down in writing, and how this process helps you clarify how your program will interact with its users and its environment. It also explores which issues are more a matter of how you implement something, not what you implement.

User stories will tell you most of the functionality that is required You will need to go through each user story, identifying the different types of words used, each of which gives you different information about the task ahead. • Look for verbs – action words, such as ‘does’, ‘is’ and ‘views’. • Identify nouns – naming words, like ‘user’, ‘home’ and ‘sequence’. • List the adjectives – describing words, for example ‘quick’, ‘simple’ or ‘precise’. Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

34

SEPARATING FUNCTION, INTERFACE AND IMPLEMENTATION

The verbs are the functions that must be provided by your application. The nouns define the parameters to these functions, and the adjectives specify the constraint conditions under which your program must operate. Using the sequence assembly example from earlier, let’s say one of your user stories contains the sentence ‘User assembles sequences, and sees the identical assembly in two sequence editors’. So, this sentence contains two elements of functionality – assemble and see; two groups of parameter information – user and sequences; and a constraint condition that must be fulfilled – identical. As you go through the user stories, make notes under these three headings – function, parameters, constraints – for every action in the story. For the assemble action above, your notes might look like this:

Action

Parameters

Assemble sequences together Location of sequence data, user name

Constraints Two editors must display identical assemblies

So, from an informal narrative we are already moving to a more formal definition of what operations your program must perform, what information it will be working with, and what conditions must be satisfied to achieve success. Now, let’s just pause a minute and look at what we have here. This particular action – assemble sequences – is a high-level description of what we need to do. Can we break it down into steps? We can, by using some very basic techniques. First, ask yourself what could go wrong when your program tries to perform the action. Let’s consider the parameters. What happens if a parameter is missing? How can we check whether a parameter is missing? What happens if a parameter is wrong? Now we can immediately see that we need to break this high-level action definition down into a more detailed definition of how it is to be performed. Before the program even starts to do what the user wants, it has to do a great deal of processing that its users may never see. Without this ‘behind-the-scenes’ work all that the users encounter is a hollow shell that will break immediately anyone tries to do any real work with it.

USER STORIES WILL TELL YOU MOST OF THE FUNCTIONALITY THAT IS REQUIRED

Let’s revise our action definitions accordingly, and turn our table into a description of what our program will do. We can say that when the program starts, it will check that it has been given the location of the data that it is to process. What will it do if this is not the case? Probably the best thing for the program to do here is inform the user that they need to say where the data to be processed can be found, and give the user a way of getting this information to the program. We cannot go any further until this is received, so it may be that we also need to give the user an opportunity to quit the program at this point. Once the program has been told where the sequence data can be found, our problems are not over. The user may have wrongly specified the location, so that the path to the data is invalid. So, we need to check that the location specified is valid, and that it contains data. Then we need to check that the data is of the right type to be processed, and of good enough quality to produce usable output. At each stage of this process of validating the supplied parameter, we need to have some way of coming back to the user if a problem is found, and allowing them to provide the program with a solution to the problem. Let’s try listing the stages involved in checking this parameter. 1. Check that the data location has been specified. If not, ask the user to specify the data location, until the user gives this information or quits the program. 2. Check that the program can access the location of the data. 3. If not, explain to the user that the program cannot find the given location, and ask the user to check and, if necessary, correct it. Continue doing this until a valid data location is given, or the user quits. 4. Check that the location given contains data. 5. If not, inform the user that the specified location contains no data, and ask them to re-enter the data location. When a new location is given, check again that it is valid, and that it contains data. Offer the user an opportunity to quit at this point as well.

35

36

SEPARATING FUNCTION, INTERFACE AND IMPLEMENTATION

6. Check that the data is of the appropriate type. 7. If not, inform the user, and give them the opportunity to specify a different location, or quit. Once the location is given, repeat the previous checks. 8. Check that the data is of suitable quality. 9. If not, inform the user, and again give them options to take appropriate action, repeating the above checks as required. Now we are beginning to get to the level of detail that will be required for a functional specification. It is also clear that if the user supplied a valid path to good-quality data when the program started, steps 3, 5, 7 and 9 will not be needed. Some might therefore argue that writing code that might never be called is a waste of time, and that instead we should be concentrating on the program’s core routines. However, if you don’t build this sort of functionality in at the start, you will be creating a program that will crash gracelessly whenever it encounters anything less than perfect data, perfectly supplied. That is not a risk I would like to take. My experience is that even if you code in routines to handle every problem you can anticipate, your customers will always try to do something that you would never have expected them to do. You will be busy enough dealing with that. Therefore, if you can see a possibility that something might be problematical, design in a way of coping with it. We have spent some time looking at how we can go from an action specification, itself derived from a user story, to a description of functionality. You will need to repeat this process for all the actions in all the user stories; once you have done that, you will have written the bulk of the functional specification. As you are generating it, always remember to ask yourself: • What can go wrong here? • How am I going to cope with it? If you can answer these two questions every time, you will be doing better than I have ever managed. Still, aim for the stars and you might reach the ceiling; aim for the ceiling and you will never leave the floor.

YOUR IMPLEMENTATION HAS TO CREATE

And now that we know how to create a functional specification, let’s look at how to define your program’s user interface.

From screen drawings to screens Screen drawings are the first stage in developing the user interface. As you work from them to define the functionality required, you may find that you need to go back and create more drawings – to illustrate, for instance, what the users will see if the program cannot find any data to process. You can use the screen drawings to identify particular elements of the user interface. Let’s take a look at what you might encounter. • There may be buttons, where the user has a limited choice of options, such as going ahead or cancelling an action. • There may be data fields, where information is either displayed to the user or received from them. • There may be menus, where the user’s choice of action is more open-ended. • There will almost certainly be action sequences, defining what the user will see as the result of selecting a particular option. Remember that the program’s user interface is often a major part of your customer’s vision, and make sure that (for example) dialogue box text is worded appropriately. Also, you will need to go back and ensure that all the actions in the user interface are correctly defined in the functional specification.

Your implementation has to create and connect the interface and the functions This may seem daunting, but no worries – you’re more than halfway there already, if you think about it. Cheer yourself up by reflecting that you’re a lot better off than you would be without what you now have – a full description of what you need to do in words and pictures.

37

38

SEPARATING FUNCTION, INTERFACE AND IMPLEMENTATION

Once you have this working definition of the required functionality, from the user stories, and an idea of the various aspects of the interface, from the screen drawings, you can then begin to think about the best way of implementing what has been asked for. Deciding how to do it will be to a greater or lesser extent determined by what you have to do. Equally, your choice of how you implement something will affect what you can do; for instance, it may mean that instead of using system calls you can use a built-in function or a third-party code library. Some of these decisions will be more obvious that others; for instance, a web-based interface means that at some point you will probably need to write – or dynamically generate – Hypertext Markup Language (HTML). Equally, if you are going to store data in a relational database, it is likely that you will need to access it using an appropriate query language, such as Structured Query Language (SQL). Beyond this, your choice of development tools and techniques is likely to be influenced by many considerations, such as the environment that it will run in, the way users wish to use it, and the custom and practice of your organization. In the next chapter, we will look in more detail at some of the things to consider when thinking about choosing the tools to get your job done.

6 Implementation Considerations

Here we will look at ways of building bioinformatics tools, and examine some alternative approaches. It may be that the decision as to how to implement your program is taken out of your hands – your organization may be committed to a particular way of writing software. However, if the choice is yours, there are many factors to consider.

Languages Choosing the programming language that you will use is probably the first implementation decision you will have to make, and possibly the most important. You can use almost any language to write bioinformatics applications, and this book does not assume that you are using any particular language, platform, operating system, or development tool. Having said that, you should bear in mind that in bioinformatics code rework is usually ongoing and it is unusual if all system entities can be defined at the outset. Make sure that your chosen language is flexible enough to cope with this challenge.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

40

IMPLEMENTATION CONSIDERATIONS

Platforms It is quite likely that your application will run on, and be delivered to, different hardware configurations. This may be a case of a server connecting to a variety of client machines, or a requirement that different CPU types be supported. There are some points that you should bear in mind when considering platforms. In general, it is always good practice not to unnecessarily optimize your software for a particular hardware configuration, and much better to ensure that your code is as platform-independent as possible. We will look at tool selection considerations in more detail later in this chapter. However, if your development work will be taking place on different platforms, as often happens with client–server applications, the tools that you choose should either run on all platforms, or be best of breed for that particular type of hardware. If you do have specific hardware configurations on which your client software must perform at least adequately, then you will need to build an independent network of appropriately set-up machines for testing purposes, and all tests will have to be run on all hardware profiles. If you also have to support multiple operating systems on the different hardware setups, managing the testing process can become complex. Writing scripts to automate test runs will help. You should also keep copies of datasets that cause you difficulties in particular circumstances, so that you can make sure that any future changes to the software will cope with these challenges. If you have to support multiple platform configurations, first tests should be on the machines with the lowest specification – earliest operating system, oldest processor, least memory, smallest storage, slowest speed. Code that runs on the most basic hardware should not encounter problems on more sophisticated platforms; sadly, however, we do not live in an ideal world.

Operating systems Operating systems are a factor that needs to be considered from at least two points of view: the environment in which your application will be developed, and the one(s) in which it will run. However, there is a third consideration, of much greater importance, and that is where you have to work.

OPERATING SYSTEMS

Whatever delivery platforms you have to support, you need a stable environment in which to create code if you want to build effective applications. As a bioinformatics software developer, I prefer leaving operating system maintenance to the specialists. And I don’t enjoy having to spend time dealing with – or working around – functionality that is missing from a particular development environment. As previously mentioned, this book is not biased towards any particular coding environment. I would suggest, however, that if the choice of operating system that you use is up to you, then you should choose a way of working that helps, rather than hinders, both your creativity and your productivity. When it comes to the environment in which users will encounter your application, this may be defined as suggested in terms of operating system types and versions that must be supported, or of protocols and language specifications that must be adhered to. Whatever the way in which these environmental requirements are specified, it is in your best interest, and that of your customers, to make sure that they are clearly understood by all parties before you start work. Retroactively applying compatibility is not a task to be undertaken lightly. If, say, you have to test a web-based application’s user interface using different browser types and versions, with support for common embedded scripting languages, on different flavours of the various available operating systems, and you have already committed yourself to deliver on multiple hardware configurations, testing is going to take considerably longer than development, and most of your development team will be working on compatibility issues, rather than core functionality. How can you deal with this? Well, you can make sure that your customers are aware of the situation, and the issues that it raises. Try to encourage them to focus on ensuring that your program works on the most basic versions of its environment, because this means that (for example) problems with later versions of a particular web browser may well not be caused by your application, but by the browser software. Unfortunately, proving that a third party has failed to make their software backwardly compatible may be more of a challenge than you and your colleagues need, making it time to say ‘hello’ to Mister Workaround again. Keeping things simple can greatly ease the process of development and testing. If you can design a user interface using only technologies that are supported on virtually every platform by almost all operating

41

42

IMPLEMENTATION CONSIDERATIONS

systems, you can focus on making what lies behind that interface even better. If your customers insist on using a technique that is only supported on the latest platforms, your testing requirements may be more straightforward; the downside of this is that you may end up debugging the technology itself. And if your program requires a specific technology to be supported for it to work properly, or which necessitates separate platform-specific implementations, it is your responsibility as a professional developer to explain to your customers why this will have such a negative impact on the speed at which you can deliver the functionality that they are asking for.

Prototyping tools You may decide that you will produce prototypes using a different tool from that which you will use for the full implementation. For instance, you may use a web-page generation tool to build mock-ups of an interface that will eventually be created on-the-fly by your code in response to user actions. When you are selecting a prototyping tool, you should bear in mind that choosing a particular tool may limit your choice of development platform, language and environment. It is also important that the chosen tool allows you to get started without knowing everything about the system that you are going to build. It is helpful if the interfaces that are generated by the prototyping tool can be used for the production application, but it is extremely dangerous to simply assume that this will be the case. For a web-based application, the first-draft user interfaces can be created in HTML, which can later be separated into static and dynamically generated code.

Debugging tools Few programmers can write non-trivial bug-free code straight off, and choosing the right tools to help you find and fix problems can make your task as a developer much easier. Your choices when it comes to debugging tools will be determined primarily by the environments in which you develop and in which your

DEBUGGING TOOLS

program will run, together with the conditions that are imposed by local custom and practice. Whatever tools you use, what is important is thinking about what you need to do right now. If you think back to when you were learning to program, you used various techniques to help you fix your applications when they weren’t working. While bioinformatics software may sometimes be more complex than that code, the same simple principles apply. Debugging starts before you code anything. For example, don’t write code for which you cannot define success or failure, as you will never know if it is doing what is needed; but the definition of success could be analogue, rather than binary. For instance, a requirement that ‘results must be accurate to within 0.02 per cent of target in at least 95 per cent of tests’, means that it would be possible to begin writing this code and then work on optimizing its functionality to the desired degree of accuracy. However, with success defined as ‘100 per cent accuracy every time’, or ‘right’, it would make sense to ensure this is provably achievable before you begin implementation of any algorithm. You need to know what should be happening, so that you can check that it is happening. You may have a usable definition of success, but to achieve that success you have to know what your code has to do, and how – for example, retrieve relevant data from storage within a specified time period under defined load conditions. This means that before writing the code to perform this operation, you need to think about how you could tell what it was doing. So, for example, your code should tell you when it is about to make a connection to the data store, as well as whether or not it succeeded. This kind of coding can make it easier for you to see how your program is running, and how far it got before it stopped, which helps you to identify where in the code you need to start looking for the bug. Remember, the more work you do now on preventing bugs creeping in unnoticed, the less you have to do when they become apparent during testing. Some problems require the use of apparently more sophisticated techniques than the use of print statements. Code checkers, whether external, built-in, or supplied as optional language extensions, may well be appropriate at various times, depending on the nature of your application. However, in this situation your development efforts will also be greatly helped by tools that can, for example, tell you what the load

43

44

IMPLEMENTATION CONSIDERATIONS

on the system is when your code is running, so that you can determine if your application is functioning according to specification. Debugging tools can also be development tools. Many very sophisticated tools are available for you to make use of, but you should never forget the most helpful one – common sense. If you do hit a problem, don’t panic. You already know what you should do in this situation, if you stop and think about it. Everybody, including you, is very good at solving problems, having started with ‘I need to indicate I’m hungry, tired, thirsty and in need of a hug NOW’ as soon as we come into this world. Let’s take a simple example problem – a flat tyre. How can we fix this bug? First, have we forgotten anything? Did we remember to pump it up? Then, when we pump it up, does it stay inflated? If not, we need to get deeper into the problem. We take off the wheel, remove the tyre, and check the tube. Does the valve leak? Okay, then we need a new tube. Is the tube punctured? Then we need to patch it. But we also need to check why the problem occurred, so we can make sure it doesn’t happen again. Is the tyre worn, for instance? Or have we picked up a thorn? Or imagine a car won’t start. The first thing to do is look at what is still working on it. If the lights come on, then you know it’s not the battery that’s at fault, so you go on to check the rest of the ignition system, and so on. In this way, you break each problem down into manageable steps, and go through each one until you reach a solution. Don’t presume that a variable contains a particular value – check it, either by printing its contents or inspecting its value. And there is nothing like trying to show someone else your problem to make you realize what it is that you need to do next, or what you should have done earlier. Situations that you are also likely to come across when debugging include seeing what you think should be there, but actually isn’t, and cascade errors, where the visible effect is a long way downstream of the original cause. Most of the time, it’s the data. Start there.

7 Proof of Concept, Prototyping and Buy-in

Here we will explore how you can explain the development process to your customers, and why you should do this before showing them anything you have done. We will discuss ways to increase their confidence in your ability to deliver what they want, and see why this is vital at this stage. Then we will go on to look at the evolutionary or adaptive approach to software creation, the particular advantages of this methodology in bioinformatics, and how you can use this approach to keep your customers on your side during development.

Explaining the development process with prototypes and proofs of concept It can be hard explaining the software development process to people who are unfamiliar with it. Code that to you is nearly finished is simply not working to them, and seeing their dream in bits on the workbench can be disappointing to customers, especially when they were expecting to be able to take it for a test drive. When an application consists of many complex interconnected parts, or where the proof of concept makes up the bulk of the core functionality, it is difficult to demonstrate anything until you are nearly ready to demonstrate everything.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

46

PROOF OF CONCEPT, PROTOTYPING AND BUY-IN

And when customers are paying a lot of money for what you are building for them, it is extremely useful to be able to give them something more as positive feedback than reassuring noises. With user stories and screen drawings, you have something to show your customers from the moment you begin detailed analysis of their requirements, and you can build on this and encourage buy-in to the project with proofs of concept and prototypes. Just as you can prototype a user interface by creating (for example) a series of mocked-up screens that show one path through the program, so you can use the same principles to prove to your customers that the functionality they require is actually deliverable. To do this, try to identify some essential piece of processing that can be separated out, and, just as you mocked up a static version of what the users will see, attempt to determine whether you can show that the results they want are achievable, at least on a minimal subset of data in one particular set of circumstances. For example, you may need to demonstrate that your program can work effectively with a particular format of data. Get hold of a small set of appropriate examples, and missing out all the subsidiary steps – data validation, error handling, post-processing, result analysis – concentrate on nursing that data though your core transformations. It may be that you have to run the data through one step at a time, and perform manually many of the tasks that you will later automate; the important point is that you are able to show your users input data and output results which prove to them that you will be able to give them what they are asking for. Moreover, you will then have code that you can adapt into your final deliverable, and both you and your customers will have increased confidence in your ability to get the job done. As well as this, it may be that you discover that there is a mismatch between what you think they want and what they think that they are going to get. The earlier that you can determine whether or not this is the case, the better for all concerned.

‘Too much information! Too much information!’ You need to be careful not to give your customers more information than they need. What the eye doesn’t see, the heart doesn’t grieve over. It may be that at one particularly difficult stage of the development process, you find that you are wrestling with an implementation challenge which threatens to jeopardize the success of the entire project.

MARKETS: HOW TO SELL YOURSELF AND YOUR WORK

Yet, in a week or so, it is more than likely that you will have dealt with this problem and will have moved on to deal with others. Generally speaking, the time to tell customers about problems, if ever, is when they have been solved; we will look later on at particular situations that are exceptions to this rule. When it comes to explaining what it is that you are trying to do, the more you yourself appreciate the intricacies of the task of application creation, the more you will find it getting easier to give an overview of the process to others. If you don’t yet feel up to this, then consult colleagues with greater experience; people like to feel that their opinions are valued, and will be happy to help you benefit from what they have been through. They may find it easier to describe what you are trying to do in terms that your customers may find easier to understand. Keep in mind, though, that even understanding something completely is not enough, if you cannot then communicate this understanding to your users and customers. If in doubt, keep quiet. One thing to be particularly aware of is that when you are dealing with the people for whom you are writing software, you should be conscious all the time that they are viewing you as a seller. We will explore this more in Part 3, where we will look in more detail at how to present work in progress to customers. For now, even if you work in a not-for-profit environment you should understand something about markets, if only because you are involved in one already – the labour market.

Markets: how to sell yourself and your work, and why you need to Where you have a buyer and a seller, you have a market. You sell your labour for reward, financial or otherwise. It is sometimes difficult for technically oriented people to appreciate that the better you are at dealing with those who buy your labour, the greater the rewards you can receive. There are a few simple things that you can easily do to get better at dealing with market situations. You will find it easier to sell something if you know someone who wants to pay you what you want for it. So, if you can answer these three questions, you will be much more confident at selling yourself: 1. What are you selling? Think about why anyone should pay you for what you can do. Try to look at what the benefit is to them.

47

48

PROOF OF CONCEPT, PROTOTYPING AND BUY-IN

Then try to put it in words that they could use to persuade someone else to hire you: not just ‘I write good code’, but ‘I have a track record of successfully delivering user satisfaction in a complex and demanding environment, and I can prove this’. 2. Who wants to buy it? Be realistic. There may be many potential customers out there; think about the ones you can reach easily – the easy pickings are what you start out on. Don’t spend too much time and effort on people who don’t want to buy, but always remember that though they are not buying now, they may want to do so soon – just as soon as they understand what you have to offer, and realize what they could do with it. 3. How much? In general, things are worth what someone will pay for them. The important point to make here is that you can make people realize that you are worth more than they thought, if you can make it clear to them how good a fit there is between what they need and what you can provide. I would be less willing to hire someone who sold themselves as ‘a programmer’, and more interested in the candidate who had demonstrably analyzed their strengths and matched them to the job description. Going through this process for yourself will better equip you to do something on similar lines for your customer’s product – what it is, who it is aimed at, why they will prefer it to the alternatives. If you understand the product which you have been asked to build, and the market it is competing in, then you are able to demonstrate to your customers that your ability to deliver what they want will not be compromised by any misunderstanding of the nature of the job in hand. So, think about how these questions might be answered by everyone involved with the project. For example, your employer buys your labor and sells your services; your customers buy your product and sell it to end-users. What are their motivations, their drivers, their priorities?

HyperCard, genetic algorithms, evolutionary development and ‘doing science’ Obviously, there is more to life – even working life – than markets. Sometimes it can be useful to step further back from the hubbub and bustle of the everyday, and look at your situation as a bioinformatics application developer from a different perspective.

HYPERCARD, GENETIC ALGORITHMS, EVOLUTIONARY DEVELOPMENT

‘When I was a lad, all this was fields’, in a manner of speaking. Bioinformatics is a young discipline, and I suspect my background and history have made me an unconventional practitioner. It’s a funny old game, bioinformatics, as is science itself, especially to someone who comes at it sideways. I’m old enough now to have seen fashions come, and go, and be revived. I first encountered a widely available, easy-to-program, objectoriented user interface development toolkit with the release in the late 1980s of Apple’s HyperCard, and I used its elegant HyperTalk scripting language for my first attempts at programming using genetic algorithms. These make use of models of evolutionary operators – such as reproduction, mutation, crossover, and selection by fitness – applied to a population of candidate solutions. Some papers have been published exploring the applicability of genetic algorithms to sequence assembly. Other interesting work has focused on genetic programming, where the aim is to evolve not just a solution, but the program code to solve a problem. I have been considering evolutionary strategies in application development – if you like, the other side of adaptive programming, the one that doesn’t involve typing. At its simplest, this just means appreciating that software is created, grows, interacts, and adapts or is adapted to changing circumstances, like any other thing that exists. My interest in how genetic algorithms can be used to evolve optimal solutions to complex and changing multi-parameter problems has also provided an interesting framework for observing how our customers in the world of bioinformatics application development – the scientists – actually do science. One can look at the scientists in a particular field in terms of a population of individuals, each of whom is adapted to their circumstances to a greater or lesser extent. Those individuals who are best suited to the prevailing conditions will do better than others; their fitness, measured by such factors as publication record, history of successful collaboration, development of innovative techniques, or ability to attract external funding, will result in their becoming more influential in the field. However, circumstances can rapidly change, and characteristics that are valuable in one situation can become liabilities in another. Thus, over time, the population of scientists as a whole demonstrates different features in accordance with the changing demands of their environment.

49

50

PROOF OF CONCEPT, PROTOTYPING AND BUY-IN

Some things persist. Certain techniques, skills and attributes – analytical strength, the ability to explain how results are arrived at, the imagination needed to come up with novel solutions – are less subject to the whims of fashion and the winds of change. And in bioinformatics, as in application development as a whole, core skills such as being able to understand and describe complex and vaguely expressed requirements, and to adapt your skill set to new circumstances, are likely to be of benefit, whatever the latest orthodoxy dictates.

Using adaptive development to encourage customer buy-in One major advantage of the adaptive approach adopted here is that you can make frequent informal deliveries, or explanations and descriptions of where you are in the development process, virtually from the beginning of the project. We have already looked at how you can increase customer and user involvement – and, in consequence, their commitment to the project’s success – through the joint creation and development of user stories and screen drawings, and by the building of prototypes and proofs of concepts. In order to continue and enhance this, you should be planning during this pre-development stage the ways in which you can take advantage of how you do adaptive development, in order to ensure that your users continue to keep involved with the development of their application. Long periods of no contact are not good for anyone. It is dangerous to adopt an approach where once we have agreed requirements and specified the functionality to be delivered, our customers will see nothing until the delivery date. Instead, we can ensure that our customers keep feeling involved in progress with mock-ups of screen drawings, sample results from proofof-concept code, outline test plans, and prototypes of functionality, and by continuing to do this throughout the whole of the project. Before delivering anything to your customers, check that what they see is not going to disappoint them. You will need to make sure that they understand that a prototype exists merely to demonstrate a limited subset of functionality, in order to ensure that it is being implemented

USING ADAPTIVE DEVELOPMENT TO ENCOURAGE CUSTOMER BUY-IN

in a way that is acceptable to the customer and to identify any changes that need to be made as soon as possible. You will also need to manage their expectations, so that they don’t anticipate a finished product until the final delivery, and you will have to explain to them that what they receive until then may not necessarily have been exhaustively tested. We will look more at this matter later, but you should be aware that you need to start planning these deliveries, and considering the impact of the issues involved, right now.

51

Part 3 Getting it Done

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

8 Data in, Data out and Data Transformation

In the previous parts of this book, we have been looking at what you need to do before you start work in order to be able to deliver effective bioinformatics applications. Now we will move on to examine the next stage – designing and developing your program, moving from specification to execution. If you make sure that your design is clear before you start writing code, then you can ensure that you are building what is needed, and only what is needed. This greatly increases the chances of your being able to give your customers exactly what they want when they need it. This chapter suggests a useful way of looking at what your code has to do, by adopting a data processing perspective. It describes two simple techniques that you can use to illustrate both what will be happening as your program runs and what information it will be working on.

Begin, process, end: process flow diagrams Process flow diagrams are like a map of the screen drawings that you create with your customers. They focus on the decision path. Developing them is simply a matter of asking what alternative actions can be performed at which point, and what happens after each choice. There are three different elements to a simple process flow diagram (Figure 8.1). First, there is the basic process. Then there are its con-

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

56

DATA IN, DATA OUT AND DATA TRANSFORMATION

Assemble

Initialize

Process

Deliver

Figure 8.1 Process flow diagram: linear process

stituent parts, and these are connected to each other and the process of which they are a part by lines. The name of the process – in this case, ‘Assemble’ – goes in the top box. The stages that we can divide the process into – initializing, processing and delivering – are shown in the boxes below. Each task is carried out once only, and in the order, left to right, that the boxes are shown. What do we do if we need to show that an action is performed more than once? We put a star in the top right-hand corner of the box, to indicate that this process takes place a number of times, and we note to the right of the box how many times the process needs to be performed. Figure 8.2 shows an example where a file is being processed, and every line in the file needs to be read in, until the end of the file is reached. Now that we can model repetition, there is another element of any task that we need to model – the making of decisions. The choices that can be made in a particular situation are represented in the same way as the basic linear process and the repeated process considered above, but with the difference that in the top right-hand corner of each option is a small circle. Figure 8.3 shows a simple choice: when we try to open a file, we exit if it can’t be opened, otherwise processing continues normally. Both selection boxes and repetition boxes can involve sub-processes, as we will see in the next example. In Figure 8.4, if the program has been given a valid path to a data file, it will read the file and check each line’s contents until the end is reached or unexpected data is found.

BEGIN, PROCESS, END: PROCESS FLOW DIAGRAMS

Read File

Open File

Read Lines

* Read Line

Close File

Until End of File

Figure 8.2 Process flow diagram: repeated process

Open File

O Exit

If Fail

O

If Success

Continue

Figure 8.3 Process flow diagram: selection process

There are a couple of points to note about the example overleaf. First, the use of an exit box is optional. One alternative would be to design the process flow diagram so that (for example) ‘Process Lines’ descended from the ‘Continue’ option of ‘Open File’. Instead, you can

57

Exit

If Fail

Open File

If Success

Exit

Read Line

If Unexpected Data

Figure 8.4 Process flow diagram: complex process

Continue

* Process Line

Process Lines

Process File

Continue

Check Data

If Expected Data

Until End of File

Close File

58 DATA IN, DATA OUT AND DATA TRANSFORMATION

BOXES IN BOXES: DATA STRUCTURE DIAGRAMS

see that I have arranged the major sub-stages of the program – open a file, process its contents, close it – in a sequential horizontal line at the top of the diagram. I prefer the method used here as I feel it makes the logic clearer. Secondly, there is more processing that could be added to this diagram, such as how to handle failing to open the file, and how to compile the report from the data read in. We will look later on at creating flow diagrams for these processes. But it is important to be clear that by drawing process flow diagrams to represent your understanding of what is happening at the various stages in the program, you can break down the required functionality into sections, each of which has a beginning, a middle and an end. And each of these sections can be made up of further sub-units of functionality, which themselves may have a beginning, a middle and an end. Breaking down the work to be done into its component parts like this should continue until you are happy that the processing to be carried out can be clearly defined. In the next chapter we will go on to look at pseudocode, which is a simple way of describing the processing involved at each stage of a program. The reason for this is that one good test of whether your analysis is adequate is trying to turn your diagrams into the minimal subset of natural language that pseudocode is written in. You may well find that you don’t need to do this much analysis for all areas of the program. However, even if you decide not to do process flow diagrams for the whole system, it is a very useful technique when planning the implementation of more logically complex modules or code sections. So, now that we have seen a way to graphically represent what our program is going to do, let’s explore how we can clarify what will be going into it, and what will be coming out, by drawing pictures of its data.

Boxes in boxes: data structure diagrams Just as we can draw how your program will work, so we can model the shape of the data that your program will need to deal with. Just as a process flow diagram defines a beginning, a middle and an end for each processing element, so data structure diagrams describe for each stage what data goes in, what data comes out, and thus how it is transformed.

59

60

DATA IN, DATA OUT AND DATA TRANSFORMATION

Let’s return to the sample program that we looked at briefly earlier, to work with sequence assemblies. First we will look at how we can use data structure diagrams to describe the data that it will be dealing with, and then we will examine how we can use process flow diagrams to show what happens to that data. Remember that there’s no point in writing code if you don’t know how to prove it works, and equally there’s no point even reading in data unless you know what data to expect. So, let’s make sure we know what data will be involved in (for example) reading in the details of an assembly of sequences into contiguous sub-sections, and summarizing the information that we discover into a report. Each assembly will be made up of a number of groups of sequences of individual DNA bases, represented by a long string of letters, like ‘ACTTGGTCCAATTGGCACAC’, that have been assembled together. These groups are known as contigs, because they are contiguous – each sequence in a contig contains one or more sections where the arrangement of its bases matches a section of another sequence. If you’re not a molecular biologist, think of a painting of some flowers. It will be made up of lots of small dabs of paint, of various colours, shades and thicknesses. If we then group a number of these paint marks together, we may have a leaf, a shadow, or a vase. And if we put all of these groupings together, we have a masterpiece, especially if your own child created it. The sequence data is like the individual paint marks. The contigs – the groups of sequences that go together – are like the leaf, the shadow and the vase. And the assembly is what we get when we put all these parts of the picture together. Let’s just slightly rephrase our definition of how our data is structured. • Each assembly is made up of contigs. • Each contig is made up of sequences. • Each sequence is made up of bases. We can represent this graphically, by using a simple crow’s foot symbol to represent ‘is made up of’. Take a look at Figure 8.5, which shows an assembly of contiguous sequences. This is a very simple representation of the data that we will be dealing with – we’ve not even shown the bases. In a moment, we’ll see a more

BOXES IN BOXES: DATA STRUCTURE DIAGRAMS

Assembly

Contig

Sequence

Figure 8.5 Data structure diagram: simple assembly structure

exhaustive way of representing the data we’ll be working with; for now, just make sure you understand what we’ve done here. Try doing a data structure diagram of your favourite painting, for example. Remember, to produce these drawings you don’t need any fancy software, merely a handy 1-bit analogue portable recording device, or pencil, and the world’s most popular information transmission medium, paper. And with the addition of an eraser, we have an infinitely adaptable way of capturing ideas, whether in the form of user stories, screen drawings, or data structure and process flow diagrams. You may wish at a later stage of the project to convert these pencil sketches to more formal representations for the purpose of documenting your application, as I have done for my publishers. However, the important thing for us is to capture an understanding, shared between ourselves, our customers and our program’s users, of what information the application needs to be able to cope with. The technique illustrated here is simple and clear enough to be understood by customers, especially if they work with this type of data. Representing their data like this helps to ensure that you and they are in agreement about what will go into, be processed by, and come out of your program. Furthermore, the act of trying to create data structure diagrams can often lead to clarifying discussions that can help both you and your

61

62

DATA IN, DATA OUT AND DATA TRANSFORMATION

Assembly

1,n Contig Count

Contig

Contig Index

Length

Sequence

Sequence Index

Sequence Name

Position

Position Index

Base

Tags

Repeat

Vector

Assembly Length

2,n Average Depth

1,n Average Quality

0,4 Quality

0,2

Sequencing Vector

Poor Quality

Cloning Vector

Figure 8.6 Data structure diagram: complex assembly structure

customers reach a better understanding of the details of the work that you will be doing for them. For example, we showed a simple diagram of how an assembly of is made up of groups of contiguous sequences of bases. Let’s expand that into a more accurate – though arguably still incomplete – representation (Figure 8.6). Note that this version shows not just ‘one or more of’ relationships, but it is more precise: either ‘only one of’ or ‘between (a number) and (another number, possibly unknown) of’ connections are used.

9 Where to Start?

Having looked in the previous chapter at how to make sure that you understand in detail what your program has to do, and how, we are now going to look at what to do when you are unsure where to begin your implementation. Now this may be because parts of the specification or design still need more work, and you may feel that you would rather wait until all these stages are fully complete. However, in bioinformatics, this is often not a good idea, as scientists tend to always think of new things that they would like your program to do, as soon as they see what is possible. And if you are producing prototypes and proofs of concepts, you will almost certainly be analyzing, designing and developing in parallel. We will look later on at how acquiring this capability may well be helpful later in the development process; for now, let’s just say that as soon as you feel confident that you know how to do something, it’s a good idea to get to work on it. So the first step in starting coding is to determine in your mind what you know, and what you don’t know. You will have to wait to find out what you don’t know you don’t know, but that will become clear soon enough: for now, just deal with what you know you have to do. There is some basic functionality that you will need under almost all circumstances, so let’s think what might be involved in implementing it. Then we will explore how we can go from the information we have, to pseudocode (if required), and end up with working code.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

64

WHERE TO START?

You will need to get in data Few programs have no need for data to be supplied to them, so it is likely that you will need to plan how to accept user input, read files, or handle messages. You will need to check that this data has actually been supplied, that it is of the right type or format, and that it contains information of usable quality. Start as simply as possible – get in data and display it. Then think about how you can check that it is the right data, and if it is, what you need to do with it. • Does it need to be passed to another part of the program? • Does it need to be reformatted, or modified, or transformed in some other way? • What will you do if there is a problem? These questions may highlight areas where, for instance, we need to add more detail to our designs, or we may find things we hadn’t thought about. But if you’re wondering where to start the actual programming, take another look at that last question. The second code section in nearly every new program of mine handles reading in the data it needs – but the first code I write handles what happens if I forget the program’s calling parameters, or don’t tell it where the data is, or give it the wrong type of data.

You will need to cope with errors, because they will happen Whatever can go wrong, will. So, therefore, for any program that takes a non-trivial amount of time to run, try designing error-catching code that sends you an email and/or prints to a log file. If you are writing proof-of-concept code, or developing a module in isolation, this may be overkill at first, but as your code grows in size and complexity it will become more and more necessary. Start by handling errors that you can deliberately create – try to open

YOU WILL NEED TO REPORT RESULTS

a file that you know is not there, for instance. What does your program do in this case? • Does it stop running silently, with no indication of what has gone wrong? • Does it keep going for a while, then crash later on when an uninitialized variable is encountered? • Does it give an error message, saying ‘Something has gone wrong with the program’? • Or does it politely inform the user that it is unable to find the specified file, and give them the opportunity to correct the situation? Diagramming how you handle this situation can rapidly become a nontrivial challenge, particularly if you consider cases where an error can be repeated, such as when a user keeps specifying the wrong type of data file, or compounded, where the user becomes flustered and starts pressing keys at random. A simple solution is to show it in a repeat loop until the user exits or corrects the errors, as in Figure 9.1.

You will need to report results After error handling and data gathering, it is time to write code that displays the required output. In the beginning, make this as straightforward as possible, and just print your results out to screen, to email, or to a log file, until you are happy that they are valid. Then – and only then – should you start to think about more complex and sophisticated output presentation techniques. At this early stage of development it is sometimes useful to combine the result-reporting mechanism with the error handling discussed above, so that one simple code section is responsible for giving all messages to you. You will also need to consider what sort of results you will plan to deliver at each stage of the development process. For example, proof-of-concept code may get away with producing correct output from a given dataset without falling over. But as your application develops and matures, it is likely that your customers may

65

Exit

If Yes

Exit Selected?

Continue

If No Continue

Restart

If Yes

Report Error

If Yes

Figure 9.1 Process flow diagram: looping until success or exit

Continue

If No

Valid File?

Valid Filename?

Process File

Report Error

If No

Return Filename

66 WHERE TO START?

FROM DESIGNS TO PSEUDOCODE

want other information provided back to them – how much data was processed, how long the processing took, how many resources were consumed, and other administrative, analytical, or system management details. But, for now, let us begin here by looking at our simple program, and think how we would generate a summary report of the data that we have read in – giving us the number of contigs in the assembly, for instance. Figure 9.2 is an example of a high-level process flow diagram. It contains elements that we have already diagrammed in more detail, and itself could make up one part of a larger application, like a complete laboratory information management system.

Known and unknown coding During the programming stage of the development lifecycle, it is likely that you will find yourself involved in two different types of coding, which I refer to as ‘known’ and ‘unknown’. Known coding is where you are implementing a clear-cut design, which details what data will be processed, and how. Unknown coding is when you are finding out whether you can perform some functionality with a particular piece of code, trying to get a proof of concept to work, or attempting to come up with a solution to a problem. Whatever type of coding you find yourself doing, and you will be doing both at many stages of a project, it is important that you enjoy each. Personally, I find sometimes I need to make the switch from one to another according to how I feel; some days you feel more creative than others, and those are good days to leap into the unknown. And some problems are best solved by no coding – by stopping wrestling with them for a while, and doing something more straightforward in order to give yourself a break and a better chance of returning refreshed and more able to see a possible solution.

From designs to pseudocode Depending on how happy you are with the languages and tools that you will be using for the project, you may find it useful to write pseudocode before you begin real coding.

67

Contig Count = 0

Contig Count++

Open File

Continue

Until End of File

Close File

Otherwise

Open Report

Produce Report

Figure 9.2 Process flow diagram: report construction

If New Contig

* Read Line

Read Lines

Create Report

Print Contig Count

Close Report

68 WHERE TO START?

FROM DESIGNS TO PSEUDOCODE

As you become more familiar with your tools, and more experienced in application development, you may feel comfortable about skipping this step and going straight into coding. However, writing pseudocode can still sometimes help you get a clearer picture of how to implement complex functionality, and comparing pseudocode to code is often a useful way of clarifying why a program is not performing as it should. So, what is pseudocode? Well, it is functionality defined with a subset of everyday language. If you like, it is your program at a midway stage between a written description of what is to happen and the executable code that will actually do what is required. Let’s look at some pseudocode for the program to create a report on a sequence assembly. We’ll use the sub-process diagrams shown earlier of handling file-opening errors and checking each line of the data, as well as the high-level overview we’ve just seen. /* Create report of assembly */ contig_count = 0 if exit_selected exit end if /* Have we got a valid filename? */ unless valid( filename ) exit_message = “Invalid file name” exit end unless /* If we can we open the file, read it in */ if open( filename ) / * Read the file to its end */ until EOF line = readline( filename ) /* Check for the start of a contig’s details */ if contains( line, “Contig” ) contig_count++ end if end until close( filename )

69

70

WHERE TO START?

else exit_message = “Can’t read file” exit end if /* Generate the report */ if open( reportfile ) report = “Assembly contains”, contig_count, “ contigs.” print reportfile report close reportfile else exit_message = “Couldn’t write report” exit end if /* Validate a filename */ sub valid filename = parameter[ 0 ] /* Have we got any name at all? */ unless( filename ) return false end unless /* Is there a file called this? */ unless( exists( filename ) ) return false end unless /* At this point, we have a name that points to a file */ return true end sub

If you have trouble knowing how to start going from your diagrams to pseudocode, start by writing the comments as descriptions of the processing that the boxes and their connections represent, the decisions that have to be made, or the conditions that must be checked. If even this is problematic, are you sure you’ve done enough analysis to understand what you’re supposed to be doing?

FROM PSEUDOCODE TO CODE

From pseudocode to code Pseudocode like that shown above can be more than halfway to the real code that you are to implement – or it can be nothing like it at all, depending on the syntax and grammar of the programming language that you are going to use. Whatever language you choose, some changes will have to be made in the transition from one to the other. Variables will have to be declared and initialized; brackets, braces and other appropriate punctuation may well be needed; modules and libraries might have to be called. There may also be further modifications required which are language- or platform-dependent. One area that may be affected by these considerations concerns the syntax and style of implementing input from, and output to, files and processes. Another involves how you can access system error messages, extract useful information, and return a helpful message to the user. But, as we saw earlier, whatever you’re building, and whether you write pseudocode or not, you needn’t ever worry again about where to start. Handle errors. Take in data. Show output. Get going!

71

10 Functional, then Optimized

This chapter emphasizes the importance of getting your code working, and only then getting it working as well as it possibly can. It also explores issues involved in the process of incremental delivery of functionality.

Get it out of the door, and in front of your customers, as soon and as often as safely possible ‘Safely’ is the important word here. Let’s just step back a little. We have already looked at prototypes and proofs of concept, and discussed how they can help to keep your customers involved with the project. The point that needs emphasizing here is that it can be tricky deciding when to hand over a deliverable to your customers. As a rough rule of thumb, the time to let the customers see your program in action is when you feel confident that all the new functionality will pass successfully through the bulk of your tests. It may be that you need to warn your customers that not all tests have been successfully completed, or that this version of the software has only been tested under special conditions – with one particular dataset, for example, or only on a minimal system configuration.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

74

FUNCTIONAL, THEN OPTIMIZED

You will have to balance the need to demonstrate that you are progressing against the dangers of giving them software that is not ready for release. This one is always a tough call. On the one hand, there is the potential of positive feedback from the delivery. On the other, there are the possible downside outcomes – such as an unexpected program termination during the grand finale of your most senior customer’s pitch to potential investors in their venture. One way that is sometimes suggested for avoiding such pitfalls in releasing early versions of software, which you might want to consider and assess against the resources and requirements of your situation, is to set up a small internal web site that guides visitors through the various screens which (if all goes well) your users will encounter in the real program. But the trouble with this sort of solution is that you just end up swapping the risks of users finding bugs in your program for the risk of users finding bugs in the web site that demonstrates your program. I would stick to fixing functionality, and, rather than spend time designing a web site navigable by your customers, plan a show that you can lead them through.

Planning shows There are advantages to demonstrating new functionality to users under controlled conditions – by showing it in a presentation, rather than giving them the new version to play with. You have more control over what they see of the software, and you can structure what they see for maximum impact; you are also on hand to answer any questions that they may have as you demonstrate to them the new features of this release. However, this is not a matter to be approached lightly. You need to plan what you will show and this planning will itself show – but it won’t be half as apparent as a lack of planning would be. Make sure that you have considered the following aspects.

Datasets

Again, only use data that you have tested your program on. The last thing you want in a demonstration is something bizarre happening

WHEN NOT TO DISPLAY WORK IN PROGRESS

because you are trying to process data you have never encountered before. Feel confident you can handle it? Remember, errors will happen.

Controls

When something needs clicking on, ensure that the person clicking on it has practised enough times to know what should then happen. This is a show, not a test. So senior people leading a demonstration, or presentation, should hold their audience spellbound with their oratory, needing merely a brief nod to their assistant at the side of the stage (who is the one that knows which button to press when) in order to display the next scene in the unfolding story of ‘What this software means for you!’.

Visibility

Think about what you do and don’t want people to see, from what is on the screen to what is on the walls. Your customers will take many factors into account when they consider your demonstration, from the clothes that you and your colleagues wear, to whether you are interrupted. If your physical development environment isn’t particularly salubrious – for instance, you work in a windowless concrete-walled room on an anonymous industrial estate – consider a neutral venue which offers more comfort and refinement. If you show functionality to customers, they will want to take it away and play with it. If you feel brave enough to do this, at least give them the sample dataset that you used in the most recent demonstration, explain that you cannot yet guarantee how it will behave on any other data, and get ready for bug reports.

When not to display work in progress – getting them used to having it Once you have made a delivery of functionality, make sure that your users have enough time to explore it in full and get to grips with what it does before you give them more. Better still, sort out any problems

75

76

FUNCTIONAL, THEN OPTIMIZED

that they identify and give them the corrected version before making another offering. This approach has two advantages – you and they can both be happy that what they have so far works; and you can put more work into the next delivery and make sure it is as right as you can get it.

Customer retention and repeat buyers There is an old adage that it is more difficult to get a new customer than to keep an old one. Your existing customers know what you can do, and as long as you can keep them happy they are more likely to come back to you when they want more work done. The better you understand your customers, the more you can anticipate their needs and reactions, and the closer a match there will be between what they want and what you give them. If they always like to start meetings with coffee and a chat, let them; if they want to get straight down to business, follow their lead. However, you should not try to sell them the next version of the software before you have proved to their satisfaction that you can successfully deliver version 1. Why? Well, one good reason is that when you start telling them about your ideas for version 2, you can bet that they will want the best bits straight away – and these will almost certainly be the most difficult to implement. Instead, put yourself in their shoes. Think about the places where you are a customer, and try and analyze why it is you go there rather than elsewhere. Is there some way you can reproduce these sorts of benefits for your customers? For instance, you may always use a particular store because it is the most convenient for you to get to. Maybe you should think about taking the next demonstration to your customers, rather than getting them to come to you? And don’t underestimate the power of lunch as an offensive weapon. After a good lunch, customers may well be inclined to focus more on the positive aspects of a situation. You yourself will probably remember times when tiredness, hunger, or other preoccupations made you more picky and irritable than usual. Try and minimize the effects of any factor that may make it more difficult to bring your customers and users along with you.

BENCHMARKING, ASSESSMENT, REFINEMENT, HARDWARE

Optimization: benchmarking, assessment, refinement, hardware Once you have got basic functionality – such as a piece of proof-ofconcept code – then you can look at optimizing it. However, don’t rush into this. The time to make things work better is when they aren’t working properly, not when their performance is acceptable. For example, you should not waste any of your finite amount of development time making something faster until your customers complain about its slowness. Demonstrations of software to customers and users, with their opportunities for instant feedback, can often indicate areas where enhancements can be usefully implemented. For example, sections of the program where the processing really does needs to be speeded up are often indicated by comments such as ‘Why hasn’t anything happened yet?’ If there is a real requirement that program performance be improved, then there are a number of techniques that you can call on. First, find out where the bottlenecks in your code are, by printing out the current time in between each section of code. This will tell you which processes take up most time to perform, and allows you to focus on the areas where your effort will have most impact. Then look at your code, and at the algorithm behind it. • Are your loops as efficient as possible, doing the minimum amount necessary for the minimum number of times? • Is there some way that you could end a loop early and still achieve the desired results? • Could you redesign what the program does so that the timeconsuming steps can be performed as a background task? • Can you pre-calculate variable values, or cache results, in order to speed things up? Once you have refined your algorithms and their implementation so that you feel no more improvement is possible, you may need to consider hardware enhancements.

77

78

FUNCTIONAL, THEN OPTIMIZED

For instance, if your server software has to deliver hands of cards to users, you have a requirement to generate random numbers. Softwarebased random number generation may be satisfactory in the early stages of development, but a hardware alternative may be necessary when you go live, for reasons of security as well as speed.

Focus on essentials, because your customers will If it doesn’t need doing, don’t do it. You have a limited amount of time to get your job done. Don’t build a completely customizable interface, where the users can control the colour of every button and dialogue, before you can produce accurate reproducible results. If the customer really wants bells and whistles, leave the design and implementation of them until last, in case time constraints mean that some requirements have to be dropped. Again, during demonstrations, you – or a colleague – should be listening out for phrases that may highlight areas which need to be urgently addressed. For example, someone asking ‘What is that for?’ indicates an interface element whose purpose is not immediately apparent to people who have an idea of what to expect, and know what they are looking for. Users should never even have to think about this. The best interfaces are almost not there at all – their transparency simply allows you to do what you want without inconvenience or awkwardness. If your program doesn’t do the job that you are being paid to make it do, all the fancy trimmings are just a waste of time. Remember: get it working, and then add the rest.

11 Coding Style

This chapter aims to help you write better code, whatever tools you use to do so. I am sure that you write good code already, and equally certain that when you learnt programming you were taught, as I was, the best way to go about things. However, we are all human, and in the real world we sometimes don’t do everything that we know we should. Sometimes you feel you just have to get functionality out of the door, and this can seem a good short-term solution when your customers are shouting. But if you want to build code that is both robust enough to cope and adaptable enough to be easy to enhance, it is a good idea to imagine yourself looking at your program’s source after six months of doing something else. Or, better still, think how you would feel about it if someone else had written it and it was now your responsibility. This chapter is based on my experience of having to do that more than once, with my code and with other people’s, and I hope that it saves you at least some of the troubles I went through.

Don’t write it if you don’t have to First, let’s look at not writing code. Using libraries and other programs can often be useful, although sometimes there are occasions when you would be better off creating a custom solution. Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

80

CODING STYLE

For example, a popular library of bioinformatics programs used to include a routine to generate graphical sequence alignments – indeed, it probably still does. However, last time I looked, if you wanted the output to include vector information, you had to write your own code. There are useful libraries that provide a simple interface to complex functionality. Database access and web page generation are two areas where I regularly benefit from other people’s work. The question that you need to ask yourself when considering using third-party software is ‘Will this make my job easier?’ Sometimes using pre-written code can provide a temporary solution in one area of the program while you work on refining another area. For example, in developing code to show a sequence assembly as a series of web pages, you might first create simple images using a standard graphics generation library, in order to check the algorithm that allocates the right image to the right page. When using external code, avoid clogging up your program with unnecessary functionality. If possible, selective loading of parts of libraries reduces overheads and resource requirements. And give some thought to the bandwidth considerations that may be involved in users loading and then processing large amounts of clientside code, particularly on low-grade machines and networks.

KISS – Keep It Simple, Stupid ‘Obfuscation’ is just a fancy word for making things difficult for yourself and everyone else. It’s not big, and it’s not clever. Don’t do things the complicated way, because the simple way is hard enough to get right. This becomes more important as functionality becomes more complex. At each stage, make what you are trying to do as easy as possible to understand.

When you look at your code, does it make your eyes hurt, or does it look like poetry? Aesthetic considerations are important, especially when you need to get to the sense of something quickly but the form is obscuring the content. Without aspiring to the pureness of haiku, it will help if you use line breaks and white space; avoid clutter and aim for clarity.

NAMING OF PARTS

Try to group related processes, and put things in the best places for them. For example, if you scatter your variable declarations throughout your program, perhaps because you added some new functionality to old code or left a hack in because it worked, you could eventually leave yourself open to problems of name conflict, unexpected values, and other hard-to-track bugs. Moreover, sometimes looking at a list of variable declarations at the start of a program can act like the introduction to a book, giving you a feel for what is to follow.

Naming of parts Avoid ‘write-only’ code. You should be able to understand what a program does by reading its source code. When you return to your code, or look at someone else’s, you will be asking yourself: so, what does this do? And to answer that question, you have to keep asking it, down to an atomic level. For each subroutine, each loop, each variable, you are trying to understand the same thing – what is this for? And you can make solving this problem much easier when you write code, by making sure that every element of your program has a name that makes its function, purpose or usage easier to understand. So, don’t have variables called ‘a’ if possible. My only exception to this is for counter or index variables used for cycling through loops, where I will use ‘i’ for the index, as in for i ( 0 .. 10 ) print “I is”, i end for

Even here, though, one could argue that a more descriptive name such as ‘count’ would be better; on the other hand, I feel that complex nested loop operations look neater using single-letter index values starting at ‘i’. for i ( 0 .. width ) for j ( 0 .. height ) print “At”, i, “along and”, j, “ up” end for end for

81

82

CODING STYLE

It also helps to avoid using the same name for variables of different types, especially without prefixes. ‘tmp = tmp(tmp, tmp[tmp])’ is not a good idea. And, yes, I have seen this sort of code. Another simple way to make programs easier to understand is to put in at least a one-line comment at the start of each code section to explain what it does – ‘parameter checking’, for example, or ‘data validation’. More complex code sections may require more description – and a lengthy description may indicate that the code needs to be reworked.

Going gracefully into the darkness Things fall apart – and many things, including software, do so frequently. Again, expect to get errors. They are easier to solve if you are prepared for them, and you can be prepared when you write your code by developing good habits. The first good habit to have – or acquire – is knowing when not to write code. Deal with external factors that affect your ability to think clearly before trying to write software – otherwise what you create will be so bad that it will take you three times as long to sort out as it did to write, and that itself will not have been quick. Some days are meant for filing. When you do write code, you must make your error handling code the first you write, because you will need it. Make it helpful – don’t write error messages that merely say ‘an error has occurred’. Especially, don’t write error messages with no informative content that try to be humorous. Just say as concisely and as clearly as possible: • exactly what has happened (‘Can’t open file’ . . .) • what the program was trying to do (. . . ‘for reading’ . . .) • what variable values are involved (. . . ‘at file_name’ . . .) • and any system information that you can get (. . . ‘Result: system_error_code system_error_message’) Design in a way of catching and coping with errors that you can use throughout your program – say, a mail_me_bug subroutine. I have

FLEXIBILITY IS VITAL

production code where this functionality is labelled die_gracefully( error_message ). You get the picture.

Make sure that you check return values, and act on unexpected ones. You don’t have to check for every possible result, but you should check for a value other than those you want to see. Make sure that what you think is happening is really what is going on. Sometimes where the program falls over isn’t where the problem is. Waterfall errors occur when there’s a big splash, and something higher up caused it; for example, when you begin processing by trying to open a file, but you haven’t loaded the file-name variable in the initialization stage.

Prevention is better than cure It may take longer to write in code that checks that what you think is happening is in fact what is happening, rather than ignoring possible alternatives, but not doing this is just storing up trouble for the future. If a system call can return a value that you are not trapping for, your code will only work in perfect situations. And just how common are they? Use the checking features that are built in to your coding language and development environment. This will save you time hunting down the problems that can be caused by your doing silly things that you really should know better than to do, such as leaving out a piece of punctuation that completely alters the logic of your code. Use these tools to give you warnings of what may be wrong, because this is easier than having to try and deduce what the problem is when its effects only become apparent several lines – or pages – after the point in your code at which it occurs.

Flexibility is vital In bioinformatics, requirements are constantly changing, often as a result of what you successfully deliver; your customers see what you can give them, and then they realize what else they can ask you to do on top of it. It follows that your code should be easy to adapt, extend, modify and maintain, but it also must provide accurate results as quickly as possible in such a way that your users will come to take it for granted.

83

84

CODING STYLE

There are several ways in which you can ensure that your code is extensible and flexible. First, make your programs modular, with good interfaces between the separate elements. Remember to focus on core functionality, not extras. Adopt a defensive programming approach to algorithm design: for example, when dealing with a number of possible options, always include a code section to catch the ‘none of the above’ choice. Good housekeeping will also help: tidy up by closing connections to files, processes or databases just as soon as they are no longer needed, actually check that everything is okay rather than assuming what you’ve done has worked, and report problems – and progress – appropriately to those who need to know. At the coding level, there are several things you can do to make your programs both robust and adaptable. Use concise but descriptive variable names, and use variables rather than hard-coded values. Put in helpful comments. Be clear, not obscure, and make your code easy to read. Don’t over-optimize, or use platformspecific tweaks. Remember: if it can go wrong, it will, and if it can’t go wrong, it probably still will. Build gradually. Know what you are supposed to be doing, and don’t write unnecessary code.

Keeping track of what you are doing One thing you will need to consider is what might be called ‘the administration of development’. Obviously, you should try and upwardly delegate all the more brainnumbing tasks, but it is likely that you will still need some way of keeping track of (for instance) which is the latest version of the program, which is the most solid, and which one contains that great new feature but keeps unexpectedly crashing. Applications to do this job go by various names, but their commoncore purpose is to allow you to keep track of which program version currently works best. Depending largely on your working environment, you may want to consider full source control and code management systems, a handmade version-numbering control method, ways that you could roll back to previous versions, and alternative techniques for identifying the functionality associated with particular stages of code development.

KEEPING TRACK OF WHAT YOU ARE DOING

You may also at this point wish to ponder the implications for your project of some of the issues we will address later in the book, when we go on to look at testing, rollout and delivery of functionality to customers. Analyze your requirements, and work with others to define a satisfactory solution; bear in mind that implementing such a system can be a separate project in itself, and can add to the time it takes to deliver your program.

85

Part 4 For Some Values of Done . . .

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

12 Writing the Friendly Manual

This chapter looks at adding documentation to your deliverables. From self-documenting code to installation, operator, user and programmer guides, here is what you can offer them, and how you can create it. We talked about coding style in the last chapter. Let’s take a look at some of the beneficial effects of adopting good practices.

Start as you mean to go on: make your code clean and clear Your first – and sometimes only – documentation deliverable is the program code itself, so let’s make a start with this. One simple rule to improve the appearance of your code is to use white space and simple comments more. Here is a piece of pseudocode from earlier: /* Have we got a valid filename? */ unless valid( filename ) exit_message = “Invalid file name” exit end unless /* If we can we open the file, read it in */ if open( filename )

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

90

WRITING THE FRIENDLY MANUAL

/ * Read the file to its end */ until EOF line = readline( filename ) /* Check for the start of a contig’s details */ if contains( line, “Contig” ) contig_count++ end if end until close( filename ) else exit_message = “Can’t read file” exit end if

Here’s the same functionality: unless valid(filename){exit_message=“Invalid file name”;exit;} if open(filename){until EOF{line=readline (filename); if contains(line,“Contig”){contig_count++;}} close(filename); }else{exit_message=“Can’t read file”;exit;}

If you – and others – cannot quickly grasp what a piece of code is meant to be doing, you will make it much more difficult to document – and to debug.

Using what you have done so far: it’s half-written already User stories and screen pictures can help to document what the system can do. For more formal documentation, look at the data structure and process flow diagrams. As well as helping you work out what you should be implementing before you start coding, they can help to document what you have implemented once it has been built.

ASK YOURSELF, AND OTHERS, WHO STILL NEEDS TO KNOW WHAT?

Writing documentation from functional specifications, and vice versa Working through the functional specification, point by point, and using it as a checklist will ensure that there are no major areas of functionality that you have omitted to document. The focus now is on describing how something is implemented, not what is to be implemented. But as you go through this checklist it can happen that you discover that something that you promised to have ready in this release is not yet fully functional. Better you find it than your customers; get it done, and then get it documented.

Ask yourself, and others, who still needs to know what? Once you have gathered together all the existing program documentation – functional specification, screen drawings, user stories, data structures, process flow diagrams and source code – you will need to synthesize it all into what your customers need to know about using your program. Try and think of questions that may be asked by the various people who are likely to come into contact with your software, and which the documentation should answer. For instance, users will ask, ‘What do I have to do to achieve my goal?’ Your documentation needs to take them through every step. For example: 1. Start up your web browser. 2. Go to this address: http://your.org/internal/help/. 3. Click on the ‘Start My Program’ button. 4. (etc.) Operators may have to deal with more complicated challenges, particularly if you have built a complex multi-tier application supporting many simultaneous clients. Make sure that if necessary you have answered all of these questions: • How do I start each program – server, client and middleware? • How do I check the status of a program?

91

92

WRITING THE FRIENDLY MANUAL

• How do I restart a program if necessary? • When should I restart a particular program? • Under what status conditions do I need to alert management? • Are there other tasks I need to perform? Installers and systems administrators will be more concerned that a program is capable of peaceful coexistence. You will need to describe known conflicts between your applications and any others which will need to share resources with them, and preferably provide workarounds or (better) solutions. You will also need to detail system requirements and (if possible) expected resource usage. You may also need to create detailed instructions which specify how to install the software, how to check that it has been installed correctly, how to configure it according to the resources allocated to it and the demands expected of it, and which also explain how future upgrades can be applied. It may be that your application is going to be called from other programs. If so, you need to think about the application programmer’s interface. Developers planning to use the functionality provided by your software will need to know in detail, and preferably with examples, how to extract the most value from it. This documentation should answer at least the most basic programmer’s questions about a new piece of software: • How do I pass input to it? • How do I receive output from it? • How do I access exposed functionality? • Where can I find out more? It may also be appropriate for your documentation to cover Frequently Asked Questions, give details of online help and support that will be provided, or direct users to a helpdesk facility. In the latter case, you may need to provide helpdesk staff with access to special information about planned enhancements and known bugs that are being addressed. Finally, you may need to consider a less technical overview of the program for an audience that may be more focused on markets, sales, finance, or strategy. I’ve heard it suggested that for this type of documentation you should use crayons.

13 Testing – What and When

Here we will look at why testing really does have to begin with the proof of concept. Okay, so no one but you normally notices any tests that are done at this stage (the silent – or not-so-silent – ‘Yesss!’) – unless, of course, you haven’t done enough testing, in which case everybody notices. This chapter outlines how you can define test plans which detail the program parameters and sample datasets to be used, the functionality that should be demonstrated, and the expected results or outputs. It goes on to highlight a few things to watch out for during the test–fix cycle.

Test plans Your test plan should state what tests will be needed for each unit (an element of functionality, such as code to get data from a supplied source and store it into a program variable), module (a self-contained collection of units, such as code routines to handle database access), and system (the complete application, or executable collection of modules, that delivers requested data to users). We will go on to look at how to define appropriate tests in the next few pages, and how to implement a test plan. The formality and level of detail of your test plans will depend on your situation; they may be in your head, or specified in triplicate. Always remember that whatever

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

94

TESTING – WHAT AND WHEN

the format and style of your test plan, the most important thing is that it exists.

Writing a test plan The best way to write a test plan if you have never had to write one before is to go through the program’s performance specifications to clarify what it must do. You will have to test all of these functions. In most cases, you will be testing code as you develop it, then putting it back into development to fix any problems, then re-testing to ensure that the revised code hasn’t broken anything else. Let’s look at how we could validate our example program, which reads in sequence assembly data and then outputs a summary report giving the number of contigs in the assembly. In particular, let’s consider what tests are needed at unit, module and system levels. Unit testing

Unit testing looks at the lowest level of operation. If a file cannot be found, does the program return an appropriate message? Once a file is read, is it properly closed so that it can be opened cleanly again? In many cases this type of testing is simply part of the developer’s job; should further robustness be required, this type of testing is straightforward and amenable to automation or delegation. Module testing

Even our simple sequence assembly summarizer program could still be subdivided into modules: • the initialization stage, where parameters are checked • the data acquisition stage • the data processing required to generate the summary report • the delivery of the report to the user • the termination section, if required • error handling code that may be called by any of the above modules.

WHY DEVELOPERS SHOULDN’T BE THEIR OWN TESTERS

Tests should be designed to demonstrate that each module operates as it should do – for example, that the program’s data acquisition module can properly handle correct, incorrect and missing file paths, and that, given a correct file path, it can read in all the data contained in the specified file. System testing

At the system level, testing needs to show that the whole program behaves as expected given the data it has to process. So, if data is supplied describing an assembly of 43 sequences into 3 contigs, success is the program reading the file and generating a summary report which says it found 3 contigs; anything else is a problem. A balance has to be struck in many cases between testing every possible situation and actually delivering product. Try to make your tests representative of real-world conditions, and check for situations on the boundaries of what is expected.

Expect the unexpected Eighty per cent of the work takes 80 per cent of the time; unfortunately, the other 20 per cent also takes 80 per cent of the time. Bugs happen. Things do go unexpectedly wrong, and not just in software development. You are on your way somewhere and your car gives you grief you couldn’t possibly have anticipated – you lift up the bonnet in order to see what has gone wrong this time and find that your carburettor is no longer attached to your engine, for example. Or you might be managing a project to build a complex client–server product and you discover only when it is too late that your technical authority has no previous experience of developing multi-threaded software, with all its associated resource management and conflict avoidance issues. Hope for the best, but prepare for the worst.

Why developers shouldn’t be their own testers A bug is hardest to find when you are seeing what you expect to be there, not what is there.

95

96

TESTING – WHAT AND WHEN

Often an outsider will look at a piece of code that you have been trying to fix for hours, and go straight to the typo that you just didn’t notice. A colleague of mine encountered a new and extremely scary error message. When we looked at the program code, I noticed that just before the newest section, a line ended abruptly. One accidentally introduced return character had turned the whole program into garbage. Moreover, when you are developing code you tend to test functionality on data that you know, because it makes it easier to identify what is going on. But your users will try to put all kinds of stuff through your program – and not just the naïve ones. I know someone who was experimenting with offering a sequence assembly program of mine to our users through a new distributed delivery system he was developing. When he was playing with the application, he had no appropriate data, so he fed it the plans of our building ‘to see what happened’. And because you know that trying to do something a particular way would be silly, you never do it. But your users won’t know that it’s silly, and may – indeed, probably will – try it out. We are creatures of habit, we do the same things in the same way, but other people have different habits. Deal with this, and use it to your advantage. If you yourself are both the development team and the test team (and the system administrator and the helpdesk operator and . . .), you have a number of options. You could try and set up a reciprocal arrangement with another developer in the same situation; you could call in friends and family; or you could get your customers and users to help in the testing process.

Involving users in the testing process If you are making regular deliveries of continually enhanced prototypes to your customers, you should try to get them involved in the testing process from an early stage. In most cases the difficulty is keeping them away from the new functionality until you have got it working – never mind solid. Try to turn this into a positive thing, but make sure that your customers and users realize what is involved in testing, and be prepared to deal with their

SOME THINGS TO WATCH OUT FOR

disappointment if their testing highlights areas where you need to do more work. It may be best, particularly at the early stages of a project, not to involve those of your users who are least familiar with the software development process, and who may expect everything to work right first time under all circumstances. If your users do identify a situation where your code behaves in unexpected ways, check that this is reproducible, fix the problem, and add the conditions which caused it to happen to your test suite. Testing can also highlight where documentation is incomplete, but also it can prove that the documentation meets its requirements as well as the program that it describes does.

Some things to watch out for There are a number of things that commonly cause grief to developers during the testing process. Let’s take a look at some of them.

Boundary conditions

Remember: developers tend to use the data that their program works well on, so if something suddenly goes wrong they know it’s the newest bit of code that is broken. Equally, developers tend to avoid sample datasets that take a long time to process, cause their application to grab a lot of resources, contain poor-quality information or are fatally corrupted. In this respect they are completely unlike the average user. It may not physically be possible to test every potential combination of situations that could occur as users interact with a program. But one thing that can be done is to test an application at the agreed extremes of its capability: the maximum number of simultaneous users it has to support, the minimum system configuration it must run on, the lowest communication speed it must cope with, and the most complex operations it must perform. If your program can cope with conditions at the edge of its performance envelope, it is less likely to encounter difficulties in dealing with less challenging situations.

97

98

TESTING – WHAT AND WHEN

Illogical users

They will do things you wouldn’t have dreamed of trying. ‘Why did you try to use the results as input data?’

Instance clashes

If you have a program that is capable of being run by two users on the same dataset at the same time, then you will need to consider the best way for the first user to arrive to tell any others that the data is already being processed, otherwise you will get some very strange results.

Illogical users

‘Why didn’t you come and ask me if it could handle that type of file, before “just giving it a try”?’

Test team vs. development team

Feelings can run high when one team of people are doing their best to create finished code, but are being harassed by complaints from socalled colleagues about bugs they cannot reproduce, or are working on, or they are sure were fixed two releases ago, or aren’t bugs at all. And meanwhile, another team of equally vital people are trying to work out how to not only prove that a program does what the customer wants when the customer won’t sign off the functional specification, but also persuade the systems team to finish installing the clean network before new versions of the client and server operating systems are released, and all this while they attempt to get the developers to actually acknowledge that they really are raising issues that need to be addressed, thank you so much, if you wouldn’t mind. Whether dealing with the stresses of this situation by encouraging all staff to take part in team-based networked first-person shoot-’em-up games is a good idea or not depends largely on the characters involved, and the attitude of the organization concerned to conflict resolution in the workplace.

WHEN TO STOP TESTING

Illogical users

‘You tried to do what with it?’

When to stop testing When the bugs that you are finding turn out to be caused by problems in other people’s applications that you are calling, the operating system you use, or the hardware that you run on, it is probably time to stop testing. Having identified some points that may be useful to consider when planning the testing of your program, let’s now go on to look at the mechanics of this process. How exactly will you deliver what you’ve done, first to your testers, then to your program’s customers and users?

99

14 Rollout and Delivery

Here we look at the process of increasing the number of users of your system above one, as your code takes its first tentative steps out into the world. When you write code, you are obviously checking that it does what you expect on your sample datasets, but getting it robust enough to handle other people’s ways of using it requires some thought and preparation. You may have already been through the early stages in the process when someone other than you has tried to test your program, and you have had to do three things: make the program accessible to them, explain what it does, and warn them about what it cannot yet do. This is what you will have to do in more detail for your customers. You should at this stage be prepared to learn as many new things about your program as you did when it went into testing. If you have not yet tried even informally to prove that your program can do what it is supposed to, I can only recommend that you consider very carefully whether you want your customers to even see your software in action, let alone take it away and test to destruction. That’s the polite way of putting it.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

102

ROLLOUT AND DELIVERY

You will need separate development, test and production environments Why do you need three separate environments? Because you will not want to give your customers the version with all the latest bells and whistles. Well, yes, you will – but only after they have been thoroughly tested. What this means in terms of ‘who is coding what right now’ is that while Developer A is continuing to incorporate the customer’s comments on Whistle Version 1.1 into Whistle Version 2.0, Developer B is still fixing the last of the problems that the test team found in Bell Version 3.0. In this situation, someone has to make sure that the software going out in fifteen minutes to the customers contains Whistle Version 1.1 and Bell Version 2.2, and not confuse it with the program that is being worked on, or the one that is being passed between the test and development teams. The usual solution here is to have three or more separate environments, either virtual or physical. Here’s how I’d do it. Development environment

• The development environment is where you can find the code that is currently being written. • It may be the first time that this code has been worked on. • On the other hand, it may be being modified in order to fix bugs, improve performance, or include enhanced functionality. • In our example, Developer A would be working here on code that is not yet ready to go into testing, while Developer B would also be here, working on fixes to reported bugs. Test environment

• The test environment is owned by the test team, • they control access to it: when a release is ready for testing, • it is passed to the test team with any necessary documentation,

SEPARATE DEVELOPMENT, TEST AND PRODUCTION ENVIRONMENTS

• they can then install it and examine it against their test plans and schedules. • Once it has passed all required tests, it is released by the test team from the test environment into the production environment. • If the test team feel that they cannot sign up to a setup like this, you need to find out why not. • If your situation requires a more informal approach, it is a good idea to introduce at least some separation between test and development environments, even if it’s only borrowing a colleague’s machine to run your code on.

Production environment

• This is where the latest release to the customers, and immediate-access copies of all previous fully tested releases, are kept. • Whoever is in charge of relationships with the customer should coordinate this environment, taking in releases only when they have been signed off by the test team, and ensuring that only production releases are delivered to customers. In some organizations, a separate ‘system environment’ may exist between the test and production environments. The test environment is used to determine whether the program produces the right results; in the system environment, the functional program’s interaction with the software that it will have to coexist with is examined. I take the view that both areas are best assessed in the same environment, possibly by individuals with different expertise. Furthermore, the test environment may even be a quiet backwater of the production environment, which itself may be remarkably similar to where the program will finally go live. As long as customers and colleagues are happy, this can be a perfectly acceptable situation. However many separate environments you have – and I suggest three is optimal – they may be implemented in any suitable form, from a directory structure to separate self-contained client-server networks.

103

104

ROLLOUT AND DELIVERY

You may decide to keep your development, test and production code in different places on your file system, moving code from one place to another as the situation dictates. Alternatively, your organization’s custom and practice, or the need to test how your program works when multiple instances interact, may make the process of rolling out enhancements more complex. Therefore you may need a physically separate network of clean machines – a server box and several client computers that have the specified operating system and software configuration reinstalled before every new release of your program is put on for testing. This way of working can help to avoid any possible resource conflicts or confusion over the cause of a particular problem.

Delivery notes A delivery note, or some other method of describing what is in the box that your customers are being asked to sign for, is vital for every release of your software to your customers. This is especially true if you have committed to delivering functionality at defined stages. The delivery notes should detail the changes made in this release, the requirements that this release satisfies, and the testing that it has undergone. The sort of information I would expect to see in a delivery note might be something like this: Acme Software Product v0.2, Delivery Date 2 Notes: New functionality, bug fixes and comments incorporated. Requirements satisfied: 1, 3, 7, 8 and 14. Test passed: 1.1, 1.2, 1.3, 3.1, 3.2, 7, 8.1, 8.2, 14. Relevant tests failed: None. New in this release: Changed text of error messages. Program no longer crashes on badly formatted data (sample file included to demonstrate this – BAD_DATA.TXT in SAMPLES directory).

105

BUILDING THE INSTALLATION PACKAGE

v.01, Delivery Date 1 Notes: First release of software. Requirements satisfied: 1, 3, and 7. Test passed: 1.1, 1.2, 1.3, 3.1, 3.2, and 7. Relevant tests failed: None. New in this release: Reads valid format data file and summary of contents.

displays

Verbal handovers These are common in more informal, dynamic environments, and mean that feedback is often instantaneous. This can be a good, or bad, thing; it is up to you, and what you can deliver. You may find that verbal handovers can be quite rewarding – so long as you get them right. They can also be nerve-racking experiences. For example, you might find yourself sitting next to your customer, installing the program they have been waiting for, and then being watched by them as you try to cope with the fact that they are using different web server software from that which you used to develop and test the code. In general, don’t try informal handovers unless you feel you could do a more formal handover if required. If you could, then that certainty will underlie what you are saying, and increase your customer’s confidence in you. If you couldn’t, your customers may sense this, and you may set off their bullshit detectors. A happy medium between complete informality and documentation overload might be to send an email that (for example) gives the URL of a new code version, the location of some sample data that it has been tested on, and a note of the important new features.

Building the installation package Whether you use a more or less formal method to hand over what you have done, it is important to ensure that whoever is to receive it is able to install it and get it started without fuss.

106

ROLLOUT AND DELIVERY

This is particularly important when your software is supposed to be able to run under multiple platform configurations, especially when it also requires that some specific third-party software be available. Good installation package creation software should allow you to define the steps to be taken at all stages of checking for appropriate hardware and software, and help you to cope with any problems that may arise. For example, if your code has been tested only on one operating system, its installer should give a warning if a user attempts to run it under any other. Your installation package creation tool should give you the ability to check for dependencies – such as a processor that is fast enough to run your program, or an up-to-date version of a required library of graphics routines. Whether you use third-party tools or write your own installation script, ensure that the person doing the installation is given appropriate information about hardware and software compatibility issues as the installation proceeds. If your program needs externally provided software, you may also need to consider licensing appropriate versions of any necessary thirdparty software and providing it as part of the installation, or ensuring that your users are told how to how to obtain it for themselves before trying to install your program.

Instant gratification mode It can be useful to knock out a quick fix – quick in terms of delivery, at least – to give your customers at least partial results straight away, perhaps by using code libraries that almost do what you want, or maybe by putting up with sub-optimal response times. As we’ve discussed, you might start out by using a code library routine to generate a sequence alignment from an assembly which you can then import and process, and then later on replace it with your own code that includes details of vector information. Another situation might be where you use a graphics package to generate images of assembled sequence, and then speed up processing by replacing this with generated HTML. The advantages of this approach include demonstrating to your customers that you can deliver the core functionality they need; the disadvantages include trying to explain to them why you now need to completely rewrite the code.

THE USEFUL/STABLE BALANCE

As long as you have tried to make your code adaptable and extensible, using the techniques we have been discussing, this strategy will allow you to prove that a concept works – that you can deliver the big idea – and you can also demonstrate that your program is adaptable enough for you to make such module-level enhancements. This modularization of code also has other benefits besides allowing you to separately develop and test related functionality. For example, a clear separation between front-end interface and back-end processing could make your code easier to port onto the next generation of grid-based service-oriented application delivery platforms.

The useful/stable balance This isn’t an either/or, and it’s not static. There is always a trade-off to be made between the amount of functionality that you can deliver and the level of testing that you can carry out on that functionality. Make sure that your customers know whereabouts on this continuum each release lies. Throughout the whole process of rollout and delivery, you should always remember that you are selling to customers. If possible, make a presentation of the new features, even if this just means you demonstrating how to use the program on their machine. Invite feedback, and outline the work that you will be focusing on during the next stage of the project. In particular, it will help when you deliver the goods to highlight the things that they have specifically asked for in this release. Your own situation will determine what method of rollout and delivery you adopt. As always, do what works. For a preliminary delivery, concentrate on code, not documentation. You may have to test the installation instructions by working through them with a customer’s representative, for instance. While you are doing this, you can be trying to determine with them what else they will need in later stages of the project – perhaps a demonstration of the features of the new software to its eventual end-users, or a desk reference to its commands. And if you have not done so before, you can use this time to learn what they would like you to offer in terms of post-delivery support and feedback. In the next chapter, we’ll take a look at what this may involve.

107

15 Support and Feedback

This chapter helps you define what you should think of offering to your customers after you have delivered what was asked for, and what you can expect to get back from them in return.

When do you start? If you are sensible, support and feedback begin the day you pitch for the opportunity to tender for the contract – right at the beginning of the project. The first thing that you offer your customers is support and assistance to help them accurately define their requirements, and you will use their feedback to determine how well this process is going.

Pre-emptive support In an ideal world, you would do the support that was needed before the customers know that they need it – so the manuals are ready before the users start to use the program, for example. This goal is not always achievable.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

110

SUPPORT AND FEEDBACK

What are your local customs? In bioinformatics, particularly where you are developing software that is mainly or wholly used internally, giving support and getting feedback is often simply a case of walking down the corridor and having a chat with a colleague. Other environments have different priorities – adapt and survive.

Development costs vs. support costs Development costs are normally finite, whereas support costs are potentially infinite. Don’t over-commit yourself. For example, your company may have agreed a fixed price and timescale for a development contract, with a signed-off functional specification and test plans agreed with the customer. The software is on course to be completed on time, within budget, and to agreed expectations. Suddenly, you realize during the course of a conversation with your customers’ representative that their understanding of the specification’s one-line reference to ‘Provision of post-delivery support as necessary’ is more complex than yours. Where you foresaw six months of you checking mail for any bug reports, and dealing with them as and when they arrive, they envisage the provision of an offshore call centre, complete with a maintenance and support team, and a 24-hour helpdesk run on a rota system. Alternatively, you could be making available to a wider audience some software which had previously only been used inside your organization, at which point you find yourself learning about all the different ways that a set of instructions – which make perfect sense to people who know what they are doing – can be completely misunderstood by others who are trying to do something moderately complex for the first time with an unfamiliar tool. We would all like to spend more time developing great new stuff, rather than keeping the old stuff doing what it should be doing. Still, I got into bioinformatics through developing software for metaanalyses, and I got into that through maintaining code, cleaning data and providing general user support, such as showing people how to reboot their PC. Subsequent spells on helpdesk duty have taught me that you can learn a lot from being involved in user support, and you can get some

WATCH OUT FOR UNANTICIPATED FEEDBACK

very valuable feedback, but I have also learnt that it plays havoc with the software development process. If you are in an environment where front-line user support responsibilities are taken away from developers and testers, so that someone else addresses the customer’s concerns while determining what the problem is and who to pass it to, then such matters can be passed into your bug tracking system (whatever that may be) without interrupting the smooth delivery of functionality to customers on schedule. Alternatively, if there’s just you, the solution is simpler: if you have to do helpdesk work, don’t plan to write any code at all that day. Just do helpdesk.

Watch out for unanticipated feedback requiring urgent priority readjustment It will happen. Things do. I am certain you can think of examples from your own experience – if not in application development, then in the real world. You realize – too late – that you’ve just put the wrong fuel in your car, and instead of the pleasant day out that you had planned you will be spending the next few hours getting the tank pumped out. Or you get a phone call from a customer to say that Feature X (scheduled for delivery in several weeks’ time) must be in the next release, and they want the next release to arrive, fully tested, in two days’ time. We will look at what we can do about these challenges in the next chapter.

111

16 Planned and Unplanned Enhancements

Here we will discuss how to handle those moments when someone says ‘There’s something wrong . . .’ (bad) or ‘We need something else as well . . .’ (good, unless you originally agreed it would be in that release, in which case it is Very Bad Indeed).

Good applications are never finally finished There is always another feature that could be added. This is not the same as having to adjust to demands for extra post-delivery testing because something doesn’t work. Don’t be there. Instead, when you start coding Version 1.0, prepare yourself for the likelihood that your users will already be specifying their requirements for later versions before you deliver this one. Running more than one development cycle in parallel can help here. You can be at different stages of different modules – coding one while testing another, for instance. I don’t advise trying to do the same stage simultaneously for two different projects, unless you have assigned a team to each and feel confident that you will be able to help them cope. This setup may prove useful if you get a message urgently requesting an immediate change in the order that the agreed elements of functionality are to be delivered – the ‘We need Feature X by Tuesday’ problem. If you are running parallel cycles, move all available resources

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

114

PLANNED AND UNPLANNED ENHANCEMENTS

to accelerate progress on the development effort which will deliver this feature – but make sure your customers are aware of how this may impact your ability to deliver other requested functionality to the original schedule. This method of working on different functionalities which are at different stages of development can be helpful even if you are a solo developer; moving from trying to debug a complex transformation to running finished code on a new dataset can mean you are productive even when you are stuck.

Things not to say in meetings, No. 94: ‘Oh, you found that one . . .’ Not unless you get on really well with your users. No, not even then. That’s when you find out that you were wrong about how well you get on with them. It’s not worth the risk. And the effort of explaining geek humour. And the awkward silence that will follow. Don’t be funny about your product. Or any competitors. That sort of thing spreads. Okay, the good stuff does that too, but it’s a lot slower. If you don’t treat your own work – and your customers – with the respect that is due to both, why should anyone else respect you and what you do?

‘It’s not a bug, it’s an opportunity to further enhance the user experience’ Two sides of the same story. Focus on the positives. However, saying ‘While we fix this, shall we add in the enhancements we talked about yesterday?’ is not recommended. ‘At least you found it, not a real user!’ is also a high-risk strategy. If your users find a genuinely reproducible bug in production code, apologize, fix it fast, and then fix the system that allowed it through. And tell your customers what you are doing, and why, so they will be confident that it will not happen again. Everybody makes mistakes. Don’t make the same ones twice.

Big shoes: managing change in the workplace Sometimes you find yourself doing a colleague’s job, as well as your own. Sometimes you find yourself covering for more senior colleagues.

SLIPPAGE

The question ‘Why aren’t they here to handle this?’ may spring to your mind. This sort of situation is sometimes described by recruitment consultants as ‘a challenging and dynamic environment with plentiful opportunities for growth and development’. Most of us can recognize it by the overwhelming feeling of responsibilities outweighing rewards. If this suddenly happens to you, don’t panic, and do the best you can. If you are in front of customers, don’t exceed your authority, but promise to get the problem sorted out straight away, and get out as fast and as politely as possible so that you can start work on delivering your promise – and making sure the situation doesn’t happen again. And when you find yourself thrown in at the deep end, the same principles hold true. Take things step by step. Work out what needs doing first – deal with what is in front of you. For example, if you find yourself on a project where customers keep adding requirements so that nothing gets delivered, agree a subset of requirements that they cannot manage without in the next version, and let everything else fight for priority with all the new features that will be thought of while you finish that version. Get help wherever you can, and bear in mind that when the pet shop owner wants a big shark, he takes a small shark and puts it in a big tank. What size is your tank?

Priority 2: It’ll be along real soon now, when we’re all less busy Priority 2 never gets done. Not all of Priority 1 gets done sometimes. You will soon get a feel for what is really important to deliver.

Slippage Sometimes it becomes apparent that you have underestimated the time it will take to get your application to your users. Before you go back to them and give them this news, make sure that you have explored every possible avenue to avoid this situation, including round-the-clock coding and bringing in extra resources. If you have done all this, and there is still no way that you will make your deadline, work out when you will be able to deliver. Then minimize the blow: let your customers know that you have run into a major problem, so the first thing that they think is that they are not going to

115

116

PLANNED AND UNPLANNED ENHANCEMENTS

get anything at all; this means that when they find out that you are only going to be a few days late, and that you have already taken steps to rectify the situation, they are more likely to be relieved than angry. Another option is partial delivery – give them what you can get finished by the due date, and make sure that you explain when the missing functionality will follow.

17 Project Signoff

This chapter looks at bringing a project to a close – often something that you need to take charge of in bioinformatics software development. Moving to a new job headquartered on another continent can help, but alternative approaches may also be applicable. Part of the problem in drawing a line under your work is the tendency of users to suggest ways that your program can be improved. If you are largely self-managed, and therefore responsible to at least some degree for what you do with your time, there are ways that you can deal with this; in other circumstances, it may be best to set your manager onto your customers and let them work out what you should be doing. The primary method of breaking out of the circle of requirements capture, development, testing and delivery is to recognize that it is a circle. If you find that you are faced with an ever-expanding wish list, don’t add it to the existing work that you are doing. Instead, by cross-checking the project specification against the test results, you should be able to demonstrate to your customers that you have achieved at least an acceptable proportion of what you set out to do. If not every element of the specification is currently implemented, take a close look at what is left to do. It may be that some requirements were quietly dropped, while others might have been found to be unnecessary.

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

118

PROJECT SIGNOFF

This is the moment when you could suggest to your customers that, as they can now envisage potential additional functionality having used that which you have delivered, those features not yet implemented in the current version could be put into the requirements prioritization stage of what will be a new project – the next version of the program. So you can avoid being trapped in the continuous expansion of your project by starting out afresh with a blank sheet of paper. Once you’ve delivered what they asked for, that’s it. If it isn’t what they now need, then provided that you have agreed a specification, and can demonstrate delivery, there is no problem. On the other hand, resolving matters where there is a vague or incomplete description – or worse, different and unwritten conceptions – of what was to have been done, can be much more of a challenge.

Dealing with bad stuff: focusing on the successfully delivered objectives There is nothing so bad in application development that some good cannot be found in it. If nothing else, everyone concerned has learnt what not to do again. If you haven’t managed to deliver everything that was wanted, at least you’ve got some of the way there, and can do better next time. In some circumstances – such as when there is no agreed definition of the functionality that is to be delivered – success will always be impossible. If you should find yourself in this situation then try to involve a trusted third party, preferably one with experience of resolving these kinds of disputes, as a mediator. Alternatively, if you find that your users are concentrating on a small percentage of functionality that you might have been unable to deliver due to entirely unforeseeable external circumstances, try to bring their attention to bear more on the enormous amount of good stuff that you have actually done, whilst you simultaneously attempt to work out how soon you could give them the bits that they are shouting most about. In a (relative) disaster such as complete failure to get the customer to accept delivery of the final release, the first thing that you should do is work out what is the worst that could happen, and then plot how to cope with that. After this exercise, everything else will be easier to handle. So, for example, once you have worked out how you could defend your actions in court if necessary, and decided whether such action

WHERE NOW?

could be fatal to your organization, and thought about what you could do if it is, negotiating an agreed settlement will be much less stressful for you and your colleagues, and any outcome that is better than your worst-case scenario will feel like an unexpected bonus.

Identifying potential endpoints and agreeing an exit strategy However bad the situation – even if your customers are threatening legal action (or even illegal action) – at some point everyone will have to draw a line under the situation and move on. Find a way that saves most face for all involved, if that is what your customers are most worried about. If anyone has to end up with egg on their face, make it you. Egg doesn’t hurt, and your customers will appreciate the effort. On the other hand, make sure that you don’t set yourself up as a scapegoat, particularly if your customers are looking for someone to take the blame for everything that has gone wrong.

Where now? Whenever you finish a project, try and take the opportunity to mark its passing, and assess what you did well, and how you could do better the next time. If the project has been a success – you knew what you had to do, and you’ve done it – then this is the moment to enjoy. However, it’s unlikely you’ll have much time for this; instead, you’ll be faced with dealing with all the rest of the stuff you’ve got to take care of. The solution? Ruthless prioritization. Write down a list of all the things that you have to do, and then work out the order in which you need to do them. I tend to use the backs of envelopes for ‘To do’ lists, as there is only so much you can get on them. If you sometimes find it hard to decide what to do next, even when you’ve written your list, simply identify all the top priorities, and ignore all the others. If you’ve only got one top-priority job, then you know what you have to do; if you have more than one top-priority task, put a pin in the list to choose one. The others can be dealt with later, by which time the situation may well have changed.

119

120

PROJECT SIGNOFF

Once you have identified your top-priority task, look for prerequisites and dependencies – does any vital task need something else to be done before it can be started? Finally, use the ‘want/need’ scale. It may be that although you have a job that has to be finished in three days, you feel that in order to be able to do it properly you need to be refreshed and unworried, so you decide to take the afternoon off in order to get the car to the garage to sort out that funny noise it’s started making, and also catch up with an old friend in town. It is very important to sometimes stop being a software developer. We are all more than that, and it is important that we make time to remind ourselves of this. It may be that at the end of a project you find yourself assessing more than just how the project itself went. You may find that you need to move on to other challenges, even another career. There’s no harm in that. Or you may feel that you want to put what you’ve learnt from the last project into practice straight away, and improve your ability to deliver, even exceed expectations. That’s great. Whatever route you take from here, I hope you enjoy it, and I hope this book has helped you. Let me know how you get on, at [email protected].

Index

Note: Figures are indicated by italic page numbers. action definition 34, 35 adaptive development 12–13 advantages 50, 106 customer buy-in encouraged by 50–1 administration of development 84–5 agile computing 12–13 application lifecycle 4–6 assembly of sequences 60 see also sequence assembly example back-end processing, separation from front-end interface 33, 107 bottlenecks 77 boundary conditions, and testing 97 bugs action to be taken 114 finding 74, 95–6 reports via e-mail 82 buyers 48 attracting/retaining 76 cancelling an action 23 see also exit box cascade errors 44 checking features 83

clean machines, network of, as production environment 103 closing of project 117–20 code checkers 43 code management techniques 84 code rework, language to support 39 coding hints and tips 5, 79 libraries used 79–80 style 79–85 types 67 comments in coding 82 common sense 44 compatibility problems 41 conflicts between developers and testers 98 contigs 60 core requirements 22 creativity, software development 9–11 customer contact 22–3, 24, 50, 96–7, 107 customer expectations 24 managing 51 customer retention 76

Bioinformatics Software Engineering: Delivering Effective Applications. Paul Weston Copyright  2004 John Wiley & Sons, Ltd. ISBN: 0-470-85772-2

122

INDEX

customer-focused approach 6 cut-off date for new features 24 data input 29, 64 data processing 55–62 data structure diagrams 59–62 for complex assembly structure 62 as documentation 90 ‘is made up of’ symbol 60, 62 for simple assembly structure 61 database access library code for 80 query language for 38 dead ends 29 debugging tools 42–4 decision making, modelling of 56 defensive programming 84 delivery dates 18 delays 115–16 factors affecting 73–4 feedback from 74 notes 104 partial 116 demonstration of program, on minimal set of data 46 descriptive names for variables 81–2, 84 design tools and techniques 4–5 desktop (screen drawing) 25, 26 developers not as testers 95–6 versus test team 98 development costs compared with support costs 110–11 explaining by prototypes 45–6 hints and tips 5 see also adaptive development development environment 102 dialogue box screen drawing 25, 27 wording and appearance 21, 37 disciplined approach, benefits 11 DNA bases, sequence assembly of 60

documentation 89–92 data structure diagrams as 90 defining 5 from functional specifications 91 process flow diagrams as 90 program code as 89–90 screen drawings as 90 testing of 97 user stories as 90 drawings/sketches data structure diagrams 59 process flow diagrams 55–9 screen drawings 25–9 effective bioinformatics applications, how to deliver 6–7 endpoints, identifying potential 119 engineering approach benefits 11 and creativity 9–11 limitations 11–12 enhancement 6 error handling 64–5, 82–3 error messages 23, 65, 82, 96 error-catching code 64, 82–3 essentials, focusing on 78 evolutionary development 12–13, 49 exit box 57, 59 exit strategy 119 failures, ways of dealing with 118–19 FAQs (frequently asked questions), documentation to cover 92 feature creep 31 feedback 74, 96–7, 105, 107, 109, 111 urgent readjustments required as result of 111 flexibility 83–4 functional specification(s) 19, 31 documentation from 91 stages in writing 33–7 functionality 19–20 delivery of 73–4 identified in user stories 33–7 implementation of 38

INDEX

functionality changes cut-off date for 24 in upgrades 23 genetic algorithms 49 ‘go back’ action 23 handover of software formal 104 verbal/informal 105 hardware considerations 40 helpdesk facilities 92, 110–11 housekeeping 84 humour, misplaced 114 HyperTalk scripting language 49 Hypertext Markup Language (HTML) 38, 42 illogicality of users 98, 99 implementation considerations 39–44 purpose of 37–8 informal handover 105 information, excess 46–7 input data 29 information needed to define 23, 35 installation instructions (for engineering modification) 10 installation package 105–6 installers, information required by 92, 106 installing of program 23, 92 instance clashes 98 integration with other software 24, 92 interface see user interface KISS (Keep It Simple, Stupid) approach 80 known coding 67 legal action 118–19 library routines 79–80, 106 advantages and disadvantages of use 80, 106

lifecycle, software engineering 4–6 listening, importance of 6–7 loops 29 lunch, as marketing tool 76 maintenance 6 manual(s) 89–92 availability 109 marketing 47–8 meetings 24 menus 25, 27, 37 modular programming 84, 106 advantages 106 module testing 94–5 ‘naming of parts’ 81–2 new features 113 cut-off date for 24 in upgrades 23, 118 urgent request for 111, 113 obfuscation 80 online help 92 operating systems, effects 40–2 optimization 77–8 factors affecting 77 when to do 77 option selection confirmation screen 25, 28 menu (screen drawing) 25, 27 in user story 23 output 23, 29 parallel development cycles 113 usefulness 113–14 parameter(s) 34 stages in checking 35–6 partial delivery 116 performance statistics 24 platforms 40 post-delivery support 110–11 post-project activities 119–20 presentation show(s) 74–5 control of 75 datasets used 74–5

123

124

INDEX

visibility/appearance considerations 75 prevention 83 price 48 prioritization for next project 119 of user requirements 31, 115 Priority 2, 115 problem checking 44 problem messages 23 problem-handling routines 36 problems ways of dealing with 118–19 when to tell customers 47, 115–16 process flow diagrams 55–9 benefits 59 for complex process 58 as documentation 90 error handling loops 66 for linear process 55–6 for repeated process 56, 57 for report construction 67, 68 for selection process 56, 57 processing completion (screen drawing) 25, 28 production environment 103 program code aesthetic considerations 80–1 as documentation deliverable 89 transparency 89–90 programming languages 39 project definition 17–20 project proposal 17, 20 project signoff 6, 117–20 proofs of concept 45–6 prototypes, purpose of 46, 50–1 prototyping tools 42 pseudocode 59, 67, 69–70 example for report generation 69–70 good and bad examples of 89–90 meaning of term 69 transition to code 71 purpose of application software 17–18 quitting 23

random number generation 78 reliability 11 report generation 65, 67 process flow diagram for 68 pseudocode for 69–70 report-back facility of code 43 representation show, to demonstrate the program 74–5 reproducibility 11 requirements capture 4, 21–31 getting it right at early stage 31 requirements specification 31 measurement of success 19 test results checked against 117 restarting a program 92 result reporting 65–7 return values, checking 83 rollout and delivery 5, 101–7 sample dataset(s) 29 in presentation show(s) 74–5 in testing 97, 101 saving 23 scientists 49 screen drawings 24–9 distribution of copies 29 as documentation 90 examples 26–8 hints on use 25, 29 user interface developed from 37 selling 47–8 senior colleagues, covering for 114–15 sequence assembly example 29–30, 34 data structure diagrams for 60–2 library routines used 80, 106 modules 94 process flow diagrams for 55–9 pseudocode for report generation 69–70 testing 94–5 sequences 60 slippage 115–16 software development process 3–4 software engineering 9–13 solidity 11

INDEX

speeding up of processing 77 stability 11 balance with usefulness 107 Stage 1 functional specification 31 staged deliverables, agreement on 30 starting the coding 63–71 starting the program 23, 91 start-up screen (screen drawing) 25, 26 Structured Query Language (SQL) 38 success definition 19, 43 success measurement 18–19 by whom 19 successful delivery 118 test for 30 support 5–6, 92, 109–11 costs 110 different views on what is to be provided 110 local customs on 110 pre-emptive 109 responsibility 111 system administrators, access for 24 system environment 103 system testing 95 systems administrators, information/ documentation required 92

testers developers not as 95–6 users as 74, 96–7 versus developers 98 top-priority requirements 31, 115

technical writing 10 test environment 102–3 test planning and execution 5, 30, 93–9 things to look out for 97–9 when to stop testing 99 test plan(s) 93–4 for module testing 94–5 for system testing 95 for unit testing 94 writing 94–5

variable declarations 81 variable names 81–2, 84 verbal handover 105

undoing 23 unit testing 94 unknown coding 67 upgrades 23, 118 usage statistics 24 usefulness/stability balance 107 user interface 37, 78 buttons 37 data fields 37 menus 37 screen drawings and 37 separation from back-end processing 33, 106 user requirements 21, 23 information/documentation 91 prioritization of 115 user stories 22–4 as documentation 90 functionality identified in 33–7 users illogical 98, 99 involvement in testing process 74, 96–7

waterfall errors 83 web based interface(s) 38, 42 web page generation, library code for 80 web site, to demonstrate the program 74 worst, prepare for 95

125